AEA365 | A Tip-a-Day by and for Evaluators

TAG | performance management

My name is Stanley Capela, currently Vice President for Quality Management and Corporate Compliance Officer for HeartShare Human Services of New York, a 140 million dollar multi-service organization.

As with most government funded organizations, we have to show we are compliant with regulations and at same time meet certain performance metrics. As a result, I am confronted with how to create a system that focuses on quality assurance that meets performance metrics and incorporates quality improvement process. Using graphs we identified a series of deficiencies and sites that had poor performance. Then we drilled down further identifying areas that were cited as repeat deficiencies by state auditors. With this information, we developed a series of trainings focused on those deficiencies. As a result, we reduced repeat deficiencies in developmental disabilities. The key was to graphically present the data in a way that we were able to pinpoint specific sites that had the problem and developed a plan to improve performance.

Hot Tip: When setting up an internal monitoring system, we focus and prioritize areas that require the program to be compliant with government agencies. We select five to ten items and develop performance metrics. For our child welfare programs we focused on a number of areas such as adoption finalizations, AWOLs, client contacts, service plan timeliness and length of stay. Next, we set up a dashboard with appropriate charts; convene leadership team; review reports; identify challenges; develop interventions; and review progress after three months. After reviewing data we pinpoint which sites fail to meet targets. Over time, program sees improvement and realizes data utilization can lead to positive change.

Lessons Learned: One major problem when using this approach is when you focus on too many areas you get bogged down and accomplish little or no improvement. Make sure everyone has clear understanding that we are a team and that we are not out to get you. Often program directors focus on placing blame as opposed to dealing with problem. The key is focusing on program staff owning the data and realizing there are successes as well as challenges. In other words, perceptions can make a difference on how you approach quality assurance and performance measurements as you create a quality improvement culture. The other major issue is making sure the facilitator and the individual preparing data is independent and separate from program.

Rad Resources: Quality Evaluation Template: How to Develop a Utilization Focused Evaluation System Incorporating QI and QA Systems by Stan Capela.

Council on Accreditation – look at the Performance Quality Improvement (PQI) standard.

Council on Quality Leadership and their method Personal Outcome Measures (POMS)

The American Evaluation Association is celebrating Internal Evaluation (IE) Topical Interest Group Week. The contributions all this week to aea365 come from our IE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Zhao Min, Deputy Director of the Asia-Pacific Finance and Development Center (AFDC) in Shanghai, China. AFDC is a member of the CLEAR Initiative and hosts the East Asia CLEAR Center. If you follow what is happening in international evaluation capacity building (ECB), you might have heard about CLEAR – the Regional Centers for Learning on Evaluation and Results. Through a network of regional centers, CLEAR provides regionally relevant ECB, p promoting practical knowledge-sharing and peer-to-peer learning. Our center provides many types of ECB activities to people in Asia and internationally – on topics such as the basics of M&E, impact evaluation, and performance based budgeting. In addition to CLEAR, we also receive support from the Independent Evaluation Group of the World Bank, the Independent Evaluation Department and also the Strategy and Policy Department of the Asian Development Bank (ADB), and the Ministry of Finance of China. Today I’m writing about performance management systems (PMSs), their uses and how they intersect with the budgeting process in the government sector. This is an area of my own research and my agency’s knowledge sharing efforts within Asia, especially in China.

Lessons learned: Worldwide, budgets are tight and citizens expect high performance on public sector projects and programs – be they in health, education, transportation and, really, any sector. I first became introduced to how PMSs, or results-based management systems, through my work with international financial institutions such as the ADB. Increasingly, I’m seeing the use of electronic PMSs in countries, states and municipalities – across the globe.

Some of the many uses of the systems are below.

  • They provide a systematic approach to management and monitoring of the performance of projects and programs. Instead of ad-hoc management, information can be systematically collected and monitored.
  • Access to information increases.
  • They provide structure in the early, design stage of projects, programs or policies. They typically set out important indicators and targets to enable meaningful monitoring.
  • Robust PMSs can be a source of key data for evaluations – and make evaluations stronger.
  • Evidence-based decision-making is more likely, resulting in greater efficiencies and effectiveness.
  • Decisions about budgeting and tying budgeting to performance are made possible by PMSs.

PMSs are likely to become more commonly used. It is an exciting time to learn and share with one another across regions in our experiences with PMSs. We can continue to refine such things as which indicators are most useful to track, experiences in collecting data, and how to communicate information to the public.

Rad Resources:

Clipped from http://www.theclearinitiative.org/Clear_about.html

The American Evaluation Association is celebrating Centers for Learning on Evaluation and Results (CLEAR) week. The contributions all this week to aea365 come from members of CLEAR. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! We are Laura Beals (program evaluator) and Noah Schectman (database administrator) from Jewish Family and Children’s Service (JF&CS), located in Boston, MA. At JF&CS we use a cloud-based case management system to facilitate data collection about our clients and services. We are part of the internal evaluation department at JF&CS (i.e., he is not part of IT) and so we work closely together to ensure that we are collecting data in a way that allows us to complete our analyses in the most efficient manner possible.

Lessons Learned: Though our system has built-in reporting tools, we often download data for more complex analysis in another tool, such as Access or SPSS. In addition, though the data collection tools are designed as easy-to-complete forms in the system, we do have to bulk upload data regularly.

Many case management/performance management systems allow for back-end customization of the data collection tools—you may have the ability to do so in-house (as we do) or you may have to work with a third-party developer. Regardless, as an evaluator, if you are working with an online performance management system, you should ask yourself: “What does the data need to look like when it is downloaded? When it is uploaded?” In general, we first think about how the data will be used, then design the data architecture to match.

Hot Tips: When designing new data collection tools in our database, we ask several key questions about how the data should be formatted on the back-end, including:

–       What are the unique identifiers for each case that will need to be downloaded with or uploaded to the database?

–       Should the data be arranged so that a case is on a row or each assessment is on a row?

–       For each variable, are the variable labels or the numerical values used?

–       How are multiple response variables formatted? As dummy variables?

–       If names are used, how will they be formatted? What about addresses? What about dates?

Even when we think we have it figured out, we always enter fake assessments for fake clients in the system, through the online form and through a bulk upload, and then download the data. We then review the resulting import/download and triple-check that the data is formatted in the manner we expect. We find doing the work to prepare the system ahead of time saves us a lot of data formatting and manipulation down the road!

2014-02-03 AEA365 Multiple Response_2

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

We are Nichole Stewart and Laura Pryor and we’d like to share a preview of our presentation at the upcoming AEA 2013 conference. Our session, Performance Management to Program Evaluation: Creating a Complimentary Connection, will use a case study of a Los Angeles-based juvenile offender reentry program to demonstrate how “information and knowledge production” can be coordinated for performance management (PM) and program evaluation (PE).

Lessons Learned: There IS a difference!

Distinguishing between PM and PE has historically presented challenges for program directors and the public agencies and non-profit organizations that fund them. Programs have to grapple with day-to-day operations as well as adapting to evolving frameworks for understanding “what works”—from results-based accountability to continuous quality improvement to evidence-based everything. Evaluators are frequently called upon to engage simultaneously in both PM and PE, however the distinctions between the tasks are not always clearly understood or articulated in practice.

Lessons Learned: There IS a connection!

Fortunately, several authors have explored the relationship between PM and PE and outlined how PM and PE can complement one another with regard to data collection and analysis:

  • Information complementarity– Use the same data to answer different questions based on different analyses (Kusek and Rist, 2004).
  • Methodical complementarity– Use similar processes and tools to collect and analyze data and ultimately convert data into actionable information (Nielsen and Ejler, 2008).

Rad Resources

StewartPryorGraph

Source: Child Trends, Research-to-Results Brief (January 2011)

Hot Tips:

 

The American Evaluation Association is celebrating Business, Leadership and Performance (BLP) TIG Week with our colleagues in the BLP AEA Topical Interest Group. The contributions all this week to aea365 come from our BLP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. Want to learn more from Nichole and Laura? They’ll be presenting as part of the Evaluation 2013 Conference Program, October 16-19 in Washington, DC.

·

Archives

To top