AEA365 | A Tip-a-Day by and for Evaluators

TAG | quality improvement

Hello! We are Dana Linnell Wanzer, evaluation doctoral student, and Tiffany Berry, research associate professor, from Claremont Graduate University. Today we are going to discuss the importance of embedding quality throughout an organization by discussing our work in promoting continuous quality improvement (CQI) in afterschool programs.

CQI systems involve iterative and ongoing cycles of goal setting about offering quality programming, using effective training practices to support staff learning and development, frequent program monitoring including site observations and follow-up coaching for staff, and analyzing data to identify strengths and address weaknesses in program implementation. While CQI within an organization is challenging, we have begun to engage staff in conversations about CQI.

Hot Tip: One strategy we used involved translating the California Department of Education’s “Quality Standards for Expanded Learning Programs” into behavioral language for staff. Using examples from external observations we conducted at the organization, we created four vignettes that described a staff member who displayed both high and low quality across selected quality standards. Site managers then responded to a series of questions about the vignettes, including:

  • Did the vignette describe high-quality or low-quality practice?
  • What is the evidence for your rating of high or low quality?
  • What specific recommendations would you give to the staff member to improve on areas of identified as low quality?

At the end of the activity, site managers mentioned the vignettes resonated strongly with their observations of their staffs’ practices and discussed how they could begin implementing regular, informal observations and discussions with their staff to improve the quality of programming at their sites.

Hot Tip: Another strategy involved embedding internal observations into routine practices for staff. Over the years, we collaborated with the director of program quality to create a reduced version of our validated observation protocol, trained him on how to conduct observations, and worked with him to calibrate his observations with the external observation team. Results were summarized, shared across the organization, and were used to drive professional development offerings. Now, more managerial staff will be incorporated into the internal observation team and the evaluation process will continue and deepen throughout the organization. While this process generates action within the organization for CQI, it also allows for more observational data to be collected without increasing the number (and cost!) of external evaluations.

Rad Resource: Tiffany Berry and colleagues wrote an article detailing these process on “Aligning professional development to Continuous Quality Improvement: A case Study of Los Angeles Unified School District’s Beyond the Bell Branch.” Check it out for more information!

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi. We’re Sherry Campanelli, Program Compliance Manager and Laura Newhall, Clinical Training Coordinator, from the University of Massachusetts Medical School’s Disability Evaluation Services (DES). As an organization, we are committed to ongoing quality improvement (QI) efforts to enhance our services for people with disabilities especially during times of major change or crisis. It hit home recently when a blog by Sheila Robinson identified a topic that AEA365 readers were interested in and something we’ve struggled with ourselves: “how to incorporate evaluation when there’s a myriad of other seemingly more pressing tasks without pushing it to the back burner.”

When an organization is in a time of crisis (high staff turnover, new business processes, or increased work load), it’s difficult to maintain QI efforts and evaluation. There will be pressure to suspend or reduce internal evaluations in favor of more urgent tasks. Evaluation is always important, but rarely urgent. So how do we preserve a commitment to evaluation during these times?

Developed by the 34th U.S. president, the Eisenhower Box is one effective approach for addressing this challenge. It’s a systematic method to assure that reacting to urgent demands doesn’t prevent important tasks from being accomplished. It categorizes tasks along two axes and four quadrants based on degree of importance and urgency. The figure below demonstrates how the Eisenhower Box is used to prioritize and address work demands.

 

The Eisenhower Box:

Muhr

The key to preserving on-going evaluation efforts is to schedule time to perform activities while resisting the temptation to address only urgent demands. The danger of not prioritizing “important but not urgent” tasks is that you will either fail to accomplish the tasks or not finish them in a timely manner. Completing all important tasks, whether urgent or not, is critical to organizations creating forward motion.

Lessons Learned:

Here are some lessons learned on how to best integrate evaluation into the myriad of pressing tasks without pushing it to the back burner.

  • Remind staff of the organization’s commitment to QI and how it’s been helpful in the past.
  • Resist temptation to suspend evaluation in the name of efficiency or “getting more done”.
  • Support staff to remain engaged in evaluation activities by assisting them with time management skills; keep the focus away from unimportant tasks.
  • Maintain key components of your evaluation structure (e.g., maintaining a valid sample size).
  • Look for efficiencies in the “unimportant and not urgent” areas of your organization’s processes.

Thoughts to Ponder:

“What is important is seldom urgent, and what is urgent is seldom important.” – Dwight Eisenhower

Rad Resources:

“How to be More Productive and Eliminate Time Wasting Activities by Using the “Eisenhower box” by James Clear.

“Urgent vs. Important” by Hamza Khan.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello AEA365ers! We are Suzanne Markoe Hayes (Director) and Elaine Donato (Internal Evaluator) from the Evaluation and Research Department at Volunteers of America Greater Los Angeles (VOALA), a large non-profit organization whose mission is to enrich the lives of those in need.

One program we support is VOALA’s largest emergency shelter located in South Los Angeles— an area known for having the densest homeless population in Los Angeles County. As an initiative led by United Way Greater Los Angeles to end chronic homelessness by 2016, VOALA’s shelter joined homeless service providers in South L.A. to design and implement a Coordinated Entry System (CES). To develop such a system, participating service providers were required to join forces for the very first time. The collaborative was going to be a challenge due to the extensive history of homeless service providers in South L.A. having scarce resources and competing for the same scraps of funding.

Human service organizations are being asked to collaborate strategically to address social issues, and they must do so with their existing limited resources. For majority, this includes having no funding for a third-party evaluator and/or support from an internal evaluation department. Recognizing these limitations, VOALA contributed their Internal Evaluation team to assist with the collective impact of the South L.A. CES collaborative. We implemented a process evaluation to help identify the overarching collaborative goals, the processes that will occur, and to define each organization’s role. As a result, the South L.A. CES team successfully designed a unique system to link chronically homeless individuals in their community with the most appropriate services and housing.

Here are hot tips to implement a collaborative process evaluation:

Hot Tip #1: Make clear to all participating organizations that the evaluator is here to assist all agencies, not just own agency.

Hot Tip #2: Create process maps to help identify each organization’s role in the process. As a key element for continuous quality improvement (CQI), process maps can also be useful in tracking the activities related to achieving desired outcomes.

Markoe Hayes Donato

Hot Tip #3: Create a safe, open environment where team members are allowed to share their innovative ideas on how to better serve the target population and strengthen existing processes.

Hot Tip #4: Produce dashboard reports and share in biweekly meetings to inform decision-making and track team goals and desired outcomes.

Rad Resource: Check out the Center for Urban Community Services for their training CQI methods including process maps.

The American Evaluation Association is celebrating Internal Evaluation (IE) Topical Interest Group Week. The contributions all this week to aea365 come from our IE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi. We’re Sherry Campanelli, Program Compliance Manager and Laura Newhall, Clinical Training Coordinator, from the University of Massachusetts Medical School’s Disability Evaluation Services. As an organization, we are committed to ongoing quality improvement efforts to enhance our services for people with disabilities. Earlier, we wrote about common techniques that help a quality improvement (QI) team to be successful. Today we share some potholes and pitfalls we’ve encountered in group facilitation and our tips for negotiating them successfully:

Lessons Learned:

  • New problems or issues frequently arise in the middle of a QI project. Team members, management, or external events (such as changes in the industry) can generate issues unrelated to the original charge. This can be discouraging for the team members and leader and can delay completion of the project. The following may be helpful.
    • Reaffirm the team’s goals, mission, and review data as a group to ascertain if the new issue should be addressed in this venue or in another way.
    • Allow team members to opt out of participating in the new task. Seek new members for the team as needed to address the new issue(s).
    • Keep a “hot” list of issues that arise to be addressed by future QI teams.
  • Recommendations from team not fully accepted. A less than enthusiastic response from decision- makers to a team’s recommendations is a challenge for any team.
    • Set expectations with the group up front that recommendations might be accepted, rejected or amended.
    • Sustain the group’s enthusiasm during the revision process by reminding them of the importance of their work and input regardless of the outcome.
    • Emphasize the positive feedback before sharing constructive feedback. Thank team members for their efforts.
    • Ensure that relevant decision-makers are regularly briefed so the team can make “mid-course corrections” toward options likely to be approved.
  • Difficulty achieving full team consensus. This can be due to dominating or defensive team member(s), incomplete information or team members needing more time for analysis.
    • Encourage subgroup and individual work on the issue between meetings.
    • Allow the team to live with ambiguity for a while to enable consensus to develop.
    • Document what’s already been decided and refer team members back to prior discussions.

Thoughts to Ponder:

“The best-laid plans of mice and men / Often go awry” – from a poem by Robert Burns. The QI team process does not always go smoothly; however, these unexpected challenges present opportunities for better overall outcomes.

From a motivational poster by the British government in 1939, the facilitator must “keep calm and carry on” through the potholes and pitfalls of the QI team process.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi.  We’re  Sherry Campanelli, Program Compliance Manager and Laura Newhall, Clinical Training Coordinator, from the University of Massachusetts Medical School’s Disability Evaluation Services (DES). Although DES conducts evaluations regarding whether an applicant for public benefits can be found disabled, evaluation as a research endeavor is not our primary focus. Nevertheless, as an organization, we are committed to ongoing quality improvement efforts to enhance our services for people with disabilities. We use a team-based iterative approach to define and address problem functions and processes.

For example, we used the process described herein to develop Quality Assurance systems for our clinical, clerical and technical support processes. We have also used this method to tackle caseload backlogs, and effective processing of incomplete applications.

We’ve discovered over time, regardless of the issue or problem involved, that there are common techniques that help a quality improvement (QI) team be successful. We would like to share some of these lessons learned with you.

Lesson Learned: 

  • Determine and clearly state the issues to be solved and team goals.
  • Involve key staff (line staff doing the work and managers supervising the work) in the development of any QI initiative. They are in “the know” about areas that may be problematic.
  • Incorporate non-judgmental facilitation to keep up the momentum. Key components include:

o   Involving all participants in decision making/discussion;

o   Keeping meeting minutes and agendas;

o   Keeping track and sharing “to do” lists, “next steps” and progress towards goals;

o   Meeting on a regular and ongoing basis (don’t cancel meetings unless absolutely necessary);

o   Seeking management decisions and input as needed; and

o   Making sure you hear from the quiet folks in the room – they may need a little encouragement to speak up, but often offer great insights.

  • Utilize team members/subcommittees to perform specific tasks between meetings.
  • Utilize available qualitative and quantitative data.
  • Collect specific data, as necessary, to help define the problem and suggest solutions.
  • Do fact finding to support decision-making.
  • Maintain a “living” working document(s) as decisions are made to be incorporated into a final product.

Utilize pilot testing to determine feasibility and make changes (i.e., “fix bugs”) prior to full implementation.

  • Provide periodic communication to the rest of the department or organization during the project and at its conclusion.
  • Train all impacted staff on process improvements.
  • Conduct periodic assessments after implementation to assess success of the project.
  • Refine processes as new issues and changes occur.

Hot Tips:

  • Sometimes QI processes take longer than expected. “Keep going even when the going is slow and uncertain.”  G.G. Renee Hill
  • “To discover new ways of doing something – look at a process as though you were seeing it either for the first or last time.” Mitchel Martin

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Archives

To top