AEA365 | A Tip-a-Day by and for Evaluators

TAG | standards

Hi there! We’re Anne Vo, Ph.D., Director of the Keck Evaluation, Institutional Reporting, and Assessment (KEIRA) Office at the Keck School of Medicine of USC, and Jacob Schreiber, Evaluation Assistant at KEIRA. Today, we offer reflections on what we’ve learned about conducting evaluation within an academic medical center—an environment that offers rich opportunities to observe, conduct, and understand evaluation practice and policy.

Hot Tip #1: Standards Rule Healthcare, Medicine, and Medical Education

Medicine is a highly regulated field. Broad swaths of stakeholders—clinicians, clinical educators, school and hospital administrators—rely on standards to inform decision-making and drive practice. As such, systematic evaluation often manifests as high turn-around monitoring of easily quantifiable outcomes (e.g., student academic performance, residency program match rates, etc.). Successfully “chasing numbers” enables organizations such as academic medical centers to communicate that standards of care and teaching are being met. Because standards offer a common language that stakeholders can use to think through pressing issues of the day, they also become the go-to frame of reference for decision-makers throughout the organization.

Rad Resource:

Hot Tip #2: Everything is “Evaluated,” Everyone is an “Evaluator”

Because standards drive practice in Medicine, evaluation could become a decentralized activity. Aspects of evaluative practice—from question formulation, to data collection, monitoring, analysis, and synthesis—can often be divided among various stakeholder groups across an organization. This cascaded evaluation model emphasizes “local expertise” and echoes “team values” to which healthcare teams aspire. It is reminiscent of development evaluations that organizations such as UNICEF and the UNDP strongly support. And, in the medical context, it is a model that tends to immerse clinical experts in monitoring processes and largely distances them from actual evaluation.

Rad Resource:

Hot Tip #3: Democratic Decision Making is a Core Value

Decisions about how medical education is done are often made through committees and guided by accreditation standards; specifically, LCME Standard 1 on Mission, Planning, Organization, and Integrity (see above link) and Standard 2 on Leadership and Administration. Academic and administrative committees oversee and monitor the quality of Undergraduate Medical Education (what we know as the first four years of medical school). Many of the same stakeholders serve across committees as well as the sub-committees and work groups within each. For evaluation to be meaningful, expect to have many of the same conversations with the same people on different levels. Most importantly, know what each committee’s charge is; its membership; and members’ roles and stances on issues that are up for discussion.

Rad Resource:

Alkin, M.C. and Vo, A.T. (2017). What Is the Organizational, Community, and Political Context of the Program? (pp. 77-87). In Evaluation Essentials: From A to Z (2nd Edition). New York, NY: Guilford Press.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Bethany Laursen, an independent consultant and evaluation specialist for several units at the University of Wisconsin. I fell in love with social network analysis (SNA) as a graduate student because SNA gave me words and pictures to describe how I think. Many evaluators can relate to my story, but one of the challenges to using SNA in evaluation is identifying what counts as a “good” network structure.

Hot Tip: Identify keywords in your evaluation question(s) that tell you what counts as a “good” outcome or goal for your evaluand. An example evaluation question might be: “How can we improve the collaboration capacity of our coalition?” The stated goal is “collaboration capacity.”

Hot Tip: Use published literature to specify which network structure(s) promote that goal. Social networks are complex, requiring rigorous research to understand which functions emerge from different structures. It would be unethical for an evaluator to guess. Fortunately, a lot has been published in journals and in gray and white papers,

Continuing our example, we need to research what kinds of networks foster “collaboration capacity” so we can compare our coalition’s network to this standard and find areas for improvement. You may find a robust definition of “collaboration capacity” in the literature, but if you don’t, you will have to specify what “collaboration capacity” looks like in your coalition. Perhaps you settle on “timely exchange of resources.” Now, what does the literature say about which kinds of networks promote “timely exchange of resources”? Although many social network theories cut across subject domains, it’s best to start with your subject domain (e.g. coalitions) to help ensure assumptions and definitions mesh with your evaluand. Review papers are an excellent resource.

Lesson Learned: Although a lot has been published, many gaps remain. Sometimes the SNA literature may not be clear about which kinds of networks promote the goals you’ve identified for your evaluand. In this case, you can either 1) do some scholarship to synthesize the literature and argue for such a standard, or 2) go back to your evaluation question and redefine the goal in narrower terms that are described in the literature.

In our example, the literature on coalition networks may not have reached consensus about which types of networks promote timely exchange of resources. But perhaps reviews have been published on which types of brokers foster diversity in coalitions. You can either 1) synthesize the coalition literature to create a rigorous standard for “timely exchange of resources,” or 2) reframe the overall evaluation question as, “How can brokers improve diversity in our coalition’s network?”

Rad Resources:

This white paper clearly describes network structures that promote different types of conversations in social media

This short webinar reports a meta-synthesis of which networks promote adaptive co-management capacity at different stages of the adaptive cycle

Different network structures promote different system functions. This is the take home slide from the Rad Resource on ACM capacity. In this case, the evaluand's network goal is timely social learning, collective action, and resilience.

Different network structures promote different system functions. This is the take home slide from the Rad Resource on ACM capacity. In this case, the evaluand’s network goal is timely social learning, collective action, and resilience.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, my name is Diane Rogers and I am a doctoral student in the Interdisciplinary Doctoral Program in Evaluation at Western Michigan University (WMU) and a secondary mathematics teacher. My work in evaluation focuses on education and social justice, so naturally I was excited to have the opportunity to work with and learn from Dr. Rodney Hopson during his visit to WMU.

As a teacher and school improvement chair in an urban school district, I became quickly frustrated with my, and others’, lack of knowledge about using systematic inquiry to make decisions. It seemed as though decisions for school improvement were made for convenience, maintaining the status quo, or because someone had a feeling that something would work. As my school went through the restructuring process, I saw that evaluation could be part of that solution. Yet, in a school full of diverse students and needs, evaluation without cultural competence and adherence to professional standards will not help to ameliorate the complex issues of the educational system. The importance of the concepts of cultural competence and professional standards to our work as evaluators are two of the lessons that Dr. Hopson’s visit highlighted

Lesson Learned: As part of the visiting scholar series at WMU, Dr. Hopson facilitated a discussion with The Evaluation Center staff about the need for and application of AEA’s Statement on Cultural Competence in Evaluation (http://www.eval.org/ccstatement.asp). I left the meeting with a renewed understanding that cultural competency is not a checklist of knowledge and skills that an evaluator can accomplish. Rather cultural competence is a process that requires reflection, learning, dialogue, and more reflection throughout one’s life. For me this process began by exploring my various societal labels and unpacking how those identities impact my worldview, language, and life ways.

Lesson learned: I was privileged to attend a panel discussion of the Program Evaluation Standards (PES) between Dr. Hopson and Dr. Daniel Stufflebeam. (View the video here: http://vimeo.com/33177056) I gained a great deal of insight into the standards from hearing Dr. Stufflebeam’s account of the history of the PES and Joint Committee and Dr. Hopson’s experience in revising the PES. Both scholars communicated the importance of standards to our profession and the need for all evaluators to not only adhere to the standards but also participate in professional conversations about the meaning and application of the standards. It is not enough to own a copy of the PES; you have to read, process, and apply the information throughout your work.

Rad Resource: View a summary of the Program Evaluation Standards here: http://www.eval.org/evaluationdocuments/progeval.html

All this week, we’re highlighting posts from colleagues at Western Michigan University as they reflect on a recent visit from incoming AEA President Rodney Hopson. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

My name is Maureen Wilce and I’m the team leader for the Program Evaluation and Community Interventions Team in the Air Pollution and Respiratory Health Branch in CDC’s National Center for Environmental Health. My team provides technical assistance to support the evaluation efforts of 36 state partners in the National Asthma Control Program.

Together with the Environmental Protection Agency, we have created a four-part Webinar series on program evaluation basics. In the series, nationally recognized experts

1)      present a general introduction to program evaluation,

2)      note challenges in conducting useful evaluations as well as methods for overcoming those challenges,

3)      introduce the six steps of the CDC Framework for Program Evaluation using examples that are relevant to our state partners working in asthma control, and

4)      emphasize the importance and utility of the evaluation standards.

The series is appropriate for novice evaluators, program staff, and others interested in learning about CDC’s approach to program evaluation. Presenters include Christina Christie, Leslie Fierro, Carlyn Orians, and Tom Chapel. Individual Webinars range in length from 25 to 65 minutes.

Rad Resource: Our webinar series is entitled “Using Evaluation to Reduce the Burden of Asthma: A Web-based Introduction to CDC’s Framework for Program Evaluation“, and you can find it here.  PowerPoint slides with accompanying transcripts are available for each Webinar.

Hot tip: If you’re working with stakeholders—board members, for example—and you want to help them see the merit in measurement in the middle of the logic model (rather than only at the far right end), the introductory Webinar is for you! It’s an engaging 25 minutes that uses examples from our asthma work, as well as from diabetes, cardiovascular health, soccer, and even furniture production in Poland. Serve it up with a bowl of popcorn and even the most evaluation-resistant stakeholders will jump on the band wagon… or at least stop running the other way when they see you approaching in the hall.

Lesson Learned: Engaging the services of an instructional designer to review your PowerPoint presentation before turning it into a Webinar can help ensure that your learning objectives are clear—and are clearly met.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Michael Kiella. I am a student member of the American Evaluation Association, and a doctoral student at Western Michigan University in Kalamazoo Michigan. I served as a session scribe at Evaluation 2010 for Session 393: Research on Evaluation Standards and Methods. For this post, I will focus on the presentation by Dr. Linda Mabry (Washington State University at Vancouver) entitled Social Science Standards and Ethics: Development, Comparative Analysis, and Issues for Evaluation.

Lessons Learned:

1. Justification is not equivalent to doing the right thing.

Dr. Mabry indicated that ethics within our profession is not an answer for all time, but a sequence captured in context and history. She wants us to know that there is an historical backdrop in the historical development of ethical standards for modern times and has selected the Nurnberg War Trials, the Declarations of Helsinki, and the Belmont report as standard.

Dr. Mabry argues that there must be a standard of ethics which applies within social science and evaluation efforts. She offers the Professional Standards of the American Psychological Association (APA), and the American Evaluation Association (AEA) as evidence that the practitioners in these fields have addressed the issue. Yet, these standards remain problematic.

2. Is the presumption of compliance enough to be compliant?

These features are problematic because they do not include enforcement components, and both explicitly indicate that the standards do not establish a baseline of liability. Dr. Mabry suggests that a possible alternative is that government has a role in enforcing professional standards where human subjects are used in research.

3. It is reasonable for government to exercise its authority over our research endeavors.

Dr. Mabry argues that it is the legitimate place for government to exercise its role as an enforcement agency to balance the extraction of data for the public good with the protection of the subjects from which the data are extracted. But this too is problematic because the American Evaluation Association has not agreed on a common definition of what evaluation really is. The establishment of oversight committees with enforcement authority is difficult because the definition of Evaluation is so very broad and the extent of our practices is so varied that we are unlikely to agree upon compliance criteria.

4. Cultural Sensitivity as an arena for new standards.

Dr. Mabry proposes that in order to appropriately evaluate culturally distinctive features, we are required to make the strange familiar. The nuance of culture may not be immediately observable or understood; feasibility remains in conflict with ethical research.

At AEA’s 2010 Annual Conference, session scribes took notes at over 30 sessions and we’ll be sharing their work throughout the winter on aea365. This week’s scribing posts were done by the students in Western Michigan University’s Interdisciplinary PhD program. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· ·

My name is Lori Wingate. I am a Principal Research Associate at The Evaluation Center at Western Michigan University. Two closely related topics I return to frequently in my research, practice, and teaching are metaevaluation and the Program Evaluation Standards (Joint Committee, 1994).  Here I share some lessons learned from my recent dissertation research on the use of the Program Evaluation Standards a rating tool for metaevaluation.

The Program Evaluation Standards are a set of 30 standards organized in four domains: utility, feasibility, propriety, and accuracy. Correspondingly, they are grounded in the principles that evaluations should be useful, practical, ethical, and valid.

Because of their applicability to a broad array evaluation contexts and widespread acceptance, they are often used as criteria in metaevaluation. Although the Standards provide a useful metaevaluation framework, there are some significant challenges to their application when a metaevaluation is focused on evaluation reports, without opportunity to gather additional information about the evaluation’s conduct.

This claim is based on my personal experience in using the Standards to evaluate reports, and is strongly supported by the findings from my study of interrater agreement in metaevaluation. Although agreement was generally low across all the standards, the uncalibrated raters had the least agreement on standards in the feasibility and propriety domains, which are largely concerned with issues related to the manner in which an evaluation is carried out. With only reports in hand to judge the evaluation, raters had to infer quite a bit in order to make judgments about evaluation process.

If you’re thinking of conducting a metaevaluation in which you will use the Program Evaluation Standards as criteria and you have only evaluation reports for data, here are some tips and resources that may help make it a more valid and useful endeavor:

Hot Tip: Select only those standards on which judgments can be made based on information that is typically included in evaluation reports.

Rad Resources: Check out the Program Evaluation Standards at www.jcsee.org.  Watch for a new edition to be published this year. A review of Dan Stufflebeam’s Program Evaluation Metaevaluation Checklist will help you get started in determining which standards will be feasible for use in your metaevaluation.

Hot Tip: If you want to look at several reports produced for a by a single organization or in a single content area, spend some time developing tailored criteria for that context.

Rad Resource: ALNAP’S Quality Proforma is an instrument designed for assessing humanitarian action evaluation reports. The criteria are tailored to the domain in which the evaluations were conducted and are focused on report quality.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

Hello, my name is Scott Cody and I’m Deputy Director of the What Works Clearinghouse (WWC). I’m also the Associate Director of Human Services Research at Mathematica Policy Research. I’d like to share with you an important resource for researchers, educators, and policy makers. Evaluators know how important study design is to the validity of study results. Valid research can educate the public and empower them to make better decisions about everything from healthcare to education.

Resource: The WWC, founded by the U.S. Department of Education’s Institute of Education Sciences, is a central source for comprehensive reviews of education research. Each WWC review takes a thorough look at the research on a particular topic or education program, product, or practice. Our goal is to identify well-designed studies and summarize those studies’ findings for decision-makers. To do this, we measure each study against the WWC research standards. These standards apply to the study methodology, the strength of the study’s data, and the adequacy of the study’s statistical procedures. We then summarize the findings of all studies that meet WWC standards, and develop an overall rating of effectiveness. In this way, WWC reports tell educators what the highest-quality research says about the effectiveness of individual education interventions. The WWC may be accessed online at http://ies.ed.gov/ncee/wwc/.

Want to learn more about the WWC?  Join Scott for an AEA Coffee Break Webinar, Thursday, May 13, 2:00-2:20 PM EST. He’ll demonstrate how to get the most out of the WWC website and locate important information for decision-making in education. Sign up at http://ow.ly/1IkXX

Want to learn more about the WWC’s standards for research?  Join Neil Seftor, Deputy Director of the WWC for an AEA Coffee Break Webinar Thursday, May 20, 2:00-2:20 PM EST. He’ll cover the design and reporting requirements researchers must demonstrate to meet WWC standards. Sign up at http://ow.ly/1IkXX

· ·

Archives

To top