Hello! We are Johanna Morariu, Kat Athanasiades, and Ann Emery from Innovation Network. For 20 years, Innovation Network has helped nonprofits and foundations evaluate and learn from their work.

In 2010, Innovation Network set out to answer a question that was previously unaddressed in the evaluation field—what is the state of nonprofit evaluation practice and capacity?—and initiated the first iteration of the State of Evaluation project. In 2012 we launched the second installment of the State of Evaluation project. A total of 546 representatives of 501(c)3 nonprofit organizations nationwide responded to our 2012 survey.

Lessons Learned–So what’s the state of evaluation among nonprofits? Here are the top ten highlights from our research:

1. 90% of nonprofits evaluated some part of their work in the past year. However, only 28% of nonprofits exhibit what we feel are promising capacities and behaviors to meaningfully engage in evaluation.

2. The use of qualitative practices (e.g. case studies, focus groups, and interviews—used by fewer than 50% of organizations) has increased, though quantitative practices (e.g. compiling statistics, feedback forms, and internal tracking forms—used by more than 50% of organizations) still reign supreme.

3. 18% of nonprofits had a full-time employee dedicated to evaluation.

4. Organizations were positive about working with external evaluators: 69% rated the experience as excellent or good.

5. 100% of organizations that engaged in evaluation used their findings.

6. Large and small organizations faced different barriers to evaluation: 28% of large organizations named “funders asking you to report on the wrong data” as a barrier, compared to 12% overall.

7. 82% of nonprofits believe that discussing evaluation results with funders is useful.

8. 10% of nonprofits felt that you don’t need evaluation to know that your organization’s approach is working.

9. Evaluation is a low priority among nonprofits: it was ranked second to last in a list of 10 priorities, only coming ahead of research.

10. Among both funders and nonprofits, the primary audience of evaluation results is internal: for nonprofits, it is the CEO/ED/management, and for funders, it is the Board of Directors.

Rad Resource—The State of Evaluation 2010 and 2012 reports are available online at for your reading pleasure.

Rad Resource—What are evaluators saying about the State of Evaluation 2012 data? Look no further! You can see examples here by Matt Forti and Tom Kelly.

Rad Resource—Measuring evaluation in the social sector: Check out the Center for Effective Philanthropy’s 2012 Room for Improvement and New Philanthropy Capital’s 2012 Making an Impact.

Hot Tip—Want to discuss the State of Evaluation? Leave a comment below, or tweet us (@InnoNet_Eval) using #SOE2012!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · Data · Evaluation · nonprofit · organizational learning · reports

Hello, I am Maxine Gilling, Research Associate for Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP). I recently completed my dissertation entitled How Politics, Economics, and Technology Influence Evaluation Requirements for Federally Funded Projects: A Historical Study of the Elementary and Secondary Education Act from 1965 to 2005. In this study, I examined the interaction of national political, economic, and technological factors as they influenced the concurrent evolution of federally mandated evaluation requirements.

Lessons Learned:

• Program evaluation does not take place in a vacuum. The field and profession of program evaluation has grown and expanded over the last four decades and eight administrations due to political, economic, and technological factors.
• Legislation drives evaluation policy. The Elementary and Secondary Education Act (ESEA) of 1965 established policies to provide “financial assistance to local educational agencies serving areas with concentrations of children from low-income families to expand and improve their educational program” (Public Law 89-10—Apr. 11, 1965). This legislation also had another consequence: it helped drive the establishment of educational program evaluation and the field of evaluation as a profession.
• Economics influences evaluation policy and practice. For instance in the 1980’s evaluation took a downturn due to the stringent economic policies. Program evaluators resorted to lessons learned through writing journals and books.
• Technology influences evaluation policy and practice. The rapid emergence of new technologies all contributed to changing goals, standards, and methods and values underlying program evaluation.

Resources:

•  HeinOnline: The Modern Link to Legal History. HeinOnline consists of major full text legal collections including: U.S. Federal Legislative History Library where you can study the changes in evaluation policy over time.
• Elementary and Secondary Education Act (also known as the No Child Left Behind Act).
• David Cohen’s historical article:Politics and Research: Evaluation of Social Action Programs in Education.
• Caroly Weiss’ historical book:Evaluating action programs: Readings in social action and education.
• Ralph W. Tyler’s many works. For example see the chapter, Ralph W. Tyler’s contribution to program evaluation in the book Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

economics · Elementary and Secondary Education Act · ESEA · history · NCLB · No Child Left Behind Act · policy · politics · practice · technology

I’m Regan Grandy, and I’ve worked as an evaluator for Spectrum Research Evaluation and Development for six years. My work is primarily evaluating U.S. Department of Education-funded grant projects with school districts across the nation.

Lessons Learned – Like some of you, I’ve found it difficult, at times, gaining access to extant data from school districts. Administrators often cite the Family Educational Rights and Privacy Act (FERPA) as the reason for not providing access to such data. While FERPA requires written consent be obtained before personally identifiable educational records can be released, I have learned that FERPA was recently amended to include exceptions that speak directly to educational evaluators of State or local education agencies.

Hot Tip – In December 2011, the U.S. Department of Education amended regulations governing FERPA. The changes include “several exceptions that permit the disclosure of personally identifiable information from education records without consent.” One exception is the audit or evaluation exception (34 CFR Part 99.35). Regarding this exception, the U.S. Department of Education states:

“The audit or evaluation exception allows for the disclosure of personally identifiable information from education records without consent to authorized representatives … of the State or local educational authorities (FERPA-permitted entities). Under this exception, personally identifiable information from education records must be used to audit or evaluate a Federal- or State-supported education program, or to enforce or comply with Federal legal requirements that relate to those education programs.” (FERPA Guidance for Reasonable Methods and Written Agreements)

The rationale for this FERPA amendment was provided as follows: “…State or local educational agencies must have the ability to disclose student data to evaluate the effectiveness of publicly-funded education programs … to ensure that our limited public resources are invested wisely.” (Dec 2011 – Revised FERPA Regulations: An Overview For SEAs and LEAs)

Hot Tip – If you are an educational evaluator, be sure to:

• know and follow the FERPA regulations (see 34 CFR Part 99).
• secure a quality agreement with the education agency, specific to FERPA (see Guidance).
• have a legitimate reason to access data.
• agree to not redisclose.
• access only data that is needed for the evaluation.
• have stewardship for the data you receive.
• secure data.
• properly destroy personally identifiable information when no longer needed.

Rad Resource – The Family Policy Compliance Office (FPCO) of the U.S. Department of Education is responsible for implementing the FERPA regulations, and they have a wealth of resources about it on their website. Also, you can view the entire FERPA law here. The statutes of most interest to educational evaluators will be 34 CFR Part 99.31 and 99.35.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

ammendments · data access · Department of Education · FERPA · government regulations · Schools · secure data · students

My name is Anne Vo and I am a doctoral student in the Graduate School of Education at UCLA. I served as a session scribe at Evaluation 2010 and attended session number 800, A Radically Different Approach to Evaluator Competencies. I chose this session because I was interested in learning about how evaluator competencies are conceptualized in non-US settings and the possible ways in which evaluation might differ in the New Zealand context in particular.

Lessons Learned: We learned from Jane Davidson, Kate McKegg, and Nan Wehipeihana (presenters on the panel) and Rodney Hopson and Michael Scriven (session discussants) about issues of cultural competence and how they factor into the reconceptualized framework for evaluator competencies in New Zealand.

• The presenters shared that evaluation practice in Aotearoa (Maori for New Zealand), specifically, is driven by notions of partnership, protection, and participation as stipulated in the Treaty of Waitangi. This document, signed in April 1840, established New Zealand as a British colony, but retained the Maori people’s rights to land ownership as well as their benefits and privileges as new British citizens.
• The presenters argued that the idea of values is at the heart of evaluation. For New Zealand evaluators, this means that the primary ingredients needed to competently conduct evaluation include an acute awareness and deep understanding of Maori history and culture.
• The New Zealand program evaluator competencies consist of five domains: 1) contextual analysis and engagement; 2) systematic evaluative inquiry; 3) evaluation project management and professional practice; 4) reflective practice and professional development; and 5) cultural competence – at the center and overlapping with the remaining four domains.
• Placing culture/cultural competence at the center of the New Zealand program evaluator competencies and evaluation practice marks a great departure from the “Essential Competencies for Program Evaluators” that Jean King and colleagues originally developed in the US-based context.
• For the panelists, next steps include: 1) articulating which of the competencies developed thus far for the New Zealand context are non-negotiable and 2) working through the tensions that have emerged as a result of centralizing the role of cultural competence in this framework for program evaluator competencies.

Rad Resources: The following are great supplemental reading and resources related to this session:

• Patricia Rogers and E. Jane Davidson’s Genuine Evaluation Blog (http://www.genuineevaluation.com ).
• Jean King and colleagues’ (2001) article, “Toward a taxonomy of essential evaluator competencies,” which appeared in volume 22, issue 2 of the American Journal of Evaluation

At AEA’s 2010 Annual Conference, session scribes took notes at over 30 sessions and we’ll be sharing their work throughout the year on aea365. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

cultural competency · Evaluation · evaluator competencies · scribing

My name is Anne Vo and I am a doctoral student in the Graduate School of Education at UCLA. I served as a session scribe at Evaluation 2010 and attended session number 273, Truth, Beauty, and Justice: Thirty Years Later. I chose this session because I was interested in learning about Ernie House’s influence on the way in which social justice is addressed in evaluation.

Lessons Learned: We heard a number of reflections rooted in four panelists’ efforts to understand House’s 1980 publication, Evaluating with Validity. Presenters in this session included Timothy Cash, Mel Mark, Laura Peck, and Beth Weitzman. Below are some take-away messages from their presentations.

• Tim uses the evidence-based policy and practice lens to interpret House’s idea of justice. From this perspective, it seems that one must be fair in order to be just. And, in today’s context, fairness/justice is best accomplished through three primary means: 1) conducting formal evaluation, which enables change; 2) coming to terms with the idea that change is incremental; and 3) having hope for moral progress.
• Mel focuses on the idea that justice is best served by obtaining truth; that is, the most correct and valid answer that one can derive through evaluation. And, that there are different forms of justice – those dealing with process and others pertaining to outcome. He also indicates that the crux of many challenges in present day evaluations are those dealing with how to balance values-based discussions about ends versus those of means and that attention to contextual variables is key when trying to reach reconciliation.
• In an effort to understand how House’s ideas of truth, beauty, and justice interacted with different validity types, Laura and her colleagues conducted a study where they applied his evaluation typology to 75 evaluation studies that were published in the American Journal of Evaluation between 1980 and 2010. Results of their coding process suggest that truth and beauty were more heavily emphasized in the literature compared to justice. While social justice might be most important to House, there is insufficient evidence to believe that it figures centrally in the literature.
• Beth and her colleagues suggest that attaining social justice means democratizing evaluation, but the process of democratization cannot be reduced to packaging evaluation into “toolkits” so that “program insiders” can conduct evaluations on their own and without any methodological backing. Rather, evaluators ought to situate themselves as facilitators, consultants, and guides in the ongoing evaluation and social decision-making processes so that more attention and respect can be paid to local knowledge and as a means to ensure justice and fairness.

Great Resource: Additional information on Ernie House’s (1980) book, Evaluating with Validity, can be found at: http://www.infoagepub.com/products/Evaluating-with-Validity

At AEA’s 2010 Annual Conference, session scribes took notes at over 30 sessions and we’ll be sharing their work throughout the winter on aea365. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

scribing · social justice · validity

We are Paul Brandon and Landry Fukunaga from the University of Hawai‘i at M?noa. Stakeholder involvement in program evaluation is one of the most enduring topics in the program evaluation literature, but empirical research on the topic has been summarized only within limited domains. We conducted a literature review of empirical research, examining 7,580 publications from January 1985 through May 2010 that we identified in systematic searches of 11 major electronic databases. After reviewing abstracts of the publications, we closely examined 43 peer-reviewed articles that (a) described stakeholder involvement in the conduct of or the study of program evaluation and (b) collected data on stakeholder involvement. Our process eliminated reflective narratives and other reports that had did not discuss systematic data collection on involvement, articles about theory, book reviews, and literature reviews.

Lessons Learned: Of the 43 articles:

• 14 (32%) were about evaluation in general, 11 (26%) took place in the domains of education or health, 6 (14%) were about social services, and 1 (2%) was about environmental planning.
• 31 (72%) were about evaluations that collected data on stakeholder involvement in actual evaluations. Of these, 23 were single-case studies and 8 were multiple-case studies. The remaining 12 (28%) were research studies or simulations involving stakeholders that did not take place within an evaluation context.
• The types of stakeholder groups most frequently studied were program staff and/or implementers of the program (18, or 42%), program administrators or board members (16, or 37%), and evaluators (12, or 28%). An average of 2.13 of types of stakeholder groups was studied.
• 16 (37%) of the studies collected data on fewer than 25 stakeholder participants, 8 (19%) collected data on 26–100 participants, and 12 (28%) collected data on more than 100. The remaining 7 (16%) did not report the number of stakeholder participants or were simulations.
• The methods used to study stakeholder involvement included surveys in 28 (65%) of the studies, interviews in 27 (63%), document reviews in 12 (28%), observations in 11 (26%), personal reflections in 5 (12%), focus groups in 4 (9%), and the results of informal discussion in 3 (7%).

The studies paid very little attention to how the research was conducted.

We suggest that (a) the empirical literature on stakeholder involvement in program evaluation is less substantial than many might believe, (b) the quality of the literature in stakeholder involvement in program evaluation is impossible to analyze because of a lack of detail about research methods, and (c) the dearth of studies provides additional evidence for the claims that funding for research on evaluation is seriously lacking.

Hot Tip: For more detail regarding this study, check out the slides from our presentation at Evaluation 2010.

The American Evaluation Association is celebrated Research on Evaluation (ROE) Week with our colleagues in the ROE AEA Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

case studies · research · stakeholders

Hello, my name is Jeehae Ahn. I’m a graduate student at the University of Illinois at Urbana-Champaign. My research interest and hence my dissertation centers around the theory-practice relationship in educational program evaluation. Following previous works on this domain, and using my own evaluation work in a specific program context as the research site, I’ve been exploring how the evaluator’s theoretical intentions and ideals relate to her key practice decisions and actions, and how they may show up in her ‘everyday’ evaluation activities, interactions and encounters within the given program context. Here are the three built-in processes I’ve been using (my “cool tricks”) to examine this theory-practice interaction as an integral part of my evaluation work in real time and on the ground.

Cool Tricks

1. Keep an evaluator’s journal: Since the beginning of the evaluation, I’ve been keeping a detailed, regular journal of what I am doing (e.g., specific decisions made; actions taken), why (e.g., theoretical justification; practical constraints, if any), as well as any reflective thoughts on what has happened (e.g., what works, what doesn’t, how and why; what matters the most, what remains constant and what changes in my evaluative thinking). The goal of keeping such a log is to think more thoughtfully and critically about my own evaluation principles and aspirations, about what I do in practice, and with what justifiable rationale in relation to my theoretical ideals and values.

2. Engage in ongoing communications with key stakeholders: Going beyond this kind of inward self-reflection, I have also endeavored to communicate with key program stakeholders on an ongoing basis through a series of progress updates and previews of upcoming evaluation activities. This is to keep them informed about the intent and progress of the evaluation, and also get their input and feedback in real time on whether and/or how my evaluative intentions, values, as well as their instantiations in practice, are meeting their evaluation priorities and information needs.

3. Consult with “critical friends”: Evaluation is rarely a solo venture, and so I’ve been reaching out for feedback from well, anywhere to everywhere. They have included committee members, known evaluation scholars, fellow graduate students and friends, who have generously shared their own thoughts, comments and insights on the meanings and values, challenges and effects of implementing and experimenting with particular evaluation ideas in practice.

While all these “tricks” are nothing new, I’d consider them still pretty “cool,” because they keep me informed and forewarned about what I do, how I do it, and why, and because I believe forewarned is forearmed, and it is always good to be forearmed rather than reactionary.

The American Evaluation Association is celebrating ROE Week with our colleagues in the ROE AEA Topical Interest Group. The contributions all this week to aea365 come from our ROE members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting ROE resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

Research on Evaluation · theory-practice relationship

Hello. My name is Anne Heberger Marino and I am an internal evaluator for the National Academies Keck Futures Initiative (NAKFI).  My colleagues, Dan Stokols and Shalini Misra, of the University of California, Irvine, School of Social Ecology work with the program as external evaluators.

Today I’d like to give you an example of how we were able to integrate social science theory to conduct research on evaluation methods. We hope that it will inspire you to think creatively about how you might try something similar.

Surveys are one way we evaluate the annual conferences NAKFI sponsors each year.  We survey conference participants immediately following the conference and again three months later.  While discussing how to improve response rates Dan mentioned research from social psychology which found that descriptive social norm-based messages (i.e. beliefs about what constitutes “typical behavior” in a particular situation ) are highly effective in promoting conservation and environmentally-protective behavior (Cialdini, 2003).  For example, a field experiment that investigated the effectiveness of normative messages in increasing towel reuse in a hotel found that a descriptive social norm-based message (“Join your fellow citizens in helping to save the environment”, followed by information about how the majority of hotel guests do reuse their towels when asked) yielded a significantly higher towel reuse rate compared to other types of persuasive messages (Goldstein, Cialdini, & Griskevicius, 2008).  We wondered if including such a message in the solicitation emails and reminder messages could improve the response rate for our online survey and devised a simple experiment to test the hypothesis that it would.

We found that participants receiving a message stating that they should complete the survey because others at the conference were also completing the survey (i.e. descriptive social norm message) were more likely to complete the survey as compared to those who were just asked to please complete the survey (i.e.  generic message without normative information).

Lesson Learned:

Practical significance is important in real world research on evaluation.  Shalini conducted statistical tests to confirm that our new text made a difference in the response rate but, the practical significance of our results was enough for me to justify changing the wording on our surveys.

Lesson Learned:

Research on evaluation need not be complicated, expensive, or time consuming, and can be informed by other disciplines.  Setting up this experiment was straightforward and took very little additional time.

Rad Resource:

Visit the AEA eLibrary for a synopsis of this study and other research on evaluation presented at the AEA 2010 conference.

References:

Cialdini, R. B. (2003). Crafting normative messages to protect the environment. Current Directions in Psychological Science, 12, 105-109.

Goldstein, N., Cialdini, R. B., & Griskevicius, V. (2008). A room with a viewpoint: Using norm-based appeals to motivate conservation behaviors in a hotel setting. Journal of Consumer Research, 35, 472-482.

The American Evaluation Association is celebrating ROE Week with our colleagues in the ROE AEA Topical Interest Group. The contributions all this week to aea365 come from our ROE members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting ROE resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

Internal Evaluation · ROE

Salutations! We are Dr. Chris L. S. Coryn, Professor of Evaluation, Measurement, and Research and Director of the Interdisciplinary Ph.D. in Evaluation (IDPE) at Western Michigan University (WMU) and staff at The Evaluation Center (EC), and Kristin A. Hobson, IDPE student and IDPE Doctoral Associate at WMU and the EC.

Interventions intended to ameliorate, eliminate, reduce, or prevent some persistent, problematic feature of the human condition have existed for millennia. Historically, such interventions often have been derived and enacted on the basis of conventional wisdom, political agendas, untested theoretical propositions, and, in some instances, from the results of low-quality, poorly-designed, poorly-executed, over-generalized studies. In a climate of increasingly scarce resources and greater demands for accountability, now, more than ever, policy makers and practice-based disciplines and professions are seeking high-quality, non-arbitrary, and defensible evidence for formulating, endorsing, and, occasionally, enforcing, best policies and practices. In the last few decades, randomized experiments, randomized controlled trials, and clinical trials (these terms often are used synonymously), universally have become the benchmark for supporting inferences and claims regarding the efficacy, effectiveness, and, to a lesser extent, generalizability, of such actions. Simultaneously, numerous evidence-based repositories, intended to guide and inform evidence-based policy making and practice decisions, including the Campbell Collaboration, Cochrane Collaboration, and What Works Clearinghouse, among many others, have become increasingly common. Nonetheless, little knowledge exists regarding the dependability of such methods.

Currently, we are engaged in planning and executing a study, and seeking funding through the Institute for Educational Sciences (IES), intended to address this question. The purpose of the study is to investigate the dependability and probable sources of error of the research review methods used by the Campbell Collaboration, Cochrane Collaboration, and What Works Clearinghouse for classifying primary studies (not meta-analyses) as ‘evidence-based’ or ‘not evidence-based’ via a generalizability (G) study. Specifically, we are seeking to determine (1) “How consistent (i.e., dependable) are the research review methods used by these three major research repositories regarding absolute decisions for classifying primary studies as evidence-based or not evidence-based?” and (2) “What are the identifiable sources of consistency and/or inconsistency that arise from these research review methods?” The study’s findings will provide information useful for improving existing research review methods and consistent classification of primary studies. Knowledge regarding the dependability of methods for promoting ‘evidence-based’ policies and practices and sources of error and (un)reliability in those decisions and estimated variance components from this study will be used for designing a decision (D) study. In a D-study, information from a G-study is used to evaluate the effectiveness of alternative designs for minimizing error and maximizing reliability (e.g., increasing the number of raters).

Hot Tip: Reliability and validity of measurement is frequently a secondary consideration (if a consideration at all) in evaluation practice, but is, arguably, of utmost importance (e.g., reliability is a necessary but not sufficient condition for validity), and, Cronbach’s ? is not actually a true reliability coefficient. We suggest that you look into this further to confirm or disconfirm our assertions!

Rad Resources:

Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.

Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: John Wiley.

Cronbach, L. J., Nageswari, R., & Gleser, G. C. (1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16(2), 137-163.

Davey, J. W., Gugiu, P. C., & Coryn, C. L. S. (2010). Quantitative methods for estimating the reliability of qualitative data. Journal of MultiDisciplinary Evaluation, 6(13), 140-162.

The American Evaluation Association is celebrating ROE Evaluation Week with our colleagues in the ROE AEA Topical Interest Group. The contributions all this week to aea365 come from our ROE members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting ROE resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

Cronbach's alpha · research review methods · ROE

Hi, I am Miriam Jacobson and I am a doctoral student in Evaluation at Claremont Graduate University. In previous work I have done with people with developmental and psychiatric disabilities, I have noted certain challenges when collecting data directly from some in these populations or when involving them in evaluation decision-making. This got me interested in how other evaluators have chosen to address similar concerns.

To explore this, I conducted a content analysis of articles in evaluation journals from 2000-2009 that describe studies of programs for primarily people with disabilities. Questions I addressed were: Did evaluators collect data directly from program recipients (rather than just get data about them from other sources)? If so, what methods did they use? At what points in the evaluation, if any, did the recipients have input or assist in the evaluation activities?

While most programs are not primarily for people with disabilities, many evaluators encounter program recipients with diverse cultures, languages, or preferred communication modalities.  I hope that the lessons learned from my research will be of use to evaluators working with these populations as well.

Lessons Learned:

1. In most cases, data is being collected directly from program recipients with disabilities.

Of the articles that describe data sources, 77% of them describe collecting data from program recipients with disabilities. Just 6% describe collecting data only from family members and not from program recipients.

2. A variety of strategies for how to collect data from people with disabilities are available for evaluators.

Interviews were the most common way data was collected from program recipients.  Focus groups, surveys, and observations were used as well.

3. Including on the evaluation team at least one or two representatives of diverse recipients is attainable in a variety of contexts.

Program recipients with disabilities were said to provide input and/or assist in the evaluation activities in 31% of the articles in which stakeholder participation was mentioned. Examples of such participation were found in all fields assessed (health, social service, education); in local, state, and national evaluations; and in both single-site and multi-site evaluations.

4. More efforts to include certain populations of program recipients more fully during the evaluation process are needed.

Participation occurred at least once in each evaluation stage; but more often, recipients contributed to the focusing of evaluation methods and interpretation of results and less often to the development of evaluation questions and defining of evaluation scope. A greater proportion of those with psychiatric disabilities participated than those with developmental disabilities. Adults participated more often than recipients under the age of 18.

The American Evaluation Association is celebrating Research on Evaluation Week with our colleagues in the ROE AEA Topical Interest Group. The contributions all this week to aea365 come from our ROE members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting ROE resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

Disabilities · ROE