We are Alexandra Hill and Diane Hirshberg, and we are part of the Center for Alaska Education Policy Research at the University of Alaska Anchorage.  The evaluation part of our work ranges from tiny projects – just a few hours spent helping someone design their own internal evaluation – to rigorous and formal evaluations of large projects.

In Alaska, we often face the challenge of conducting evaluations with very small numbers of participants in small, remote communities. Even in Anchorage, our largest city, there are only 300,000 residents. We also work with very diverse populations, both in our urban and rural communities. Much of our evaluation work is on federal grants, which need to both meet federal requirements for rigor and power, and be culturally responsive across many settings.

Lesson Learned: Using mixed-methods approaches allows us to both 1) create a more culturally responsive evaluation; and 2) provide useful evaluation information despite small “sample” sizes. Quantitative analyses often have less statistical power in our small samples than in larger studies, but we don’t simply want to accept lower levels of statistical significance, or report ‘no effect’ when low statistical power is unavoidable.

Rather, we start with a logic model to ensure we’ve fully explored pathways through which the intervention being evaluated might work, and those through which it might not work as well.  This allows us to structure our qualitative data collection to explore and examine the evidence for both sets of pathways.  Then we can triangulate with quantitative results to provide our clients with a better sense of how their interventions are working.

At the same time, the qualitative side of our evaluation lets us lets us build in measures that are responsive to local cultures, include and respect local expertise, and (when we’re lucky) build bridges between western academic analyses and indigenous knowledge. Most important, it allows us to employ different and more appropriate ways of gathering and sharing information across indigenous and other diverse communities. 

Rad Resource: For those of you at universities or other large institutions that can purchase access to it we recommend SAGE Research Methods.  This online resource provides access to full text versions of most SAGE research publications, including handbooks of research, encyclopedias, dictionaries, journals, and ALL the Little Green Books and Little Blue Books.

Rad Resource: Another Sage-sponsored resource is Methodspace, an online network for researchers. Sign-up is free, and Methodspace posts selected journal articles, book chapters and other resources, as well as hosting online discussions and blogs about different research methods.

Rad Resource: For developing logic models, we recommend the W.K. Kellogg Foundation Logic Model Development Guide.

(Share Clip)

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · culturally responsive evaluation · logic models · mixed methods

I’m Cheryl Poth and I am an assistant professor at the Centre for Applied Studies in Measurement and Evaluation in the department of Educational Psychology, Faculty of Education at the University of Alberta in Edmonton, Canada. My area of research is focused on how developmental evaluators build evaluation capacity within their organizations. My use of mixed methods is pragmatically-driven, that is, I use it when the research/evaluation question(s) require the integration of both qualitative themes and quantitative measures to generate a more comprehensive understanding. Most recently, my work within research teams has provided the impetus for research and writing about the role of a mixed methods practitioner within such teams.

Lessons Learned:

• Develop and teach courses. In 2010, I developed (and offered) a doctoral mixed methods research (MMR) course in response to the demand from graduate students for opportunities to gain skills within MMR. The course was oversubscribed and at the end of the term we formed a mixed methods reading group, which continues to provide support as students are working their way through their research process. I am fortunate to be able to offer this course again this winter and already it is full!
• Offer workshops. To help build MMR capacity, I have offered workshops in a variety of locations, most recently at the 9^th Annual Qualitative Research Summer Intensive held in Chapel Hill, NC in late summer and at the 13^th Thinking Qualitatively Workshop Series offered by the International Institute for Qualitative Methodology held in Edmonton, AB in early summer. These workshops remind me of the reality for many researchers that their graduate programs required completion of an advanced research methods course that was either qualitatively- or quantitatively-focused and of the need to build a community of MM researchers and that the community can exist locally or using technology can exist globally! It has been a pleasure to watch as new and experienced researchers begin to learn about MMR designs and integration procedures.
• Join a community. One of the places where I have begun to find my community MM researchers was through a group currently working on forming the International Association of Mixed Methods, at the International Mixed Methods conference, and the mixed methods researchers on Methodspace.

Hot Tip:

• Stay tuned, the International Association of Mixed Methods is in the works and one of their goals is to build MMR capacity by offering learning opportunities using a variety of methods.

Rad Resource:

• Join the AEA Mixed methods TIG (Donna Mertens and Lennise Baptiste are co-chairs)
• Join the mixed methods researcher group at Methodspace
• Link to information about the 2012 Qualitative Research Summer Intensive
• Link to the International Institute for Qualitative Methodology
• Link to the International Mixed Methods conference

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · learning · Professional Development · Workshops

Hi, I’m Katherine Hay. I’ve spent the last 15 years in India working on development, research, and evaluation.

Lessons Learned:

• A mantra I use all the time is:  ‘there is no gender neutral policy, program, or evaluation.’  If I hear one of these things described as ‘gender neutral’ I start to probe.  Usually when an intervention is called ‘gender neutral,’ it is actually gender blind.

• South Asia, my home and the place I work, has the worst gender inequities in the world.

• Evaluation can reinforce or reflect social inequities – or it can challenge them. I want to play a part in challenging them. To do that, evaluation has to help us figure out what shows promise in shifting inequities and what does not.  This is what draws me to feminist evaluation.

• Mainstream development, and by extension mainstream evaluation, grapples with mainstream questions.   This has resulted in designs, approaches, and tools which are not particularly well suited to understanding inequities.   Feminist analysis brings inequity to the foreground.

Hot Tips:

• I’m often asked, ‘But how do you do feminist evaluation?’  There are no shortcuts.  The answer is, ‘by applying feminist principles at different stages in an evaluation.’   For example:

1. At the start of the evaluation feminist analysis can be used to ask, ‘whose questions are these?’ and, ‘whose questions are being excluded?’
2. A rigorous feminist evaluation uses the mix of methods that matches the questions.  But some designs factor out the perspectives of marginalized groups.  Feminist evaluation designs include them.
3. At the judgment stage, feminist evaluations recognize that there are different and often competing definitions of success in development interventions. Feminist analysis brings these differences to the surface for debate.
4. At the use stage, feminist analysis brings recognition that particular pathways may be strategic, blocked, or risky. A feminist approach also brings responsibility to take responsible action on findings.

• Get Involved. Peer support has been invaluable to my evaluation practice.  I’m part of a group in South Asia trying to strengthen our work through feminist analysis.  We share our designs, instruments, processes and challenges.  We are critical but supportive.  Being part of this group reminds me why evaluation matters. Try to find a group of peers to challenge and inspire you.  If you want to share resources or get in touch, we have a Feminist Evaluation website.

Rad Resources :

• Donna Podems paper.  describes feminist evaluation as being a way of thinking about evaluation.

• The My M&E website has free books including Evaluation for Equitable Development Results that has a section on feminist evaluation.  The site also has webinars like the one Ratna Sudarshan and I did on Feminist Evaluation.

The American Evaluation Association is celebrating the Mixed Methods Evaluation and Feminist Issues TIGs (FIE/MME) Week. The contributions all week come from FIE/MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

social inequities

My name is Elizabeth (AKA: Bessa) Whitmore. Now a retired Professor from the School of Social Work, Carleton University, Ottawa, Canada, I have been a member of the Feminist TIG since its inception. The following entry draws on a chapter I am writing entitled “Researcher/evaluator roles and social justice” in a forthcoming Handbook on Feminist Evaluation (edited by Denise Seigart, Sharon Brisolara and Sumitra SenGupta).

Hot Tips:

• There are a range of roles played by a feminist evaluator, including facilitator, educator, collaborator, technical expert/methodologist, and activist/advocate. Not everyone can do everything equally well, so self-knowledge and confidence in one’s strengths (and limitations) is essential. The personal characteristics, experience and preferences of the evaluator will dictate which role(s) she/he best plays. It is critical to recognize that what role the evaluator plays and how, is intimately tied to her/his own worldview, history, and biography. There is no objectivity; we need to be aware that we are deeply grounded in our own location and life experience.
• Good “people skills” are essential when engaging stakeholders in the process. These include active listening, cultural sensitivity, non-verbal communication, motivating participants, coordinating relationships, encouraging interactions, supporting others’ ideas, and an ability to reflect critically on one’s own reactions and behavior.
• Having fun: We should not dismiss the importance of fun in this work. “If I can’t dance, I don’t want to be part of your revolution” said Emma Goldman back in the 1930s. Long hours without some laughter tend to burn people out, or they just drop out.

Cool Tricks: Here are some questions one might ask when planning and implementing a feminist evaluation:

• In what ways are women (men, bisexual and transgendered people, etc.) treated differently within the program, and how do their experiences and outcomes differ? In what ways do class, race, and gender combine to expand or contract possibilities for participants?
• Are both women and men being consulted about objectives and activities? Which women, and which men? Has the potential for community resistance to women’s empowerment activities or organizational resistance to female managers been assessed?
• Did the project have any unexpected (positive or negative) social and gender equity outcomes?

Lessons Learned:

• A feminist lens enhances validity in all evaluation approaches. For example, an experimental design pays attention to the sample distribution among men and women, considers gender related factors in the questions asked, and in data analysis. A utilization focused evaluation attends to the gender (and other) distribution (in decision-making). Social justice approaches (such as empowerment, participatory, collaborative, transformative, etc.); consider the equality and quality of gender participation.
• Get involved: A good place to discuss these and other issues is the Feminist TIG.

The American Evaluation Association is celebrating the Mixed Methods Evaluation and Feminist Issues TIGs (FIE/MME) Week. The contributions all week come from FIE/MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

activist · advocate · facilitation

Greetings, I am Denise Seigart, Associate Dean for Nursing at Stvenson University.  Like a great novel, a great feminist evaluation creates the conditions for learning and change, particularly for the benefit of women. In 2008-2009 I implemented a feminist evaluation to study school-based health care in the United States (U.S.), Canada, and Australia. In the process of implementing a qualitative study of school-based health care, I utilized a feminist lens and feminist methods, including reflexivity, interviews focused on active listening and the experiences of the interviewees, collaborative examination of the data with interested stakeholders and other feminists and non-feminists, and diverse dissemination of the results for the purpose of promoting dialog, health care reform, and social justice for children. It was my intent to create conditions for a critical feminist exploration of school health care for children across the three countries, to share this information, and ultimately, promote community learning, action and change.

Lessons Learned:

• Feminist evaluation is like other evaluation. It is concerned with measuring the effectiveness of programs, judging merit or worth, and examining data to promote change. The difference between feminist approaches and other evaluation models generally lies in the increased attention paid to gender issues, the needs of women, and the promotion of transformative change.
• Feminist evaluation is interested in promoting social justice for women, but includes other oppressed groups as well. Attention is paid not only to gender but to race, class, sexual orientation, and abilities. In my study, I interviewed 73 school nurses, parents and administrators in Canada, Australia, and the U.S. regarding the presence and quality of school health care in their countries. I paid particular attention to emerging themes that indicated problems with racism, sexism, and classism, and asked additional questions as these emerged. For example, it was apparent that certain groups had more difficulty accessing health care in schools (aborigines, children with special needs) and that depending on the school district, services could vary widely. Teachers (largely women) were often asked to act as health care providers to save school districts money, and nurse practitioners (largely women) experienced difficulty gaining access and approval to provide care in schools.
• To implement a feminist evaluation, think carefully about the questions you want to ask, the methods you want to use, and the setting. Some facilities may bar access to evaluators who declare themselves as feminist, so the language you use should be carefully chosen. Be sure to involve other feminists and non-feminists, so when planning your design or analyzing data, you can check for misinterpretations or “what would a feminist see?”

Rad Resources:

• Handbook of Feminist Research: Theory and Praxis
• Feminist Evaluation: Explorations and Experiences: New Directions for Evaluation

The American Evaluation Association is celebrating the Mixed Methods Evaluation and Feminist Issues TIGs (FIE/MME) Week. The contributions all week come from FIE/MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I am Alessandra Galiè, a PhD Candidate at Wageningen University in the Netherlands. From 2006 to 2011 I collaborated with a Participatory Plant Breeding programme coordinated at the International Centre for Agricultural Research in the Dry Areas (ICARDA) to assess the impact of the programme on the empowerment of the newly involved women farmers in Syria. The findings helped to understand how empowerment as a process can take place, and were useful to make the programme’s strategies more gender-sensitive. I chose to work with a small number (Small-N) of respondents (12 women) and a mixture of qualitative methods to provide and in-depth understanding of changes in empowerment as perceived by the women themselves and their community.

Lessons Learned

• Small-N research is valuable. Small-N in-depth research is often criticised for its limited external validity. However, it was an extremely valuable methodology to explore a field of research that is relatively new with the aim of providing an understanding of complex social processes, of formulating new questions and  identifying new issues for further exploration.
• Systematic evaluation should include empowerment. Empowerment is an often cited impact of development projects but rarely the focus of systematic evaluation. Assessing changes in empowerment required an approach that was specific to the context and intervention under analysis and that was relevant to the respondents and their specific circumstances. This revealed different positionalities of women in the empowerment process and the inappropriateness of blue print solutions to the ‘empowerment of women’.
• Measure gender-based implications. An analysis of the impact of a breeding programme on the empowerment of women showed that ‘technical interventions’ have gender-based implications for both technology effectiveness and equity of development concerns.

Resources

• Bent Flyvbjerg’s article in Qualitative Inquiry: Five Misunderstandings about Case-Study Research. On the value of Small-N research.
• Alessandra Galiè’s forthcoming article: ‘Empowering women farmers: the case of participatory plant breeding in ten Syrian households’. Frontiers: a Journal of Women Studies. To read more on the evaluation of empowerment mentioned above.

The American Evaluation Association is celebrating the Mixed Methods Evaluation and Feminist Issues TIGs (FIE/MME) Week. The contributions all week come from FIE/MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

agriculture · empowerment · international

Hello, I am Donna Podems, founder and director of OtherWISE: Research and Evaluation, a small monitoring and evaluation firm in Cape Town, South Africa.  We work with a wide range of international and local donors who fund a wide variety of technical interventions in areas such as environment, education, health, community development and human rights.

We encourage evaluation use through choosing and mixing different evaluation approaches that will bring credible and useful evaluation findings. Feminist evaluation is one of the approaches that I often draw upon, and this often surprises many of my colleagues.

Feminist evaluation can be useful– even for non-feminist evaluators.

Hot Tip:

• You do not need to be a feminist to use feminist evaluation. It is important to understand that not all feminist evaluators (or evaluation theorists) agree with me. Over 18 years of conducting evaluation in more than 25 countries, I have had the privilege of working with many talented evaluators, most of whom were not feminists. In more than 15 different evaluations in Africa and Asia, my team members agreed to incorporate various elements of a feminist approach that resulted in useful evaluation processes and findings.

Lessons learned: Three lessons I have learned about addressing the question I hear the most, “How do you apply feminist evaluation if you are not a feminist?”

• Be knowledgeable about what feminist evaluation is, and is not. Many people I work with have a strong reaction to feminist evaluation and yet few can explain what the approach entails. Demonstrate how elements of the approach could enable a credible and useful evaluation.

• Remove the label. Having two words that often elicit strong reactions together in one phrase is a challenge. Remove the label and explain the approach.
• Adapt as needed. In my experience, feminist evaluation often provides a useful complement to other evaluation approaches.

Rad Resource:

• Feminist Evaluation: Explorations and Experiences: New Directions for Evaluation. Denise Seigart (Editor), Sharon Brisolara (Editor) New Directions for Evaluation, Volume 2002, Issue 96, Winter 2002.

The American Evaluation Association is celebrating the Mixed Methods Evaluation and Feminist Issues TIGs (FIE/MME) Week. The contributions all week come from FIE/MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

community development · education · Environment · health · human rights

Hello!  We are Donna Mertens and Mika Yamashita, Chair and Program Chair of Mixed Methods Evaluation TIG.  This week, we offer five posts written by Feminist Issues in Evaluation TIG members. Why is the Mixed Methods TIG co-hosting this week with the Feminist Issues in Evaluation TIG?  Because this week’s posts touch upon issues associated with theoretical and paradigmatic choices and their implications to evaluation design and methods.   “Mixed methods” may give you an impression that it is all about techniques of using quantitative and qualitative methods in one evaluation study.  It is one area of discussion. Mixed Methods Evaluation TIG views that mixing can occur at the level of inquiry purpose, philosophical assumptions, methodological design and/or specific data gathering technique. So, Mixed Methods Evaluation TIG sees our discussion can include the relationship between paradigmatic and theoretical lenses and methods.   The authors of this week’s posts will walk us through how the feminist lens informed inquiry purposes, choice of evaluation design, and methods.

Highlighted for FIE/MME week are:

• Authors will explicitly talk about their worldviews, such as:
  •  what they believe in (they believe social justice),
  • issues they are concerned about (they are concerned with gender issues and marginalized populations),
  • and how their worldviews influenced evaluation questions they asked and their choice of methods.

Lesson Learned: Your evaluation lens is important. The feminist lens helps evaluators to see conflicting views of what the problem is.  With this understanding, evaluators consciously make decisions about what and whose evaluation questions to be asked.   The feminist lens also helps evaluators to see the diversity among a disadvantaged population.

Hot Tips:

• Be reflective. You will also notice that evaluators are reflective of how they and their evaluations may be perceived by other people. They provide lessons learned from establishing relationships with evaluation participants, evaluation commissioner, and audience.
• Match your analysis to your evaluation design. Evaluators decided data collection and analysis methods by considering evaluation questions, purpose of evaluation, and settings in which data collection took place. How to include perspectives of marginalized population is an important consideration for deciding methods.

Rad Resources:

• Hood and Cassaro’s article in New Directions for Evaluation: Feminist evaluation and the inclusion of difference.
• Podems’ article: Feminist evaluation and gender approaches: There’s a difference?
• Merten’s book: Transformative Research and Evaluation
• If you are new to mixed methods evaluation, we recommend Greene, Caracelli, and Graham’s seminal 1989 article: Toward a Conceptual Framework for Mixed-Methods Evaluation.
• The slides from the June 11, 2012, Washington Evaluators’ brown bag session: Where are we in mixed method evaluation? The slides include paradigms mixed methods evaluation often drawn upon.

The American Evaluation Association is celebrating the Mixed Methods Evaluation and Feminist Issues TIGs (FIE/MME) Week. The contributions all week come from FIE/MME members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We’re Allan Porowski from ICF International and Heather Clawson from Communities In Schools (CIS). We completed a five-year, comprehensive, mixed-method evaluation of CIS, which featured a several study components – including three student-level randomized controlled trials; a school-level quasi-experimental study; eight case studies; a natural variation study to identify what factors distinguished the most successful CIS sites from others; and a benchmarking study to identify what lessons CIS could draw from other youth-serving organizations.  We learned a lot over the years, and wanted to share a few big takeaways with you about conducting evaluations on interventions for at-risk youth.

Lessons Learned:

• Sometimes, you have to catch falling knives: We found that the students coming into CIS were targeted for services because they were on the strongest downward trajectories on a number of factors (e.g., academics, behavior, family issues, attendance, etc.). There’s an old adage in stock market trading that you should “never catch a falling knife” – but that’s what CIS and other dropout prevention programs do every day. This has implications for how you evaluate the relationship between dosage and outcomes. A negative relationship between dosage and outcomes doesn’t necessarily indicate that services aren’t working – it can actually be an indication that services are going to where they are needed the most.
• Look for the “Nike Swoosh”: The general pattern of outcomes among CIS students looked like Nike’s “swoosh” logo: There was an initial downward slide followed by a longer, more protracted period of improvement. Reversing that initial downward slide takes time, and this pattern is worth investigating if you’re evaluating programs for at-risk youth.
• As the prescient rock band Guns n’ Roses put it, “All we need is just a little patience”: Needless to say, it takes a long time to turn a child’s life around. So many evaluations of at-risk students don’t have a long enough time horizon to show improvements, which may in part explain why we see such low effect sizes in dropout prevention research relative to other fields of study.

Rad Resources:

• Executive Summaryof Communities In School’s Five-year National Evaluation
  • Communities In Schools has great ideas and resources for dealing with at-risk youth. CIS surrounds students with a community of support, empowering them to stay in school and achieve in life. Through a school-based coordinator, CIS connects students and their families to critical community resources, tailored to local needs. Working in nearly 2,700 schools, in the most challenged communities in 25 states and the District of Columbia, CIS serves nearly 1.26 million young people and their families every year.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

at-risk · dropout · vulnerable populations

Hello! We’re Allan Porowski from ICF International and Heather Clawson from Communities In Schools (CIS). We completed a five-year, comprehensive, mixed-method evaluation of CIS, which featured  several study components – including three student-level randomized controlled trials; a school-level quasi-experimental study; eight case studies; a natural variation study to identify what factors distinguished the most successful CIS sites from others; and a benchmarking study to identify what lessons CIS could draw from other youth-serving organizations.  We learned a lot about mixed-method evaluations over the course of this study, and wanted to share a few of those lessons with you.

Lessons Learned:

• Complex research questions require complex methods. Disconnects exists between research and practice because the fundamental research question in an impact evaluation (i.e., Does the intervention work?) provides little practical utility for practitioners in their daily work. CIS leadership not only wanted to know whether CIS worked, but also how it worked, why it worked, and in what situations it worked so they could engage in evidence-informed decision making. These more nuanced research questions required a mixed methods approach. Moreover, CIS field staff already believed in what they were doing – they wanted to know how to be more effective. Mixed methods approaches are therefore a key prerequisite to capture the nuance and the process evaluation findings desired by practitioners.
• Practitioners are an ideal source of information for determining how much “evaluation capital” you have. CIS serves nearly 1.3 million youth in 25 states, which opens up the likelihood that different affiliates may be employing different language, processes, and even philosophies about best practice. In working with such a widespread network of affiliates, we saw the need to convene an “Implementation Task Force” of practitioners to help us set parameters around the evaluation. This group met monthly providing incredibly helpful in (a) identifying language commonly used by CIS sites nationwide to include in our surveys, (b) reviewing surveys and ensuring that they were capturing what was “really happening” in CIS schools, and (c) identifying how much “evaluation capital” we had at our disposal (e.g., how long surveys could take before they posed too much burden).
• The most important message you can convey: “We’re not doing this evaluation to you; we’re doing this evaluation with you.” Although it was incumbent upon us as evaluators to be dispassionate observers, that did not preclude us from engaging the field. Evaluation – and especially mixed-methods evaluation – requires the development of relationships to acquire data, provide assistance, build evaluation capacity, and message findings. As evaluators, we share the desire of practitioners to learn what works. By including practitioners in our Implementation Task Force and our Network Evaluation Advisory Committee, we were able to ensure that we were learning together and that we were working toward a common goal: to make the evaluation’s results useful for CIS staff working directly with students.

Resources:

• Executive Summary of CIS’s Five-Year National Evaluation
• Communities In Schools surrounds students with a community of support, empowering them to stay in school and achieve in life. Through a school-based coordinator, CIS connects students and their families to critical community resources, tailored to local needs. Working in nearly 2,700 schools, in the most challenged communities in 25 states and the District of Columbia, Communities In Schools serves nearly 1.26 million young people and their families every year.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

benchmarking · case studies · complex · impact · quasi-experimental · randomized · research to practice