I’m Taj Carson, the President of Carson Research Consulting (CRC) in Baltimore, MD. CRC is a research and evaluation consulting firm and we’ve seen first-hand how neighborhood-level data is increasingly being used for tasks such as identifying community conditions and trends or measuring population-level outcomes in research and evaluation.

Recently, I was in a meeting where a group of human service providers met to discuss where to locate a program for pregnant and parenting teen mothers in Baltimore City. They realized that what they really needed to know was— which communities have the highest teen birth rates in Baltimore?  A colleague went into a storage room and retrieved a huge, styrofoam-backed map from 2008, showing teen birth rates across the city, so the group could decide where to locate the program. While I was impressed with the fact that they valued data enough to use it in the early planning stages, and that they were actually able to remember where they had stashed a map from 2008, I was encouraged to get the word out about the DataMind as an interactive mapping tool that would allow them to see this information without rummaging around in a closet.

Lessons Learned:

• Creative, forward thinking program planners know when to bring data to the table to make decisions.
• While there is a wealth of available data on our communities, we are short on ways to visualize that information on the spot and identify patterns across different parts of the city.
• Visualizing several data sources together in an interactive map allows for a more complex understanding of data, even if it is on paper.

Rad Resource: The Baltimore DataMind is an interactive mapping tool that allows users to visualize data for Baltimore City neighborhoods to promote collaboration, advocacy, informed decisions, and effective policy making. Users can compare data across neighborhoods, create a community profile of a neighborhood, and combine data indicators and community resources and assets in one map. In the above-described situation, the Baltimore DataMind could have provided the information this group needed by looking at the “Teen Birth Rate” data in the Children and Family Health Indicators section of the map. They also could have mapped out the location of schools and community health centers in those neighborhoods. Hot Tips:

• You Don’t Have to be a GIS Expert: Neighborhood-level data could have easily been pulled up in theBaltimore Datamind “widget” in an easy-to-use interface developed by Policy Map.
• Data can be shared: The maps can be printed and then shared with neighborhood stakeholders, funders and community residents.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

community mapping · GIS · neighborhood mapping

My name is Emily Spence-Almaguer and I am an Associate Professor of Behavioral and Community Health at the University of North Texas Health Science Center. I spend most of my professional time serving as an independent evaluator for community initiatives and conducting assessment studies. I am a social worker by training and have found that the conversational skills used in Solution-Focused Therapy have great application in the realm of evaluation and community assessment.

Hot Tips: My favorite ways to use solution-focused dialogues are in:

• Focus group and individual interviews because they help generate rich qualitative data and great ideas for continuous program improvements.
• Evaluation planning meetings because they help stakeholders articulate a wide range of potential outcomes and describe how those outcomes might be observed (i.e., measured).
• Meetings where stakeholders are being debriefed around disappointing evaluation results. The nature of solution-focused dialogues avoids finger-pointing and helps drive forward momentum.

Hot Tips:

• It’s all about the questions!! Solution-focused dialogues are driven by questions that promote deep reflection and critical thinking.
• Context: Use questions that help situate people’s minds in a particular context and use details in your question that will encourage an individual to imagine him or herself in that moment. Here’s an example that I use with consumers at a program trying to help lift individuals and families out of poverty:
  • I want you to take a moment and imagine that you just learned that the Bass [local philanthropist] family recently donated $100,000 to the United Way for this project. They want you to help them figure out how to best spend the money. What is the first thing you would advise them to do? What would you advise them to do next?
  • Expertise: I love the way that Gaiswinker and Roessler referred to this as the “expertise of not-knowing”. In solution-focused dialogues the words of questions and tone of delivery are carefully crafted to amplify the assumption that the stakeholders have exceptional knowledge, skills and capacities.

Rad Resource: For an introduction to solution focused concepts, I like Coert Visser’s Doing What Works Blog.

Download from the AEA Public eLibrary to View the Poster in Full Size!

Rad Resource: I presented on Solution-Focused dialogues in evaluation at AEA’s Evaluation 2012 conference. You can download my poster and resources list from the AEA public eLibrary here.

Lessons Learned: A direct question, such as “What would you recommend to improve this program?” often fails to generate detailed or meaningful responses. In focus groups with program consumers, I find that this question is interpreted as “what is wrong with the program?” and may lead to comments in defense of the program staff members (see my 2012 AEA poster for an example of this from my data).

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Susan Wolfe and I am the owner of Susan Wolfe and Associates, LLC, a consulting firm that applies Community Psychology principles to strengthening organizations and communities.

I serve as the local evaluator for four programs that are funded by Federal grants for a five year cycle. Each program was required to set objective for each of the five years of funding for a pre-set group of performance indicators. At the end of each year they report their performance objective and their actual performance.

This year I facilitated staff retreats for two of the programs. I compared the objectives and actual performance for each indicator. I tagged the indicators where the program fell short of the objective by a good margin as “red light,” the indicators where the program just missed, or will miss next year as “yellow light,” and the ones where the program met the objective and will do so next year as “green light.” During the staff retreat I reviewed the program’s logic model with staff to show how their activities connected with each indicator, and then we went through the red and yellow light indicators and discussed the challenges associated with meeting the objectives. We then celebrated the success with the green light indicators — which were the largest in number.

For the last two hours of the retreat, I facilitated staff discussions whereby staff developed specific strategies, complete with timelines and clear responsibilities to improve performance for red and green light indicators. Staff at both programs were fully engaged and fully participated in the process, with the end result being a clear plan that was realistic and included accountability.

Lessons Learned: Program staff is interested in learning about the bigger picture of their program, and how their activities relate to performance reporting. If they are empowered with information and an opportunity to develop strategies, they will engage and build their capacity to make their program successful.

Hot Tip: Review logic models with all program staff to show them how their role fits into the larger picture. It helps them to become invested in maintaining accurate data records and in the evaluation process.

Lessons Learned: Staff may need more information to determine which direction to take to improve their performance. Evaluators need to be ready to facilitate that process by gathering or analyzing more data and allowing staff input into the evaluation process

Rad Resource: For more information about Community Psychology, its principles and values and see how they guide our work, see the website for the Society for Community Research and Action.

This is a bonus post from the week sponsored by the American Evaluation Association Community Psychology Topical Interest Group. The contributions from December 9-14 all came from CP TIG members, be sure to return to aea365 and take a look! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

SCRA

Hello! My name is Ann Price, the owner of Community Evaluation Solutions, Inc., an evaluation firm that focuses on program development and capacity-building evaluation. I always tell people that evaluation is what I DO, but a community psychologist is what I AM. In this 365 I would like to discuss community psychology’s values and how I incorporate them in my evaluation practice.

Hot Tip - Community Psychology Values: Community psychologists view individuals within the context of their communities and the larger society. One of my long-standing clients is an after-school program in rural Georgia. The community’s on-time graduation is very low, about 56%. We could simply assume that the school is failing. But an important contextual issue is some local families do not value education – parents and grandparents give some students the message that they have done well with a 9^th grade education and encourage their children to drop out. In order to improve the graduation rate, we need to address this cultural issue. Community psychology also values social change through research and action. We worked with a local nonprofit to update Georgia’s juvenile justice code and another initiative to protect heir property owners from sale of their property without their knowledge. Many of these owners are minority owners unaware of their rights. Our evaluation work values participatory action research in which we view share control with stakeholders and together generate knowledge and experience for collective use. In our practice we start every evaluation by identifying key stakeholders and forming an internal evaluation team that works with us throughout the life of the evaluation from design through data interpretation and communication of results.

Rad Resource – The Society for Community Research and Action (SCRA): SCRA, Division 27 of the American Psychological Association, is an international organization devoted to advancing theory, research, and social action.

Clipped from: www.scra27.org (share this clip)

Hot Tip - Collaborate with other Community Psychologists Within AEA: Last year the Community Psychology Topical Interest Group (TIG) became the newest AEA TIG. Twenty-five community psychologists came together for its first meeting and quickly got to work organizing (we enjoy community organizing). This past year the TIG launched its website and hosted 16 sessions at Evaluation 2012 in Minneapolis.

Rad Resource: Visit the AEA Community Psychology TIG website at http://comm.eval.org/CommunityPsychology/Home/.

Hot Tip: Calling all Community Psychologists in AEA and evaluators in SCRA. This year at our business meeting we discussed how we can foster cross-collaboration between our two professional homes. Within AEA we discussed growing the TIG, developing more sessions at AEA, sponsoring a field trip during AEA to a community program and perhaps hosting a charity drive at AEA. Within SCRA we discussed promoting AEA as a place where community psychologists might find professional development and training opportunities. Join us next year at the Community Psychology TIG meeting and stay tuned for more opportunities to collaborate!

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

No tags

Greetings State-side colleagues! My name is Alison Baxter and I have the privilege of being an internal evaluator with Pathways to Education Lawrence Heights in Toronto, a holistic after school program for high school students who live in a low income community. My experience has been that engaging youth in evaluation activities is not always an easy task. As a Community Psychologist, it is important to me to ensure that the youth we work with have a voice and input into what happens in the program and decisions that impact on them. Getting the students excited about sharing their perspectives can, at times, be challenging.

Hot Tip: Providing incentives to the students to complete surveys has made a huge difference to me as an evaluator and generated excitement among the students about evaluation research. Collecting data has become a lot easier. Free lunches, ten dollar gift cards and draws for iPods have all been offered and well received by the youth. Since the youth come from families on low incomes and may face barriers to employment, the practice of providing incentives has become like a small community economic development initiative.

Cool Trick: Keep your incentives youth friendly and your data collection tools short and you will boost your response rates.

Lessons Learned: Implementing an incentive initiative takes time for an evaluator. There is administrative work involved. Make sure you are able to carve time out in your busy schedule to complete the work that is required.

I faced an ethical dilemma when I made the decision to provide incentives. The students, living on low incomes, may have participated in the research and the accompanying risks involved more readily due to the cash incentive than they might have otherwise. On some level, it could be seen that I was taking advantage of their vulnerable situation for my own need to acquire data on youth. I resolved my ethical dilemma by ensuring that the data would only be shared with service providers within Pathways to Education who have the power to act on the findings and use the research to make a positive difference in the lives of the youth. Not only were youth being compensated for completing surveys, but their perspectives will assist us to plan and implement programs and initiatives that respond to their needs and interests. Furthermore, I reassured students that their participation was voluntary. The students who were not interested in participating did not hesitate to refuse the request.

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

incentives

Hi, my name is Brian Hoessler and I’m a consultant who works with community-based organizations to build their capacity to make change. Previously, I worked in the non-profit sector in Ontario, most recently as a researcher at Kingston Community Health Centres for two projects including a support program for high school students called Pathways to Education. This program was based on a successful social innovation in Toronto’s Regent Park, and as part of the national Pathways initiative we followed their core four-pillar model of support: however, we soon realized that our students in small-town Kingston, and the challenges they faced, were very different from those in multicultural downtown Toronto.

To help our site adapt the national model to local realities, I drew on Developmental Evaluation (DE), a new approach well-suited to projects facing uncertainty and complexity. Instead of assessing adherence to pre-determined plans (formative) or outcomes (summative), DE helps infuse evaluative thinking into the process of program development by framing assumptions and rapidly providing data: in the case of Pathways to Education, I presented findings to the team and helped facilitate conversations around what it meant and how we should respond.

Hot Tips:

• Relationships are key. Building trust is important for any evaluative activity, but more so in DE given the integral role of the evaluator in the team. The evaluator has to feel comfortable in bringing forward feedback (positive and negative) and challenging assumptions; likewise, team members have to trust the evaluator and understand that uncomfortable questions are being raised to help improve the program rather than to pass judgment. Explicitly defining your role is important.

• Build trust by being there. Participate in meetings, help out with programming and events, chat with colleagues during coffee breaks, share interesting finding – demonstrate that you share your colleague’s aim of helping the program to meet its goals.

• Draw on your team members’ front-line experiences. For example, our initial student cohorts were progressing well overall through grades 9 and 10 – findings which didn’t match the experiences at other sites. Speaking with one of our support workers, I learned that there would be a “perfect storm” of pressures and challenges for our students starting in grade 11, an insight I confirmed through historical cohort data from the school boards. Armed with this information, the team could then discuss how to best support those at risk.

Rad Resources:

The McConnell Foundation’s website has two free resources in pdf on DE:

• A Developmental Evaluation Primer
• DE 201: A Practitioner’s Guide to Developmental Evaluation
• A great book, covering theory, methods, and case examples is Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use by Michael Quinn Patton

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

developmental evaluation

I’m Tara Gregory, Research and Evaluation Coordinator for Wichita State University’s Center for Community Support and Research (CCSR). CCSR works with non-profit, community and faith-based organizations across Kansas and was originally supported through the Community Psychology graduate program at Wichita State University. Many of our staff members, myself included, are graduates of this program so we’ve maintained a strong community psychology orientation in our principles and practices. Given the principle of meeting people where they are, we often use forms of storytelling to help organizations develop logic models

We use the following techniques to facilitate creative discussion while still attending to the elements in a traditional logic model. These processes encourage participation by multiple staff, administrators and stakeholders and can use the organization’s vision or impact statement as the “happily ever after.”

Hot tip – Script writing: We ask participants to think of their program and it’s outcomes in terms of a movie trilogy. In small groups, they create scripts for each part of the trilogy then report out on the significant scenes (much like they would if they were describing a movie they’d just seen). These scenes inform the elements of their logic model, which we typically help them to complete later, and could be focused on the individual or other contexts (e.g., community). We specifically ask them to think of Part 1 as the story of what people experience while involved in the program; Part 2 picks up at a later date (the specific timeframe depends on the program) and reflects the progression of outcomes; and Part 3 represents the transition to “happily ever after.”

The specific questions we ask participants to address in their scripts are:

• Who are the characters, settings or contexts?

• What do they experience/what happens to them?

• What actions do they take as a result?

Hot tip – Pictorial timeline: Using a similar process to script writing, we ask participants to envision one of their clients, then to draw the activities and resulting behaviors or conditions that occur at various points along a timeline. This approach offers a visual path toward “happily ever after.”

Lessons learned:

• Participants are less likely to get bogged down in concerns about the “right” way to fill out a logic model and are better able to identify outcomes, including those that are unintended or less positive, than with traditional methods.

• Whereas completing the typical logic model matrix can be intimidating for some, these processes tend to be energizing and fun

• These techniques work particularly well with organizations that are innovative and are open to playfulness and experimentation.

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

No tags

Hi aea365ers! Susan Staggs here. I’m an Associate Professor of Psychology at the University of Wisconsin – Stout, coordinator of an online graduate certificate program in Evaluation Studies, and incoming Program Director of our Master’s in Applied Psychology program, which includes a concentration in Evaluation Research. I write today on behalf of the American Evaluation Association’s Community Psychology Topical Interest Group (TIG). One of the principles we believe in is evaluation should be an “active collaboration among researchers, practitioners, and community members… undertaken to serve those community members directly concerned, and guided by their needs and preferences, as well as by their active participation.” With that in mind, I’ve chosen to highlight some wonderful community-oriented resources and organizations that have the potential to enrich your evaluations of community health initiatives by enhancing their explicit focus on community collaboration.

Rad Resource – The University of Kansas’ Community Toolbox: This amazing site offers practical, specific guidance for promoting community health and evaluating community health initiatives. There’s a whole slew of How-To Guides on topics such as stakeholder engagement and community assessment. There’s a Troubleshooting area for help solving problems such as dealing with disappointing evaluation results and unintended intervention effects. The Promising Approaches section highlights the latest evidence-based intervention research, while a Connect With Others area lets you ask questions of intervention experts. Specific guidance on Participatory Evaluation is available here, as is an Evaluation Model for Community Initiatives. A treasure trove, loved and respected by community evaluation practitioners.

Clipped from: ctb.ku.edu (share this clip)

Rad Resource – Community-Campus Partnerships for Health: Want to make sure you’re serving the community’s evaluation needs? This community advocacy organization, focused on social justice and equity in partnerships between communities and universities, can help you with that. They host conferences and training opportunities designed to promote equalization of power in collaborative work between community members and academics. A peer mentorship program for community representatives involved in work with academic partners is available, as is guidance on community-centered research ethics. There are many other valuable and practical resources available on the site as well; this is a fabulous site from a wonderful organization whose purpose is strongly rooted in community psychology values. They maintain a very active listserve on ethics in community-based participatory research.

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

No tags

My name is Sadish Dhakal and I work as the Monitoring and Evaluation Officer at the dZi foundation. We currently work in two districts in Nepal, south of Mt. Everest. Today, I am going to share something I learned during the evaluation of an income generation program in these two areas.

The income generation program has been providing agriculture training to farming communities. The model assumed that by introducing new crops that yield higher prices in the market, and by helping farmers produce them efficiently with new technology, we would be able to help them generate higher income. Accordingly, we have trained the community members on vegetable production, expecting health benefits as a potential side effect that could result from an increased vegetable consumption in the area.

The program so far has been successful. The communities are enthusiastic about vegetables. However, we learned that financial gain and health benefits are not the only positive outcomes of this project. Community members have reported that, before the program, only those who belonged to the higher class ate vegetables regularly. Now, because of the program, vegetables are consumed by people of all social strata. While it is not illogical to think that the increased income resulting from the program would reduce the class gap, we did not expect the increase in vegetable consumption to affect social class directly.

This program is only one example of many. It teaches us to adopt open ended evaluation practices, rather than evaluating only with the aim of measuring set indicators. Often, the communities know what changes are important to them, and know what changes are taking place. In fact, it is important to get the community involved in designing the indicators or even in setting programmatic goals.

Lessons Learned: Communities know best about their needs and have the most incentive to stay informed to the changes around them. Therefore, getting the communities involved in setting goals as well as measuring change is the key to successful monitoring and evaluation. Rigid evaluation methodologies which do not provide enough room for the community’s direct input can miss unexpected or unintended outcomes.

I am happy to have this opportunity to share with you the small, but exciting developments happening in the mountains of Nepal. If I can be of any help, you can contact me at sadish@dzifoundation.org. You can also visit http://www.dzifoundation.org/ to learn more about the dZi Foundation.

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

No tags

Welcome to the CP&E TIG sponsored week of tips and rad resources.

I am David Fetterman, past-president of the American Evaluation Association and co-chair of the Collaborative, Participatory and Empowerment Evaluation TIG. I have 25 years of experience at Stanford University. I am the President & CEO of Fetterman & Associates, an international evaluation consulting firm. Concurrently I am a member of the faculty in the School of Education at the University of Arkansas at Pine Bluff and co-Director of the Arkansas Evaluation Center.

Hot Tip – Let go. In empowerment evaluation it is important to learn how to let go without abdicating your responsibility. Encourage folks to take the lead but follow closely behind to make sure things get done, to help when help is needed, to facilitate and guide but not control.

Rad Resources

Ignite Lecture. Ignite lectures are 5 minutes and 20 slides. It is a wonderful way to help you refine your message and communicate more effectively. Example

 

Online Debate. Debating your views with critical friends is another way to crystallize your thinking and your message. See: Fetterman, Patton, and Scriven Debate: Promises and Pitfalls of Empowerment Evaluation

Wikipedia: Empowerment Evaluation

Web Page: Empowerment Evaluation Web Page

Blog: Empowerment Evaluation Blog

Articles:

Chinman, Hunter, Ebener, Paddock, Stillman, Imm, Wandersman’s article: The Getting to Outcomes Demonstration and Evaluation: An Illustration of the Prevention Support System in the American Journal of Psychology, (2008). [41:206-224]. Abstract

José M. Díaz-Puente, Adolfo Cazorla Montero, and Ignacio de los Ríos Carmenado’s article: Empowering communities through evaluation: some lessons from rural Spain. Community Development Journal [2009, 44(1): 53-67]. Abstract

Fetterman, Deitz, and Gesundheit’s medical education article: Empowerment evaluation: a collaborative approach to evaluating and transforming a medical school curriculum in Academic Medicine [85(5):813-820]. It is a case example of how empowerment evaluation was applied to the Stanford University School of Medicine. Full Article

Fetterman, and Wandersman’s popular article: “Empowerment evaluation: yesterday, today, and tomorrow” in the American Journal of Evaluation [28(2):179-198]. This article “is designed to enhance conceptual clarity, provide greater methodological specificity, and highlight empowerment evaluation’s commitment to accountability and producing outcomes.” Abstract

Miller, and Lennie’s article: Empowerment Evaluation: A Practical Method for Evaluating a National School Breakfast Program in the Evaluation Journal of Australasia, 2005. [5(2), 18-26]. Full Article

Recent Chapter:

Fetterman’s chapter – Empowerment Evaluation: Learning to Think Like an Evaluator. In Alkin, M. (ed.) Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences, 2012.

Forthcoming Book:

Fetterman, D.M.’s Empowerment Evaluation in the Digital Villages: Hewlett Packard’s $15 Million Race Toward Social Justice. Stanford University Press.

The American Evaluation Association is celebrating CPE week with our colleagues in the Collaborative, Participatory, and Empowerment TIG. The contributions all this week to aea365 come from our CPE TIG Colleagues. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

No tags