My name is Virginia Dick and I am currently public service evaluation faculty at the Carl Vinson Institute of Government at the University of Georgia. Most of my work focuses on assisting state and local government agencies, and other university divisions, with evaluation of programs, policies and systems.

As part of my role I often find myself working with a wide range of individuals with different backgrounds, perspectives, purposes, and information assessment styles. It has been important to find ways to help different groups examine and understand relevant evaluation data using a wide range of mechanisms.

Most recently, I have begun working with our state child welfare agency to use GIS (Geographic Information Systems) methods to examine child welfare client characteristics and outcomes spatially through mapping. Often key stakeholders (community members, agency leadership, and social work students) have expressed new and interesting perspectives and interpretations of the data when it is portrayed via mapping rather than in traditional charts and tables.

Rad Resource: ESRI (http://www.esri.com/) often provides free training and educational opportunities to work with their mapping software and may be available through some universities.

There are many open source software options out there, some of which I am currently working with at the University of Georgia Information Technology Outreach Service to explore with my current project. A list of open source options is available at: http://gislounge.com/open-source-gis-applications/

Hot Tip: When working with a group reviewing the data and relationships between variables, start with a few layers and options on the map and slowly build and add additional components as the individuals start to become more comfortable talking about the relationships between the different variables.

Hot Tip: By looking at census tracts as units it allows groups to discuss the relationship between variables without having to dig down to the individual street address level which can become much more complicated when compiling the maps. Often analysis at the census tract level can be most beneficial to communities and government agencies rather than the individual street address level.

Hot Tip: Let the stakeholders generate the ideas and discussion among themselves to get the richest information about the perceived relationship between variables. This is particularly useful when looking at small units such as counties or smaller (with the mapping done at the census tract or block level).

Want to learn more about Virginia’s work using GIS? Come to the poster exhibition on Wednesday evening in San Antonio this November for AEA’s Annual Conference.

census, child welfare, Geographic information systems, GIS, software

My name is Michelle Jay and I am an Assistant Professor at the University of South Carolina. I am an independent evaluator and also an evaluation consultant with Evaluation, Assessment and Policy Connections (EvAP) in the School of Education at UNC-Chapel Hill. Currently I serve with Rita O’Sullivan as Directors of AEA’s Graduate Education Diversity Internship (GEDI) program.

Lessons Learned: A few years ago, EvAP served as the external evaluators for a federally-funded Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) state-wide grant housed at University of North Carolina (UNC) General Administration. Part of our work involved assisting project coordinators in 20 North Carolina counties to collect student-level data required for their Annual Performance Review reports as well as for program monitoring, assessment, and improvement. For various reasons, project coordinators experienced numerous difficulties in obtaining the necessary data from their Student Information Management Systems (SIMS) administrators at both the school and district levels. As collaborative evaluators, we viewed the SIMS administrators not only as “keepers of the keys” to the “data kingdom,” but also as potentially vested program stakeholders whose input and “buy-in” had not yet been sought.

Consequently, in an effort to “think outside the box,” the EvAP team seized an opportunity to help foster better relationships between our program coordinators and their SIMS administrators. We discovered that the administrators often attended an annual conference each year for school personnel. The EvAP team sought permission to attend the conference where we sponsored a boxed luncheon for the SIMS administrators. During the lunch, we provided them with an overview of the GEAR UP program and its goals, described our role as the evaluators, and explained in detail how they could contribute to the success of their districts’ program by providing the important data needed by their district’s program coordinator.

The effects of the luncheon were immediate. Program coordinators who had previously experienced difficulty getting data had it on their desks later that week. Over the course of the year, the quality and quantity of the data the EvAP team obtained from the coordinators increased dramatically. We were extremely pleased that the collaborative evaluation strategies that guided our work had served us well in an unanticipated fashion.

Hot Tip: The data needs of the programs we serve as evaluators can sometimes seem daunting. In this case, we learned that fixing “the problem” was less a data-related matter that it was a “marketing” issue. SIMS administrators, and other keepers-of-the-data, have multiple responsibilities and are under tremendous pressure to serve multiple constituencies. Sometimes, getting their support and cooperation are merely a matter of making sure they are aware of your particular program, the kinds of data you require, and the frequency of your needs. Oh, and to know that they are appreciated doesn’t hurt either.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Administration, Evaluation planning, project management

I’m Michael Matteson and I’m pursuing a graduate degree at the University of Wollongong (Australia).

Lessons Learned – What’s the Hawthorne effect? Why would it matter in Empowerment Evaluation? Most people will remember hearing of the “Hawthorne effect “based on a major study of organizational and environmental effects on productivity which gave confusing and contradictory results over a long period. Commentators suggested that any effects of the experiments were a result, not of the researcher’s manipulation of variables, but of staff feeling important because they were being observed. Stephen Draper gives a definition of the Hawthorne effect that I find useful:

An experimental effect in the direction expected but not for the reason expected; i.e. a significant positive effect that turns out to have no causal basis in the theoretical motivation for the intervention, but is apparently due to the effect on the participants of knowing themselves to be studied in conjunction with the outcomes measured (Draper, 2009).

Looking at this in terms of Empowerment Evaluation, I’ve come to feel that the evaluation team’s experience of the evaluator is a major part of their experience of the evaluation. This makes it a legitimate part of the process use of the evaluation, which is the mechanism expected to enable the empowerment result.

If so, it’s important to clarify what the effect is in each situation. This will depend on what the evaluator is doing, including the atmosphere they’re providing, and the extent to which the evaluator’s involvement is part of the positive reinforcement that team members’ experience, along with their own decision-making, in the course of the evaluation.

The Hawthorne effect can be expected to modify results outside of the conscious parameters of the investigation unless consciously allowed for. In the case of data gathering, I have decided to combine observation of classes run by staff who were part of the evaluation team with observation of classes that weren’t, hoping that any Hawthorne effect in the result-gathering will be canceled out by a parallel Hawthorne effect in my observation of the non-participants.

Hot Tip: Impression management, common in focus groups, and based on Irving Goffman’s work, may be relevant here.

Resource: Draper’s article explains the issues involved and the many uses of the Hawthorne experiment’s continuing legacy. Draper, S.W. (2009, Dec 23) The Hawthorne, Pygmalion, Placebo and other effects of expectation: some notes

Lessons Learned: While the most common explanation of the Hawthorne experience is some kind of “people felt important” effect, Paul Blumberg’s 1969 Industrial Democracy: The Sociology of Participation (Schocken) already argued, based on the original research, that the most likely factor was the level of the participants’ involvement in decision-making. This aspect has been consistently ignored.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

empowerment, Hawthorne Effect

Hi, my name is Susan Geier and I am a doctoral student at Purdue University studying research methods, measurement and evaluation. I employ a participatory evaluation approach with the GEMscholar project and have learned much from the Native American college students and the dedicated program staff.

Lessons Learned: I would like to share my three R’s for participatory evaluation:

1. Build Rapport: In addition to conducting formal interviews and assessments, I interacted informally with the students and mentors when time allowed, during meals and in between activities. I spent time learning about Native American history and culture from the project team and students.

2. Demonstrate Relevance: I discussed with the stakeholders and participants possible benefits of the evaluation process and their unique roles in the improvement and success of the program components. For example, when the students expressed interest in helping future GEMscholars, a peer-mentoring option was added to the program. Consequently, students began to see the evaluation process as a mechanism for sharing their experiences and suggestions instead of an outside critique of their lives and activities.

3. Maintain Responsiveness: I provided the stakeholders with information in a timely and accessible format. Often these were oral reports followed by brief documents outlining the changes discussed. We had conversations about those issues that could not be resolved in a timely matter and possible effects on the program. In turn, the project team made ongoing changes, adding components where needed and modifying those elements that were not serving the objectives of the program. Assessments were modified if needed and the process continued.

Hot Tip: Journaling is a useful technique to capture real time reactions to interventions. This is particularly important when working with groups who are being introduced to unfamiliar and/or uncomfortable experiences as part of an intervention. I worked closely with the researcher and program coordinator to develop pertinent guiding questions for the students’ and mentors’ daily reflection journals. This is also a good time to develop an analysis rubric if applicable. Journals can be hand written or online (I provide a link to an online journal using Qualtrics). The journal entries provide a project team with valuable insights about how the program elements are perceived by all involved.

If you want to learn more from Susan, check out the Poster Exhibition on the program for Evaluation 2010, November 10-13 in San Antonio.

College Access Programs, Native American, participatory

My name is Susan Wolfe and I am the owner of Susan Wolfe and Associates, LLC, a consulting firm that applies Community Psychology principles to strengthening organizations and communities. Applied research skills and program evaluation were core features of my Community Psychology graduate curriculum. Over the course of my evaluation career, I have become aware of how my discipline influences my approach, and I will share three ways here.

First, guiding concepts for community psychology include the use of interdisciplinary partnerships and approaches, and a participatory, empowering approach, informed by multiple perspectives.

HOT TIP: Incorporate the perspectives of multiple stakeholders into the evaluation design. Include stakeholders as active participants in all phases of the evaluation. That will facilitate buy-in for the results, broaden the utility of the findings, and help to identify potential unintended consequences for groups and individuals other than those targeted by the project.

Hot Tip: If possible, work with an interdisciplinary evaluation team. My collaborations with public health researchers, educators, social workers, and other disciplines have introduced me to alternative perspectives and methods, while enriching my content knowledge.

Second, one of Community Psychology’s guiding principles is attention to, and respect for, diversity among peoples and settings. If a program or evaluation design or content conflicts with the culture of the target audience, it may affect participation rates or receptivity and undermine the potential results.

Hot Tip: When you are evaluating a program include an assessment of the extent to which the program design, staff, and materials are culturally appropriate. Likewise, consider whether the questions you are asking are culturally relevant and your methods ensure that all participants have a voice in the evaluation.

And, third, Community Psychology takes an ecological perspective and recognizes the importance of looking across multiple levels and viewing programs within their context. To understand how well a program or policy is working, I have often found it helpful to look at contextual factors, such as culture, policy, physical environment, and history.

Hot Tip: When you design an evaluation, include assessing factors that might affect whether the context of the program or policy change will facilitate or inhibit its success, and the interactions between those factors and program components. Work within the context.

Rad Resource: For more information about Community Psychology and its principles, goals and guiding concepts go to www.scra27.org and The Community Toolbox.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

community psychology, context, cultural competency

Hello, I am Kimberly Kay Lopez, I have a community-based evaluation and research practice based in Houston, Texas. My work is concentrated in participatory evaluation methods used for the evaluation of youth programs and services.

Hot Tip: Using Photovoice in an Empowerment Evaluation: When working with youth, I have used the Empowerment Evaluation model many times. I have found that using Photovoice and journal writing within the Empowerment Evaluation model yields a variety of rich evaluation data. The goals of the Photovoice process enhance the evaluation process. The Photovoice process allows participants to document issues, engage in dialogue, and impact policy. I first integrated Photovoice with the Empowerment Evaluation model when evaluating the long-term impact of a multi-year teen pregnancy prevention program among urban Latino youth, I utilized the Photovoice process as a tool to “take stock” within the Empowerment Evaluation model. Youth were given cameras to capture the impact of the program. Youth were also given journals and guided writing assignments to express the impact that the teen pregnancy prevention program had on them. We also held traditional Empowerment Evaluation discussion groups. The youth and I analyzed visual data, journal data and discussion group data to develop the final evaluation report.

Rad Resource: PhotoVoice.org: PhotoVoice is an international organization that works with vulnerable populations. They offer several publications including a manual for using Photovoice. The methodology series gives further instruction on working with specific populations such as refugees. http://www.photovoice.org/shop/info/methodology-series

Rad Resource: PhotoVoice Manual: A comprehensive Photovoice Manual developed by Prairie Women’s Health Centre of Excellence. www.pwhce.ca/photovoice/pdf/Photovoice_Manual.pdf

Hot Tip: Using Journal Writing in Evaluations: I have found when working with middle school students, some students may be reluctant to participate in a discussion group. Offering youth writing opportunities allows those students uncomfortable in a group discussion a way to contribute to the evaluation process, on their terms.

Rad Resource: Guide on Engaging Youth: The National Clearinghouse for Families and Youth has a great guide on engaging youth in writing. www.ncfy.com/publications/pdf/lbd_write.pdf

There are all kinds of ways to get creative with data collection-digital storytelling, video cameras, blogs, tweets, text messages! Get creative! Use your imagination! Have fun!

Want to learn more from Kimberly? She’ll be on the program this November at Evaluation 2010, The American Evaluation Association’s Annual Conference in San Antonio, Texas.

community, journal, participatory, photovoice

As one from Brooklyn might say – “Yo! Seriously, Good Day!” For those who don’t subscribe to AEA’s listserv, Evaltalk, my name is Stanley Capela. I am the Vice President for Quality Management at HeartShare Human Services of New York, a 90 million dollar non-profit that provides children and family and developmental disabilities services for over 15,000 individuals in the New York City area.

Hot Tip: I am into a variety of techniques. One is Kaizen. I learned simple ideas lead to positive change. For instance, there are times when you are trying to figure out what is going wrong with a program and need a quick solution. As someone who tends to ask a lot of questions on Evaltalk, I often find coming up with a simple question often leads to possible solutions. One technique requires an index card and a question. Specifically, you convene the various stakeholders that are involved with the program. You distribute an index card and ask them to answer the following question: “If you can change one thing that would improve the program what would it be?” I have used this technique on several occasions. It provides a simple and useful way to identify an issue that would inform to how I would evaluate the program. If you have a quick thought or idea and want to share just email me at stan.capela@heartshare.org. At some point I will put the ideas together and share it. You never know what these various ideas might provide in strengthening evaluation.

Rad Resource: Want to learn more about the basics of the concepts behind Kaizen? Wikipedia has a nice explanation, available online at http://en.wikipedia.org/wiki/Kaizen.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

improvement, Kaizen

My name is Deshonna Collier-Goubil and I am a young scholar (newly minted PhD) who has had the wonderful opportunity to collaborate with seasoned evaluators and practitioners throughout my graduate education. Most recently I collaborated with a group of practitioners and evaluators to contribute a chapter in a newly released volume specifically written for young scholars. Our book chapter highlights the importance of collaborating with practitioners.

Lessons Learned: Collaboration has many benefits for both evaluators and practitioners. For the evaluator, receiving assistance from practitioners may help to decrease barriers to rich data, the quality of research can be improved with collaboration, and the overall research process can be improved by adding the input and assistance of frontline workers. For practitioners, evaluators can aide in obtaining research funding, clarifying research goals and expectations, and can highlight the need for institutional change or can put sound research behind an excelling program.

Research collaborations can be transformative for both the evaluator and the practitioner. One should approach a collaboration with deliberation, willing to both teach and learn. An array of cognitive, technical, and interpersonal skills are needed to develop and maintain effective collaboration. Having a firm grasp on communication, trust, honesty, respect, commitment, and flexibility can make or break a collaborative relationship.

Keep in mind, however, that just as benefits exist barriers may also arise during a collaborative effort. Evaluators should try to be as open and honest as possible with practitioners in initial negotiations to attempt to eliminate issues popping up in the future. Discussing division of labor, purpose of the collaboration, timelines for completion, how research will be conducted, data ownership, and how results will be communicated and disseminated for example can eliminate misunderstandings about these things in the future. Other barriers to be cognizant of are blurred roles, divergent perspectives, differences in degree of institutional support, competing and conflicting goals, and communicating difficult results. Despite experiencing barriers, evaluators should embrace adversity and persist in the collaborative relationship. Overcoming these barriers can strengthen the collaboration.

Overall, in a model collaboration, evaluators and practitioners develop shared goals, with consensus on a few key practice and research standards. The investment of time, resources, effort, flexibility, and the willingness to think outside of the box are required. Members of the collaboration learn to enter each other’s world and appreciate the others perspective. This is where the true learning begins.

Would you like to discuss evaluator-practitioner collaboration more with Deshonna and her colleagues? She’ll be contributing to a roundtable on the topic this November at Evaluation 2010, AEA’s Annual Conference.

collaboration, practitioners

I’m Jim Burdine and a few weeks ago I posted a “tip-a-day” (see post here) on presenting community health assessment data back to the community. This is part two of that discussion. Last time I focused on “what to present” this posting focuses on “how to present . . . with results.”

There are two goals for any assessment presentation: (1) reporting the data, and (2) identifying next steps for improving health status. I’ve used a format that is effective in a variety of settings. This assumes, as discussed in my last posting, that the assessment has been a “participatory process.”

Hot Tip: Hold a “regional health summit.” Folks invited include those who have participated in various aspects of the assessment as well as others representing a broad cross section of community sectors. My experience has largely been in medium to smaller communities (500K to 10K population) but participation is roughly the same – about 150-250 folks is a good turnout. The event is split into two parts and can be accomplished in one day or two. The two parts are a presentation of the data, followed by a workshop. Ideally the data presentation is limited to an hour and the actual presentation, or much of it, is by a community member. Avoid the temptation to have more than a couple of speakers – “inclusiveness” is a nice idea but counterproductive in this context. You can invite community members to do “welcomes” and other remarks.

Following the data presentation is another opportunity to develop for community buy-in. In the workshop, we ask folks to select among the “findings” of the assessment (e.g., “access to mental health,” “childhood obesity,” “rural transportation”) and sign up for those topics. Have rooms/tables setup for each group. A convener appointed for each group opens the discussion and is responsible for encouraging folks to commit to participating in an ad hoc task group on this topic that that would agree to meet 2-3 times over the next 2-3 months. At the end of that time (3 months) the larger group is reconvened to hear reports from the various ad hoc task groups and then to endorse a regional health improvement strategy developed at that session (based on the group reports). During the 2-3 month work period, each team reviews pertinent findings and identifies “best practices” of interventions related to that issue. The group then identifies one or two priority activities that incorporate those best practices as adapted to your community.

By following this model, what might have been just a data presentation yields a comprehensive regional strategy with multiple sub-parts that is endorsed by the entire community (at least as represented by the individuals and agencies participating in the overall process).

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

communication, participatory

My name is Aimee Sickels and I am the Principal Evaluator and owner of Custom Evaluation Services, an independent evaluation consulting business. I am also currently working on a Ph.D. in Social Work at the University of South Carolina. My helpful hint relates to the use of logic models. I am an empowerment evaluator and believe that our role as evaluators is not only to evaluate the programs, but to build the evaluation capacity in the organizations we evaluate. So, how can the logic model help?

Hot Tip: The 2 minute logic model. I will often look over my client’s programming before I am going on a site visit or a monitoring meeting. I will select a single program item, it may be a single goal or a single activity, and I will bring this to my client as the 2 minute logic model. Before we get started with our planned meeting I present my single item to them, for example one of my client’s who serve vulnerable persons has a goal of actually increasing involvement of the client in the program design; having the client’s participate more in their own service design. I start this on the input column and walk the client through that single item across the output column and ultimately the outcome column. I then say spend the rest of today and the rest of this week working on this single item. Because it is a single item and a single line it is easy to follow and quick to see and understand. Put the 2 minute logic model write up at each staff person’s desk and ask them to consider it with each client they see. I can guarantee that they will see an improvement on that line of programming for that week! Try the 2 minute logic model today!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

empowerment, logic models