Greetings State-side colleagues! My name is Alison Baxter and I have the privilege of being an internal evaluator with Pathways to Education Lawrence Heights in Toronto, a holistic after school program for high school students who live in a low income community. My experience has been that engaging youth in evaluation activities is not always an easy task. As a Community Psychologist, it is important to me to ensure that the youth we work with have a voice and input into what happens in the program and decisions that impact on them. Getting the students excited about sharing their perspectives can, at times, be challenging.

Hot Tip: Providing incentives to the students to complete surveys has made a huge difference to me as an evaluator and generated excitement among the students about evaluation research. Collecting data has become a lot easier. Free lunches, ten dollar gift cards and draws for iPods have all been offered and well received by the youth. Since the youth come from families on low incomes and may face barriers to employment, the practice of providing incentives has become like a small community economic development initiative.

Cool Trick: Keep your incentives youth friendly and your data collection tools short and you will boost your response rates.

Lessons Learned: Implementing an incentive initiative takes time for an evaluator. There is administrative work involved. Make sure you are able to carve time out in your busy schedule to complete the work that is required.

I faced an ethical dilemma when I made the decision to provide incentives. The students, living on low incomes, may have participated in the research and the accompanying risks involved more readily due to the cash incentive than they might have otherwise. On some level, it could be seen that I was taking advantage of their vulnerable situation for my own need to acquire data on youth. I resolved my ethical dilemma by ensuring that the data would only be shared with service providers within Pathways to Education who have the power to act on the findings and use the research to make a positive difference in the lives of the youth. Not only were youth being compensated for completing surveys, but their perspectives will assist us to plan and implement programs and initiatives that respond to their needs and interests. Furthermore, I reassured students that their participation was voluntary. The students who were not interested in participating did not hesitate to refuse the request.

We’re celebrating all this week with our colleagues in the American Evaluation Association Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

incentives

I’m Marybeth Neal, I am a cultural anthropologist that uses ethnographic methods to engage stakeholders in creating their own ongoing systems of evaluation.

This summer I had the absolutely delightful experience working with a brilliant group of teens and young adults who train both youth and adult members of social change organizations on how to create effective youth-adult partnerships.

They wanted to create a pre-test and then two post-tests (given immediately following the training and then six months later) to accompany their Youth Adult Partnership training.  The purpose of these tests were multiple:

• To offer constructive feedback to the trainees and their organizations.
• To document and share the results of the training with external audiences (funders and potential clients), and
• For internal use (to improve the training and to customize it for particular audiences).

Together we created a simple logic model for the project, which came to be known as “the Grid.” The Grid functioned as a “sticky wall” where stakeholders could contribute their own ideas (using post-its).

Hot Tip:

We put the Grid on a wall of the office by the kitchen area – a popular spot!  It listed, in separate columns:

• What are the desired learning outcomes of our training?
• What are the training activities that teach these outcomes?
• How do we know and how can we measure the extent to which participants and their organizations are learning what is being taught?

In addition, in the upper right hand corner, we asked:

Why do we want to create assessment tools?

In its simplicity, this visual representation of the project helped everyone to understand the complexity of the project and to contribute to it.  It was an efficient way to define key concepts used in the training, to make sure that each learning objective had a corresponding activity that taught the objective, and that we did not forget why we were doing this work.   Most importantly, it offered a way for the trainers’ to share their wisdom about how to recognize successful learning both for the individual trainees and for their organizations.

Rad Resource:  We used the Descriptive Question Matrix in The Ethnographic Interview by James P. Spradley to help us generate survey questions that could be used to measure changes in knowledge, skills and dispositions.  Spradley pioneered the teaching of ethnographic methods to undergraduates and this book and its companion book. Participant Observation, are simple, accessible and profound.

mapping · youth-led evaluation

I am Alice Hausman, a professor of Public Health at Temple University. I have been working as a community based participatory research (CBPR) evaluator of youth violence prevention initiatives in urban environments for many years.

Lesson Learned:

• Involve the Community in Identifying Measures and Data. As part of the participatory evaluation planning process, I always ask community participants to define their vision of program success. But I take it one step further by looking for data that might actually measure these community-defined outcomes. The process of working with community partners to identify measures and data has been as rewarding as just asking what success would look like.

Hot Tips

• Use available data sources in partnership with the community. One community collaborative I worked with identified available data sets and survey opportunities they could use to evaluate their programs.  In another project, a randomized community trial of a multi-level violence prevention program, we found that the standardized psychometric tools being used by the evaluation trial could be used to measure community-defined constructs, such as “showing kids love”, after reconfiguring the items through a participatory review process.
• Remind yourself of the value of community-evaluator partnerships.  In our case, the indicator itself was insightful about the community’s perception of social and relationship factors related to preventing youth violence. But the actual process of discussing the instruments and constructs was rewarding for all parties. The academic researchers learned more about the lived experience of their community partners who learned more about measurement development and psychometric research.
• Don’t hesitate to collaboratively develop new measures Another important outcome of the process of identifying existing data to measure community ideas was the realization that new measures and data might be needed to accurately capture the constructs defined by the community. While our community partners were initially concerned with the burden of adding new questionnaires, their views shifted somewhat after seeing that the benefit of being able to actually measure community defined constructs would outweigh the risks of more surveys.

Rad Resource:

• Translating Community Specified Indicators of Program Success into Measurable Outcomes.

Get Involved:  I would love to hear from others who have done work in this area. We can compare notes on indicators and measures and possibly find ways to make measuring community-defined outcomes as routine as measuring outcomes defined by funders.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

participatory · youth

Hello, I am Rashell Bowerman, Educational Consultant for the Michigan Department of Education. I formally worked as a doctoral associate for the National Secondary Transition Technical Assistance Center (NSTTAC) and continue to participate in NSTTAC’s Evaluating Professional Development Community of Practice. NSTTAC is committed to data-based decision making and views evaluation as a tool for improving our work. In evaluating professional development, NSTTAC adopted a five level evaluation model looking across the continuum from participant reactions to student outcomes. The focus of this article is on Level One: participant reactions.

Lesson Learned:

• Good evaluations don’t have to be complicated. This is especially true when we look at collecting Level One data. Educators often do not pay much attention to evaluating their professional development activities. Many even consider evaluation to be too costly and time-consuming. Level one data is the simplest to gather and analyze, it is the most common form of professional development evaluation.

Terrific Tip:

• Looking at participant reactions gets at the initial thoughts and satisfaction regarding their experience. The following questions are typically addressed on a Level One evaluation:

• Did they like it?
• Was their time well spent?
•  Did the material make sense?
• Will it be useful
• Was leader knowledgeable & helpful?
• Were the refreshments fresh & tasty?
• Was the room the right temp.?
• Were the chairs comfortable?

• By listening to the raw reaction of participants at the conclusion of the professional development activity, it can provide valuable information which can help improve the design and delivery of future programs or activities. Level on evaluation data can be gathered in several ways:

• Questionnaires administered at the end of the session
• Focus groups
• Interviews
• Personal learning logs

Radical Resources:

Here is an example of a section of an evaluation form used to collect level one data.

Example of Guskey Level One

Other Rad Resources and Links:

• The Evaluation Exchange: A Conversation with Thomas Guskey

• Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.

• North Central Regional Educational Laboratory (NCREL)

• NSTTAC Evaluation Toolkit

The American Evaluation Association is celebrating the Evaluating Professional Development Community of Practice (PDCoP) Week. The contributions all week come from PDCoP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

Guskey · participant reactions · Professional Development

Hello! We’re Allan Porowski from ICF International and Heather Clawson from Communities In Schools (CIS). We completed a five-year, comprehensive, mixed-method evaluation of CIS, which featured a several study components – including three student-level randomized controlled trials; a school-level quasi-experimental study; eight case studies; a natural variation study to identify what factors distinguished the most successful CIS sites from others; and a benchmarking study to identify what lessons CIS could draw from other youth-serving organizations.  We learned a lot over the years, and wanted to share a few big takeaways with you about conducting evaluations on interventions for at-risk youth.

Lessons Learned:

• Sometimes, you have to catch falling knives: We found that the students coming into CIS were targeted for services because they were on the strongest downward trajectories on a number of factors (e.g., academics, behavior, family issues, attendance, etc.). There’s an old adage in stock market trading that you should “never catch a falling knife” – but that’s what CIS and other dropout prevention programs do every day. This has implications for how you evaluate the relationship between dosage and outcomes. A negative relationship between dosage and outcomes doesn’t necessarily indicate that services aren’t working – it can actually be an indication that services are going to where they are needed the most.
• Look for the “Nike Swoosh”: The general pattern of outcomes among CIS students looked like Nike’s “swoosh” logo: There was an initial downward slide followed by a longer, more protracted period of improvement. Reversing that initial downward slide takes time, and this pattern is worth investigating if you’re evaluating programs for at-risk youth.
• As the prescient rock band Guns n’ Roses put it, “All we need is just a little patience”: Needless to say, it takes a long time to turn a child’s life around. So many evaluations of at-risk students don’t have a long enough time horizon to show improvements, which may in part explain why we see such low effect sizes in dropout prevention research relative to other fields of study.

Rad Resources:

• Executive Summaryof Communities In School’s Five-year National Evaluation
  • Communities In Schools has great ideas and resources for dealing with at-risk youth. CIS surrounds students with a community of support, empowering them to stay in school and achieve in life. Through a school-based coordinator, CIS connects students and their families to critical community resources, tailored to local needs. Working in nearly 2,700 schools, in the most challenged communities in 25 states and the District of Columbia, CIS serves nearly 1.26 million young people and their families every year.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

at-risk · dropout · vulnerable populations

We are Hilary Loeb and Kelly Bay of the Research and Evaluation Department at the College Success Foundation.  Many of our scholarship and support programs host events in which we collect data from students and educators.  As internal evaluators, we often rely on colleagues to collect and enter survey data from these groups. The results are used for staff learning internally and external reporting.  To help evaluators increase survey relevance, decrease demands on respondents’ time, and ultimately boost data quality and response rates, below are tips on instrument design and data collection.

Lessons Learned:

Look for ways to make surveys easier for staff to administer up front and more useful to stakeholders at the back end. The key is keeping the main focus on your programs while building support for data collection and analysis efforts.

Hot Tips

Survey Design:

• Ensure that survey content is relevant: Meet with the entire program team and start with the question, “What do we want to learn about our program?” before discussing what’s needed for grant-reporting requirements.
• Draft a survey using previously tested questions:  You don’t have to reinvent the wheel. By using previously tested survey questions from existing “banks” of items, you can save time and often improve the quality of the data collected (see Rad Resources).
• Pilot test surveys with your program team and other stakeholders. This exercise never fails to elicit important feedback and takes only a modest amount of time. It’s amazing what fresh eyes can find! Where possible, use trainings and even Board meetings as opportunities to pilot and discuss surveys.

Survey Data Collection:

• Be strategic about paper versus online surveys: When event participants can’t readily access computers, paper surveys may help increase response rates.  Online surveys are more appropriate when participants are able and willing to access technology.
• Designate sufficient time and staff to collect survey data: Ensure that there is a specific time slot dedicated for survey completion. It should be near to but not at the very end of the event.  We suggest providing a script to help staff describe the survey’s purpose and value.
• Consider using scanning software for paper surveys: Scanning software automates data entry by reading the optical marks on paper survey forms, which can reduce errors and save time.  Before purchasing, it’s best to test. We piloted a Free Demo of Remark Office OMR, to confirm that this was the right software for our organization.

Rad Resources:

A Bing search of survey item banks yields over 60 million results.  Our favorites in the education and youth development field include:  Ansell-Casey Life Skills Assessments,  the Youth Behavioral Risk Surveillance System and  National Center for Educational Statistics resources .

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

data collection tool · free tools · resources

Hi, I am Laura Gagliardone. For about seven years I have collaborated with multilateral, bilateral and non-profit organizations geographically targeted to Africa, South Asia and the Middle East and focused in the following areas: research, communication, knowledge management, monitoring and evaluation, program planning, development and coordination.

RELEVANCE: I like thinking of evaluation as a way to improve programs and a chance to adjust activities accordingly to the lessons learned. A quality evaluation depends on the relevance of the quantitative and qualitative data collected and on using communication methods to raise stakeholders’ trust, interest, participation and contribution.

HOT TIP: Evaluation and communication are two disciplines which complement each other. For an evaluator, gathering and analyzing quality data is as important as transmitting the findings. Pay close attention to the communication flow among participants in order to create a positive setting where everybody is contributing to the success of the activity and gaining ownership and accountability.

HOT TIP: Position yourself as a networking hub of the program. Focus your interpersonal and communication skills in a way that installs trust, confidence and builds participant commitment. To minimize evaluation anxiety and to get a good overview of the context, an evaluator needs to be perceived as a community insider who becomes integrated and participates in daily activities.

LESSONS LEARNED: In one of the lowest income areas of Nairobi, Kenya, I evaluated an educational program for children. I used communication strategies to enhance the community’s participation and dialogue. For example, before flying to the country, I read about the cultural, economic and socio-political context and started interacting with people in the field using information and communication technologies. Once in the field, I began to empathize with locals by participating in their daily activities. While undertaking the evaluation I preferred face-to-face interviews especially with children; used open-ended questions and answers; and visited families in their houses, and brought little gifts and greeted them in ‘Kiswahili’. People in developing countries often live and work at a different speed than in Western societies, therefore they appreciate when they are given time. I have learned to appreciate peoples’ strong sense of community and capacity to interact and share. I have also noticed that individualism and independence are rarely exercised. Where possible, evaluators are advised to seek one-on-one interactions that elicit personal perspectives. As participatory techniques that elicit visioning and action planning by local community members, consider mapping exercises, supplying articles or short essays, writing brief notes, creating designs and even sharing recipes as analytical tools that inform the initiative. Enabling a participatory setting facilitates acceptance of the evaluation findings and increases ownership of the activities.

The American Evaluation Association is celebrating Washington Evaluators (WE) Affiliate Week with our colleagues in the WE AEA Affiliate. The contributions all this week to aea365 come from our WE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Africa · bilateral · communication · Middle East · multilateral

I am Lori Peterson an Assistant Professor at the University of Northern Colorado. Recently, I worked on a project conducting a series of focus groups with high school students with disabilities. I wanted to share tips and lessons learned from this experience.

Hot Tips:

• Feed them and they will come! High school students love food. We conducted the focus groups near lunch time and gave each group free pizza. We had almost 100% attendance.
• Know the school calendar. We had an issue conducting focus groups on senior skip day!
• Conduct focus groups on location. If you can work with the school and conduct focus groups on site, this eliminates the need for transportation. Students with disabilities may not drive, so this can increase the likelihood of participation.
• Conduct focus groups during school hours. This offers the additional perk that some students enjoy ‘getting out of class’. There is a drawback though; you are limited to one class period. For many high schools this means you will only have 45-50 minutes to conduct the focus group.
• Carefully consider how you group participants. Different group arrangements may inhibit student participation. Developmentally and cognitively, some participants may not be ready to open up and share information. The addition of a disability may confound disclosure of sensitive information. If you are collecting data from students with disabilities, be prepared to address the student’s comfort level related to their disability and skills.
• Provide an Advanced Organizer. Many students with disabilities thrive in a structured setting and benefit from a schedule of events. Advanced Organizers help prepare participants for what is to come. Give options for written or visual format. Be careful not to deviate from the schedule or you may increase anxiety levels.

Click to increase size

Lessons Learned:

• Provide multiple modes of data collection. Our most successful focus group for data collection allowed the students to fill out a short survey which they kept with them. This helped them communicate more effectively during the focus group.
• Plan ahead for unanticipated needs. We had an unexpected Deaf participant who needed an interpreter, a guest to our focus group. Be ready to let the ‘guest’ know ‘the rules’ of your data collection. A handout listing the expectations will be helpful, e.g. please only interpret exact words, do not answer for the participant even if you disagree, do not interrupt the flow of conversation.
• Provide multiple ways to state a question. A variety of skills and abilities may be represented. On several occasions in our focus groups, the moderator needed to provide alternate definitions and descriptions. Preparing for these will enhance consistency across groups.

Rad Resources:

• People First Language from Disability is Natural
• Social Competence and the Child with Learning Disabilities
• School-wide Strategies for Managing…Off-task/Inattention

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Deaf · disability · expectations · focus groups · graphic organizers · inattention · interpreter · multiple means · off-task behavior · proxy · questions · social competence

Hi, I’m Mary Arnold, a professor and youth development specialist at Oregon State University. In a previous AEA365 post I presented three tips for successfully engaging youth in participatory evaluation (YPE) projects. Here are a few more tips that can help ensure the success of your YPE endeavors.

Hot Tips:

• Teach the Cycle. As trained evaluators, the steps of the evaluation cycle are second nature to most of us. Identifying an evaluation purpose, developing questions, collecting and analyzing data, interpreting results, communicating, are familiar steps to evaluators. But youth new to evaluation are without a roadmap, and the steps of evaluation are found along a mysterious foggy country road… in the dark! It is our job to provide a roadmap, describe the landscape, and guide the journey until youth gain clarity and confidence. When I conduct trainings for YPE I keep a poster of the evaluation cycle on the wall. I refer to the poster many times during the training, explaining where we are in the process. One of my favorite curriculums for teaching about the cycle of evaluation is Participatory Evaluation with Young People by Barry Checkoway and Katie Richards- Schuster.

• Hit the Floor Furniture is fine, but there is nothing like a good debrief on the floor. Whenever I can I sit with youth (and adult partners if they can) on the floor in a circle to reflect on what we are learning together. I usually have a “talking stick” of some kind that youth can hold while they share their thoughts. I use these times to reflect on what we have learned together, gather ideas for what is working and what is not, and create plans for our next steps together.

• Fan the Flames I spend summers backpacking the mountains of Oregon. My favorite time of the day is in the evening when I start a small campfire from just a twig and one match. I know that the twig and match alone will not turn into a fire to cook my dinner and keep me warm without some help. I carefully add twigs and fan the flame until I know the fire is strong enough to burn alone. The same is true for engaging youth in participatory evaluation projects. Without proper support and encouragement, the initial flame will burn out. I keep things going by providing post-training support via webinars, newsletters, mentor conference calls, and inviting youth to share their projects. Today’s opportunities to connect through Facebook, Twitter and YouTube make staying in touch with projects even easier to do. You can see some of the fanning strategies I use on the website for my current program YA4-H! Youth Advocates for Health.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

social media · talking stick · youth participation

Hi, I’m Sara Plachta Elliott, Evaluation Fellow at the Skillman Foundation in Detroit, Michigan, through a grant to Brandeis University’s Center for Youth and Communities. As an Evaluation Fellow, I work with Foundation staff to create learning opportunities that inform the work of improving school and neighborhood conditions for Detroit’s kids.In 2010, Brandeis conducted a System of Supports and Opportunities (SOSO) analysis in six neighborhoods where Skillman focuses its grantmaking and changemaking efforts. Data collection involved interviewing youth program and basic service provider staff to assess program offerings, participation rates, program quality, and accessibility.

Below are learning opportunities that were created with SOSO data.

Lesson Learned: Have stakeholders review findings before the final report is prepared.

• An internal review with Foundation staff helped identify inaccuracies in participation rates and site location addresses.
• Then two-page SOSO snapshots for each neighborhood were released.
• Key neighborhood and youth program stakeholders were engaged in a review of findings and lifted up questions about how data were collected. This review process ensured “on the ground” validity of the analysis.
• For full transparency, the dataset was also released to partners for system planning purposes.

Hot Tip: Create maps or other visual products.

The Foundation engaged its partner Data Driven Detroit to create maps of agency and program site locations. In meetings with stakeholders, we reviewed these maps along with youth population maps. In one neighborhood, we learned that most kids lived on the west side but program sites were clustered on the east side. The collective “ah ha” moments helped the Foundation and its stakeholders work together to fill gaps.

Consider hosting a discussion session and ask community partners and stakeholders to interpret brief data reports and maps. What patterns do they see in the data?

Cool Trick: Engage youth and residents in reviewing data, not just program staff.

In the summer of 2011, a social service agency, Southwest Solutions, organized a community youth mapping project. Youth reviewed the Brandeis SOSO data, then designed and conducted their own neighborhood opportunities survey.

Youth researchers were paid through summer youth employment funding and walked every street in their neighborhood twice, administering surveys to youth, youth program staff, and business owners, as well as mapping vacant properties. They discovered that some local businesses wanted youth to work during school hours, thereby tempting them to drop out of school. The youth also learned that students wanted more college and career preparation opportunities, mirroring findings from the SOSO report that more of these opportunities were needed.

Evaluation reports can sit on a shelf if not accompanied by intentional learning opportunities. As an evaluator, encourage clients to create interactive learning opportunities with short, visual reporting products. Spark their collective learning!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

census · data visualization · Evaluation · grantmaking · neighborhood mapping · stipends · System of Support · youth focused evaluation · youth participatory evaluation · youth researchers