Hello! Sheila B. Robinson here, guest posting for Susan Kistler, our regular Saturday contributor. I work in PK12 education at Greece Central School District, and in higher education at the University of Rochester’s Warner School of Education. As aea365’s current Lead Volunteer Curator, I’ve had the pleasure of working with a number of groups – American Evaluation Association Topical Interest Groups (AEA TIGs), AEA Affiliates, and other groups that are united by evaluation practice in various contexts.

Hot Tip: Leave no stone unturned! In other words, don’t skip entire weeks. You can learn a lot even when a sponsored week’s group name doesn’t resonate with you. During sponsored weeks, you can read about how evaluators in different contexts from your own have grappled with evaluation challenges, learned something from working in diverse communities, or tried new technologies to enhance their evaluation practice and are now willing to share their experiences with all of us.

Hot Tip: Dig for enticing artifacts! Look for posts with content that transcends the focus of the sponsored week. For example, while I am not an environmental program evaluator, nor do I evaluate extension education programs, I found these two gems during sponsored weeks:

• In this post, Sara El Choufi shared resources for learning Excel during the Environmental Program Evaluation (EPE TIG) sponsored week.
• In this post, Melissa Cater shared information on creating a Community of Practice during Extension Education Evaluation (EEE TIG) week.

Lesson Learned: While our sponsored week authors may share evaluation foci with each other, they offer Hot Tips, Cool Tricks, Lessons Learned, and Rad Resources that appeal to and can be educative for a broad range of evaluators.

Cool Trick: Get your hands dirty! Sift through the archive and unearth your own gems in sponsored (and non-sponsored!) weeks.

Lesson Learned: Many sponsored weeks have themes that cut across evaluation contexts. In addition to TIG-sponsored weeks,we’ve hosted Cultural Competence Week, Innovative #Eval Week, Video in #Eval Week, AEA affiliate weeks, Bloggers Series Weeks, and Local Area Working Group Weeks, among others.

Rad Resource: History in the making: Check out aea365 and our archive for a list of over 1000 nuggets of evaluation wisdom from hundreds of authors. With about 70 sponsored weeks on aea365, there’s a lot to learn! So, get into comfortable clothes, get your virtual trowel, sieve, and brush and get your read on!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

aea365 · Evaluation · learning

Hi! I’m Sheila B. Robinson, AEA365’s Lead Curator. I’m also an educator with Greece Central School District, and the University of Rochester’s Warner School of Education.

Today, I’ll share lessons learned about evaluation planning and a fabulous way to get ready for summer (learning about evaluation, of course!).

Rudyard Kipling wrote, I keep six honest serving-men, (They taught me all I knew); Their names are What and Why and When, And How and Where and Who.

The “5 Ws and an H” have been used by journalists, researchers, police investigators, and teachers (among many others, I’m sure) to understand and analyze a process, problem, or project. Evaluators can use them to frame evaluation planning as well.

Lesson Learned: Use these questions to create an outline of an evaluation plan:

What: What is your evaluand and what is the focus of the evaluation? What aspects of the program (or policy) will and will NOT be evaluated at this time? What programmatic (or policy) decisions might be made based on these evaluation results? What evaluation approach(es) will be used?

Why: Why is the evaluation being conducted? Why now?

When: When will evaluation begin and end? When will data be collected?When are interim and final reports (or other deliverables) due?

How: How will the evaluation be conducted? How will data be collected and analyzed? How will reports (or other deliverables) be formatted (i.e. formal reports, slides, podcasts, etc.) and how will these (and other information) be disseminated?

Where: Where is the program located (not only geographic location, but also where in terms of contexts – political, social, economic, etc.)?

Who: Who is the program’s target population? Who are your clients, stakeholders, and audience? Who will be part of the evaluation team? Who will locate or develop measurement instruments? Who will provide data? Who will collect and analyze data and prepare deliverables? Who are the primary intended users of the evaluation? Who will potentially make decisions based on these evaluation results?

Can you think of other questions? I’m sure there are many more! Please add them in the comments 

Hot Tip: Register for the American Evaluation Association’s Summer Evaluation Institute June 2-5, 2013 in Atlanta, GA to learn more about 20+ evaluation-related topics.

Hot Tip: Want to learn more about evaluation planning? Take my Summer Institute course It’s not the plan, it’s the planning (read the description here).

Rad Resource: Susan Kistler highlighted a few institute offerings here.

Rad Resource: I think this course: Every Picture Tells a Story: Flow Charts, Logic Models, LogFrames, Etc. What They Are and When to Use Them with Thomas Chapel, Chief Evaluation Officer at the Centers for Disease Control and Prevention, sounds exciting. Read the description here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation · Evaluation planning

Hello! We are Johanna Morariu, Kat Athanasiades, and Ann Emery from Innovation Network. For 20 years, Innovation Network has helped nonprofits and foundations evaluate and learn from their work.

In 2010, Innovation Network set out to answer a question that was previously unaddressed in the evaluation field—what is the state of nonprofit evaluation practice and capacity?—and initiated the first iteration of the State of Evaluation project. In 2012 we launched the second installment of the State of Evaluation project. A total of 546 representatives of 501(c)3 nonprofit organizations nationwide responded to our 2012 survey.

Lessons Learned–So what’s the state of evaluation among nonprofits? Here are the top ten highlights from our research:

1. 90% of nonprofits evaluated some part of their work in the past year. However, only 28% of nonprofits exhibit what we feel are promising capacities and behaviors to meaningfully engage in evaluation.

2. The use of qualitative practices (e.g. case studies, focus groups, and interviews—used by fewer than 50% of organizations) has increased, though quantitative practices (e.g. compiling statistics, feedback forms, and internal tracking forms—used by more than 50% of organizations) still reign supreme.

3. 18% of nonprofits had a full-time employee dedicated to evaluation.

4. Organizations were positive about working with external evaluators: 69% rated the experience as excellent or good.

5. 100% of organizations that engaged in evaluation used their findings.

6. Large and small organizations faced different barriers to evaluation: 28% of large organizations named “funders asking you to report on the wrong data” as a barrier, compared to 12% overall.

7. 82% of nonprofits believe that discussing evaluation results with funders is useful.

8. 10% of nonprofits felt that you don’t need evaluation to know that your organization’s approach is working.

9. Evaluation is a low priority among nonprofits: it was ranked second to last in a list of 10 priorities, only coming ahead of research.

10. Among both funders and nonprofits, the primary audience of evaluation results is internal: for nonprofits, it is the CEO/ED/management, and for funders, it is the Board of Directors.

Rad Resource—The State of Evaluation 2010 and 2012 reports are available online at for your reading pleasure.

Rad Resource—What are evaluators saying about the State of Evaluation 2012 data? Look no further! You can see examples here by Matt Forti and Tom Kelly.

Rad Resource—Measuring evaluation in the social sector: Check out the Center for Effective Philanthropy’s 2012 Room for Improvement and New Philanthropy Capital’s 2012 Making an Impact.

Hot Tip—Want to discuss the State of Evaluation? Leave a comment below, or tweet us (@InnoNet_Eval) using #SOE2012!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · Data · Evaluation · nonprofit · organizational learning · reports

I am Tarek Azzam, assistant professor at Claremont Graduate University and associate director of the Claremont Evaluation Center.

Today I want to talk about crowdsourcing and how it can potentially be used in evaluation practice. Generally speaking, crowdsourcing is the process of using the power of the many individuals (i.e. the crowd) to accomplish specific tasks. This idea has been around for a long time (e.g. the creation of the oxford dictionary), but due to recent developments in technology, the ability to access the power of the crowd has become much easier.

I will focus on just one crowdsourcing website because Amazon’s Mechanical Turk is the most widely known, used, and studied crowdsourcing site. This site helps to facilitate the interactions between “requesters” and “workers” (see figures below). A requester can describe a task (e.g. please complete a survey), set the payment and allotted time for completing a task, and determine the qualifications needed to finish the task. This information is then posted on MTurk website, and interested individuals who qualify can complete the task for the promised payment.

(Share Clip)

This facilitated marketplace has some really interesting implications for evaluation practice. For example, evaluators can use MTurk to establish the validity and reliability of survey instruments before giving them to intended participants. By posting a survey on MTurk and collecting responses from individuals with similar background characteristics as your intended participants, an evaluator can establish the reliability of a measure, get feedback on the items, and if needed translate the items into another language. All this can be accomplished in a matter of days. For me personally I’ve been able to collect 500 responses for a 15 minute survey, at a cost of 55 cents per survey in less than three days.

Hot Tip: when selecting the eligibility criteria for MTurk participants choose those with 95% or higher approval ratings.

There are other uses that I am currently experimenting with. For example:

• Can MTurk respondents be used to create a matched comparison group in evaluation studies?
• Is it possible to use MTurk respondents in a matched group pre-post design?
• Is it possible to use MTurk to help with the analysis and coding of qualitative data?

These are things that are yet to be known but I will keep you updated as we progress in exploring the limits of crowdsourcing in evaluation practice.

(Share Clip)

Hot Tip: I will be presenting a Coffee Break Demonstration (free for American Evaluation Association (AEA) members) on Crowdsourcing on Thursday April 18, 2013 from 2:00-2:20pm EDT. Hope to see you there.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Data · Evaluation · survey · technology

Hello, I am Gisele Tchamba, a PhD candidate in the Interdisciplinary PhD in Evaluation program (IDPE) at Western Michigan University. During the three years I spent in this program I came to understand that knowing the difference between evaluation and research matters. I found myself defining evaluation and explaining the difference to people in various disciplines.

Lessons learned:

• Know the difference. To become good evaluators, students with social science background must learn to distinguish between evaluation and research. I came to realize that practicing evaluation does not preclude doing pure research. On the contrary, the methods are interconnected but the aim is different. Therefore after I have learnt and understood the difference between the two, I developed a personal method to lighten the burden of explaining the difference by getting two imaginary hats (researcher and evaluator), that I switch on and off as I do evaluation/research or explain to people what evaluation is and how it differs from research.
• The burden of explaining. Many people in academia vaguely know the meaning of evaluation. Those who think they do mistake evaluation for assessment in education. Whenever I meet with people whose understanding of evaluation is limited to educational assessment, I use Scriven’s definition and emphasis words like “value, merit, and worth”. This usually brings forth expression like “oooh” that is usually followed by their desire to learn more. So I give Coryn’s definition “Evaluation is driven by the need to make informed, defensible, decisions and the desire to impose classifications regarding quality or value…..” Then I explain that social science research is a truth seeking activity aimed at contributing to existing knowledge or generating new knowledge, or for application to some specific problem related to human action and interaction and does not have the burden or rendering judgment of merit or worth.

Hot Tip – Distinguishing between evaluation and social science research:

1. Theoretical and practical experiences are helpful ways to distinguish between the two disciplines. The IDPE curriculum prepares students to do both evaluation and research, e.g. research methods courses that are required for the mastering of evaluation methodology are actually social science research methods.
2. Extensive reading of evaluation literature helps to see the difference. I also like Trochim’s definition of evaluation found in his evaluation of Scriven minimalist theory, “Evaluation is a profession that uses formal methodologies to provide useful empirical evidence about public entities (such as programs, products, performance) in decision making contexts that are inherently political and involve multiple often conflicting stakeholders, where resources are seldom sufficient, and where time-pressures are salient.”

Resources:

• Mathison’s chapter – What is the difference between evaluation and research and why do we care?
• Social Research Methods
• Evaluation Exchange

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation · research

My name is Linda Delaney and I am a Learning Coordinator for the City of Memphis Office of Talent Development. I have been a consultant in evaluation for over 10 years. I work with Dr. David Fetterman as an evaluator for the Minority Sub-recipient Grant Office in Arkansas – a State-wide tobacco prevention evaluation.

In Empowerment Evaluation, simplicity is a key factor to ensure that participants can easily apply concepts that help them to identify and clarify their mission, assess their performance, and strategically plan activities to accomplish their long-term goals.

In addition, as an administrator, trainer, and facilitator of Empowerment Evaluation, I provide evaluative (positive and constructive) and non-evaluative feedback along the way. Providing feedback to community and program staff members is critical and helps guide the learning process.

Rad Source: Using Effective Communication Skills PowerPoint

Hot Tip – Provide evaluative feedback tools: Participants like measurements that simply let them know where they are: on track or not.

Hot Tip – Provide positive evaluation feedback: Participants are encouraged when they to hear from the evaluator that they have done a good job. It reinforces constructive behavior and activity, required to move the project forward.

Hot Tip – Provide constructive evaluative feedback: When you need to instruct and guide participants on how to correct actions, constructive evaluative feedback is necessary. The term “constructive feedback” often generates a negative initial response, but community or staff member response is determined by the manner in which the message is delivered. Avoiding a tone of criticism that sounds like they are being “chewed out” goes a long way in producing the desired results. Constructive feedback shared in supportive tones can help to put things back on track.

Hot Tip – Provide non-evaluative feedback: Non-evaluative feedback does not assign a value to actions. It simply acknowledges the actions and/or feelings of people. Non-evaluative feedback can be as simple as saying, “thank you for your input” or “that’s an interesting way of looking at things”.

Lessons Learned: When would participants want to hear your feedback? When they are still thinking about the work and when they can still do something about it. Giving immediate and appropriate feedback helps participants hear it and use it while the performance in question or actions are still fresh on their minds.

The American Evaluation Association is celebrating CPE week with our colleagues in the Collaborative, Participatory, and Empowerment TIG. The contributions all this week to aea365 come from our CPE TIG Colleagues. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

Evaluation · feedback

My name is Kylie Hutchinson. I am an independent evaluation consultant and trainer with Community Solutions Planning and Evaluation. I have taught workshops on various evaluation topics for both the AEA and CES and Twitter weekly at @EvaluationMaven.

Evaluators are often given the task of conducting needs assessments. Have you ever wondered how you can best ensure your needs assessment covers all the necessary bases? Several years ago I created a framework to help others conduct needs assessments that are comprehensive, provide a strong argument for support, and yield the best program design possible.

Rad Resource: The 6 D’s of Needs Assessment (http://www.communitysolutions.ca/pdf/needs-assessments-6ds.pdf). This one page reference lists six focus areas for most traditional needs assessments: deficit, develop, describe, desires, duplication, and demand, plus questions to help guide your research.

Hot Tip: Within the world of sometimes overlapping evaluation terminology, you may also find a needs assessment called a gap analysis, feasibility study, environmental scan, and/or situational analysis. It’s up to you to ask questions and interpret what exactly is required.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation

My name is Elizabeth Autio and I am an associate at Education Northwest, a non-profit organization serving educators through research, evaluation, and technical assistance.

This is a companion piece to my earlier post on Recruiting Participants for Your Study: Practical Strategies and Advice that more specifically details tips on recruiting teachers into your study.  I found it was important to do these sessions in person so that teachers could hear my recruitment message and make informed decisions about their participation.

1. Make it personal.  A potential participant’s decision is influenced by their personal rapport with you. Take the time to establish relationships.  These can be initiated via email or telephone, but are best solidified by an in-person visit.  If they will be interfacing with a study team, showing pictures of the other team members communicates more than sharing their vitas.
2. Flexibility and firmness.  Scheduling your visit requires a balance of these qualities.  Some dates might be on your calendar weeks in advance, while others might come through the day prior.  At the same time, be clear about how much time you need with the teachers to adequately deliver your information and answer their questions.
3. Add extra travel time.  Teachers are on tight schedules.  Being on time respects that; moreover, if you are late, those are lost minutes for your recruitment session that you will not get back.  Confirm addresses and directions, as schools are routinely rebuilt, closed, or temporarily housed, and such changes are not always reflected in Google or Mapquest.  These sites also often underestimate driving times; I add 50 percent, then an extra 10 minutes to get myself out the door.
4. Take snacks.  Putting out snacks to share with the group is a small thing that goes a long way in showing your appreciation.
5. Pool teachers across a district.  If possible, ask teachers to come together to a central site across a district.  This saves time and money.
6. Be clear.  Explain who you are, who you work for, and who is funding the study.  Communicate essential points of the study methodology in layperson terms.
7. Be aware of their community.  The places that you visit might be different than your own: politically, demographically, culturally, and/or religiously.  Learn about community norms in advance, if possible, or observe them while you are there.  This might affect the assumptions you make, your dress, your jokes, or even the way that you address people.
8. Put a signature on it!  Give participants something to sign – such as a memorandum of understanding (see example here) – that outlines the details of their participation.  Leave them with ready-to-sign copies and postage paid return envelopes.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation · recruitment

Hi, we are Tom Archibald and Jane Buckley with the Cornell Office for Research on Evaluation. Among other initiatives, we work in partnership with non-formal educators to build evaluation capacity. We have been exploring the idea of evaluative thinking, which we believe is an essential, yet elusive, ingredient in evaluation capacity building (ECB). Below, we share insights gained through our efforts to understand, describe, measure, and promote evaluative thinking (ET)—not to be confused with the iconic alien!

Lesson Learned: From evaluation

• Michael Patton, in an interview with Lisa Waldick from the International Development Research Center (IDRC), defines it as a willingness to ask: “How do we know what we think we know? … Evaluative thinking is not just limited to evaluation projects…it’s an analytical way of thinking that infuses everything that goes on.”
• Jean King, in her 2007 New Directions for Evaluation article on developing evaluation capacity through process use, writes “The concept of free-range evaluation captures the ultimate outcome of ECB: evaluative thinking that lives unfettered in an organization.”
• Evaluative thinkers are not satisfied with simply posing the right questions. According to Preskill and Boyle’s multidisciplinary model of ECB in the American Journal of Evaluation in 2008, they possess an “evaluative affect.”

Lesson Learned: From other fields

Notions related to ET are common in both cognitive research (e.g., evaluativist thinking and metacognition) and education research (e.g., critical thinking), so we searched the literature in those fields and came to define ET as comprised of:

• Thinking skills (e.g., questioning, reflection, decision making, strategizing, and identifying assumptions), and
• Evaluation attitudes (e.g., desire for the truth, belief in the value of evaluation, belief in the value of evidence, inquisitiveness, and skepticism.)

Then, informed by our experience with a multi-year ECB initiative, we identified five macro-level indicators of ET:

• Posing thoughtful questions
• Describing and illustrating thinking
• Active engagement in the pursuit of understanding
• Seeking alternatives
• Believing in the value of evaluation

Rad Resource: Towards measuring ET

Based on these indicators, we have begun developing tools (scale, interview protocol, observation protocol) to collect data on ET. They are still under development and have not yet undergone validity and reliability testing, which we hope to accomplish in the coming year. You can access the draft measures here. We value any feedback you can provide us about these tools.

Rad Resource: Towards promoting ET

One way we promote ET is through The Guide to the Systems Evaluation Protocol, a text that is part of our ECB process. It contains some activities and approaches which we feel foster ET, and thus internal evaluation capacity, among the educators with whom we work.

Tom and Jane will be offering an AEA Coffee Break Webinar on this topic on May 31^st. If you are an AEA member, go here to learn more and register. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation · evaluative thinking · webinar

My name is Catherine (Brehm) Rain, Vice President of Rain & Brehm Consulting Group, Inc., an independent evaluation and consulting firm located in Rockledge, Florida.  I blog at The Evaluation Forum.

Rad Resource – The Evaluation Forum: New to our website, The Evaluation Forum focuses on the why and wherefore of evaluation of health promotion and health-related, risk-reduction programming.  The blog targets program personnel with some or no background in the principles, practices, purposes and benefits of program evaluation. Content is basic, and covers issues such as hiring an evaluator, program design, and fidelity (among other future topics). We post new content monthly and expect to increase frequency of postings this year.

Hot Tips – favorite posts: We added our blog in September of this 2011. Thus far, my two favorite posts are

• 10/3/2011 7 Qualities of an Effective Universal Program Design  Nine times out of ten, program ‘problems’ are directly related to program ‘design’—or the absence thereof. This blog covers basic qualities of program design and leads readers to one of my favorite places: Theory At a Glance: A Guide to Health Promotion Practice.
• 12/11/2011 Fiddling with Fidelity? Fidelity means, in a word: faithfulness.  As a former project director and a current evaluation team member specializing in Process Evaluation, I liken adherence to a grant management plan or a program design, to following a recipe for bread pudding.  Yes, you can tweak it here and there, if you know what you’re doing. If you don’t, you might end up as I did, with a batch of botched pudding!

Lessons Learned – why I blog: I blog, because I am first and foremost a writer—I write two other blogs non-related to evaluation.  Chiefly and with relevance to The Evaluation Forum: I blog to bring basic information to clients and program personnel so that they (a) grow their knowledge about evaluation; (b) apply evaluation principles to program design and implementation; and in so doing (c) maximize outcomes.

Lessons Learned: You have to commit to a blog in the same way you do to a subscribed newsletter: often; and whether you have time for it, or not. It is an adjustment.  It also takes time to develop a following—if you want one. Linking posts to our Facebook page has added a ‘friendly community’ factor, as well.  Sometimes, folks are a little shy of evaluation and its impact on their organization or project.  Finding us on-line or on Facebook with helpful hints or solid information they can use meaningfully, may be the first step we can take as professionals to help our clients and community succeed! (It’s also nice to be ‘liked’!)

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation