Greetings aea365 readers! I’m Sheila B. Robinson, Lead Curator of my favorite evaluation blog, aea365. As I’ve written many times before, I’ve claimed the title of “aea365′s biggest fan.” Not only have I been an avid reader myself, but a frequent contributor as well. In fact, my first piece of published writing was indeed an aea365 post.

Have you been wanting to write and share a Hot Tip, Lesson Learned, Cool Trick, or Rad Resource with the evaluation community?

Get Involved! Consider contributing a tip to aea365. It’s easy, fun, and can be very rewarding. Whether your curriculum vitae is a mile long, or just ready for its first entry, we would love to hear from you. We feature clear contribution guidelines and a Lead Curator who is willing to help get your tip published.

Lesson Learned: Writing for aea365 “gets your name out there.” Whether you are rounding out a resume with some writing experience, looking for evaluation work, growing a clientele base, or composing evaluation-related blog posts just for fun (who does THAT? Oh wait, that would be me…), your aea365 contributions give readers an inkling of your evaluation expertise, along with your writing skill and style.

Hot Tip: Have you recently attended or are you planning to attend an evaluation-related conference? Have aea365 on the brain! As you attend sessions and learn new evaluation tips, tricks, or techniques, why not blog about them? Consider this: How will YOU use your new learning in YOUR evaluation practice? Even if you’re not attending conferences, I’d pose that same question to you. As YOU continue to learn, grow, and refine your evaluation practice, what do YOU have to share with the evaluation community?

Cool Trick: If you sincerely want your posts published, surf through a whole pack of past posts please. Check the archive to search and skim ones of most interest to you, and notice how they all follow contribution guidelines.  You will find that aea365 is indeed educative for a broad range of evaluation researchers and practitioners. Notice that we don’t promote businesses or sell products.

Aea365 authors are a community of evaluation “teachers” and “learners,” with each of us taking on each role as we read, write for, and learn from this amazing wealth of evaluation expertise. Won’t you join us?

This post is a modified version of a previously published aea365 post in an occasional series, “Best of aea365.” Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings!  I am Carl Brun, a social worker turned professor turned evaluator.  I have taught in the social work department at Wright State University in Dayton, OH for 21 years.

I teach evaluation in every research methods course I teach, whether it be in a human services course or an undergraduate or graduate social work course.  Students can relate to evaluation as an actual activity that occurs in social services compared to research which they see only happening in universities.

Lessons Learned:  Have students do evaluation.  Partner with community agencies to have students apply their evaluation skills to help develop and implement evaluations.  In one graduate level course, my students conducted a door-to-door needs assessment.  Their efforts helped the agency receive a $733,000 grant to begin a federally qualified health clinic.  See http://www.talberthouse.org/media/documents/Newsletter_2013%20Fall.pdf

Hot Tip:  Demystify evaluation in the very first class.  I have students discuss ways they use research in their everyday lives to make decisions, such as “how did you research coming to this university or choosing this major?”  I ask them to think of ways they have evaluated others (ex. student evaluations of professors) and been evaluated (ex. by a supervisor).

Hot Tip:  SCREAM.  This is the acronym I use to emphasize values I support for every evaluation:  Measure Strengths. Be Culturally competent.  Evaluate within the Resources you have.  Ethics, ethics, ethics.  Get Agreement from all stakeholders on all aspects of the evaluation.  Measure Multiple systems levels.

Hot Tip: I simplify research methods by discussing three types of evaluation questions and three types of data collection.  Exploratory questions = qualitative methods. Explanatory questions = quantitative methods. Descriptive questions = both.  All questions can be answered by asking questions, observation, or secondary data analysis.  I have a chart that puts these pieces together to help the students develop their research design.

Resources:  There are many electronic discussion forums in which teachers share their syllabi and teaching tips.  Among the ones I use are AEA’s Evaltalk, and one established primarily for social work educators  (http://www.bpdonline.org/bpd_prod/BPDWCMWEB/Resources/BPD-L_List/BPDWCMWEB/Resources/BPD-L_Email_List.aspx?hkey=bf39d2a2-7005-4db1-a6d0-e3587cc98956#Join).  I love talking about teaching evaluation.  Feel free to contact me at carl.brun@wright.edu.

The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We are Carolyn Sullins and Ladel Lewis of Meaningful Evidence, LLC and we have years of experience evaluating children’s mental health systems of care.  These evaluations were challenging because attention to local communities’ cultures was crucial – but so was adherence to a national battery of questionnaires and to local regulations.  The community vs. academic/professional cultures was a more salient divide than race, religion, or ethnicity.  This divide was sometimes challenging but always crucial to bridge.

Hot Tips: Get advice from members of your population.  One way is inviting representatives to your evaluation team meetings.  It can be more helpful to attend their meetings!  Members will feel more comfortable and candid on their own “turf,” and more of them will be there.  Breaking the ice with humorous examples of cross-cultural misinterpretations in research/evaluation has helped.

Here are some questions that may be helpful:

• Which survey items may be unclear?  How can they be clarified?  What local terms can be used?
• What recruiting methods and incentives are likely to work?
• How can we help people feel more comfortable answering questions?
• What items are people likely to refuse to answer or not answer honestly?
• What questions should we add that will be helpful?

Lessons learned: Some surprising distinctions between the community culture and the academic/professional culture involved conceptions of confidentiality, even for interviews that included sensitive questions:

• For confidentiality and accuracy, academia often insists that interviewers cannot interview people they know.  Achieving this goal can clash with the goal of having ample consumer representation on the evaluation team.  Our consumer representatives found this restriction condescending and counterproductive.
• Conducting interviews in a fast food restaurant can be taboo in the academic world, but several of our participants specifically requested this.
• When interviews are conducted in an interviewee’s home (the request of the overwhelming majority of interviewees), what does an interviewer do when a family member enters the room during an interview?  Respecting the interviewee’s preference may clash with the academic expectations of confidentiality. Still, when in doubt, the answer is obvious:  Pause, and ask if they would like for you to continue.
• How should interviewers act if they encounter their interviewees in public? Traditional academia recommends a cool detachment to protect privacy, but interviewees may find this cold and an indication that they had been callously “used.”

Rad Resource:  While Morse’s 2003 article “Bending the ivory tower: Communities, health departments, and academia” on Public Health, it is relevant to other areas of human service.

Paying attention to the preferences of our population while maintaining our academic integrity was a balancing act, not a contradiction.  Attending to both helped us with participant retention and the accuracy of our data.

The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

cultural competence

Hello!  I am Kathy Bolland, like many of you, a wearer of many hats.  I am an administrator and faculty member in a school of social work, where my research focuses on impoverished youth at risk and on assessment in higher education.  I also evaluate educational projects and social programs.  I am a past AEA treasurer and current co-chair of the Social Work Topical Interest Group.

Often I tell my students that the hardest thing in evaluation is framing and focusing the evaluation question.  Great evaluation designs, implementation, and analysis are not helpful when they overlook questions important to stakeholders.  Before the evaluation questions though, must come conversations with stakeholders.  This is where evaluators often must search for common ground and dispel fear of evaluation.

Lesson Learned: I have found that many professionals who need an evaluation or who must provide information for evaluation have a code of ethics. Telling them about AEA’s Guiding Principles for Evaluators and how those principles relate to their code of ethics can help.

The Code of Ethics of the National Association of Social Workers (NASW) is important to social work practitioners, educators, and students. I will provide a couple of examples of similarities between this code and to our Guiding Principles.

The AEA Guiding Principles tell us, “Evaluators conduct systematic, data-based inquiries.” The principles remind us to “practice within the limits of [our] professional training and competence….” The NASW Code of Ethics tells social workers, “Social workers practice within their areas of competence and develop and enhance their professional expertise.  Social workers continually strive to increase their professional knowledge and skills and to apply them in practice. Social workers should aspire to contribute to the knowledge base of the profession.”  I remind social workers that one way they can increase their professional knowledge and skills and apply them in practice is to conduct systematic, data-based inquiries.”

Our Guiding Principles address cultural competence: “To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence.”  The NASW Code of Ethics addresses cultural competence as well.  In brief, “Social workers should obtain education about and seek to understand the nature of social diversity and oppression with respect to race, ethnicity, national origin, color, sex, sexual orientation, gender identity or expression, age, marital status, political belief, religion, immigration status, and mental or physical disability.”

Many more points of similarity exist between these two guides and between our Guiding Principles and guidelines and codes specific to other professions.  They can provide a common ground to begin a conversation.

(Share Clip)

The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

ethics

Hello, my name is Alicia McCoy and I am the Research and Evaluation Manager at Family Life in Melbourne, Australia.  Family Life is a nonprofit community organization that supports families, children and young people.  A big part of my job is ensuring our practitioners (both social workers and other human service professionals) value and participate in evaluation.  In my experience, the second doesn’t always happen just because you achieve the first.  Here are some of my hot tips and lessons learned.

Lesson Learned:  Social workers are more likely to appreciate evaluation when they see the connection it has to evidence-informed practice.  Spending time and effort to create a shared understanding about how participating in evaluation is part of a social worker’s ethical obligation, will give you a better chance of getting engagement in evaluation activities.  I use an internal blog (called the Knowledge Centre) to explicitly and implicitly strengthen this connection on a regular basis.

Hot Tip: Two words – data visualisation!  Going the extra mile when communicating anything about evaluation will pay off in the end.  Social workers are more likely to engage with evaluation if findings are presented in an interesting and stimulating way.  Think color, interactivity and creativity, without a loss of meaning or rigour of course!  Check out the great work being done by Stephanie Evergreen and Ann K Emery for some specific tips.

Hot Tip:  Social workers are much more likely to engage in evaluation when the organization they work in has a positive relationship to learning.  Carrying out an organizational assessment using a tool such as Hallie Preskill’s Readiness for Organisational Learning and Evaluation will help you understand what your organization’s strengths and weaknesses are and how each might be influencing your social workers’ engagement with evaluation.

The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data visualization

Hi there, my name is Karen Anderson and I’m the Continuous Quality Improvement and Evaluation Coordinator at Families First in Atlanta, Georgia. In my spare time, I enjoy working with community groups and organizations that do not have evaluation capacity. It’s always a wonderful learning experience for me, as well as for the group or organization when we begin to discuss the possibilities of integrating evaluation into the work that they do.

Hot Tip: Don’t start the discussion using technical jargon. Develop a list of open ended questions, for example: “What information do you collect on your clients?” or “What information will help you to understand your clients even better?”

If information is being collected on clients, ask how it’s being analyzed, reported, and used. There are many entry points for evaluation.  The key is being flexible and a good listener to see where you can provide support.

Hot Tip: Do the best you can to make the evaluation a process, and not just an exercise to complete a report. The evaluation process and outcomes need to be useful and relevant to help with decision making along the way.

Hot Tip: Demystify the process by providing a general overview of evaluation, its benefits, and examples of reports you’ve completed in the past to provide a picture. Since we do evaluation everyday it is second nature to us, however, we can’t assume that everyone knows what we do and how it relates to the work that they do.

The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Katrina Brewsaugh, Senior Associate with the Annie E. Casey Foundation.  As part of Social Work TIG week on aea365, I’d like to tell you why social workers are an asset to an evaluation team.  Evaluation may not be the first thing that comes to mind when you think of social work, but the training we receive can be an asset to evaluation teams in several ways.

Lesson Learned:  Social workers are systems thinkers.

Ecological systems theory is at the core of social work training, from the bachelor’s degree through the doctorate.  A social worker on your team can identify issues and concerns that impact the micro- (family), mezzo- (community), and macro- (policy) levels, regardless of whether the evaluation is localized to one small program or has a national policy reach.

Lesson Learned:  Social workers are interdisciplinary.

Social work synthesizes knowledge of related fields such as psychology, sociology, education, public health, and even economics.  On interdisciplinary evaluation teams, a social worker can be an excellent translator and present a unified voice.

Lesson Learned:  Social workers are hands on.

All social workers must complete training in the field working with clients.  If you need someone skilled in engaging and interviewing consumers and stakeholders, then you need a social worker!

Rad Resources:  Here are some places to begin your search for a social work evaluator:

• Search AEA’s Find an Evaluator list using ‘social work’ or ‘MSW’ as a search term.
• Contact your local university’s social work program.
• Contact a member of the Social Work TIG (through the “members only” link on www.eval.org).
• Look for social workers on EvalTalk or AEA’s LinkedIn page.

So the next time you need to bring together an evaluation team, consider a social worker!

The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Systems thinking

Hi! I’m Sheila B. Robinson, aea365′s Lead Curator and sometimes Saturday contributor.  I’ve been a conference proposal reviewer for AEA’s annual conference for several years now, and always find it a wonderful learning experience and opportunity to get involved and contribute. As Susan Kistler, AEA’s Executive Director Emeritus, posted last year, “It is a bit like being the first in line to see a new exhibit at a museum or getting a preview copy of an engrossing new book before it is available in stores. It is that time of year. I am doing my happy dance.”

Hot Tip – Evaluation 2014 Call for Proposals Now Online: Do you work or study in the field of evaluation? If so, you are invited to propose to share your best work in evaluation theory or practice at Evaluation 2014, the annual conference of the American Evaluation Association. We anticipate that over 3000 colleagues will gather for an exciting week of learning, collaboration, and connection and hope that you will be among those in attendance.

The conference is broken down into 50 Topical Strands that examine the field from the vantage point of a particular methodology, context, or issue as well as the Presidential Strand highlighting this year’s Presidential Theme of Visionary Evaluation for a Sustainable, Equitable Future.

Hot Tip: Presentations may explore the Presidential Strand Theme or any aspect of the full breadth and depth of evaluation theory, practice, management, or consulting.

Next Steps:

• Check out our Evaluation 2014 website
  • Read about the Presidential Theme here
• Log in with a member or non-member account to submit your proposal
  • Once you’re logged in, you can access descriptions of the different types of sessions that may be proposed including pre- and post-conference workshops
• Check out the FAQ for answers to many of your questions

Hot Tip: Over 3000 evaluators will gather in the “Mile High City” –  Denver, Colorado, in mid-October 2014. Watch for special edition aea365 posts by our Local Area Working Group (LAWG) who will share evaluation tips and key information for conference attendees to make the most of their visits to Denver.

Hotter Tip: Proposals are due by 11:59 PM ET March 17, 2014!

Help us to make Evaluation 2014 an event to remember!

(Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

AEA conference · Evaluation 2014

Hello, I’m Shirah Hecht, Ph.D,  Program Evaluator with Academic Technology Services within the Harvard University Info Tech area.  Here is a simple “trick” for beginning to develop a research design.

I call this “system mapping.”  You may connect it to stakeholder analysis or concept mapping, since it blends the two in a way – but it goes a bit further than either, for research purposes.  It comes from a simple suggestion given to me by my graduate school mentor who taught qualitative field methods at Northwestern University, Howard S. Becker.  He credited the sociologist Everett C. Hughes for this method.

Essentially, the technique is to identify a central event or person, then radiate out from there to consider all the constituencies or positions or groups that connect to that central event or person.  This is a way of jump-starting your thinking about what the relevant data sources might be and to identify questions about your central topic.

For example, in education, the central event might be the classroom; the radiating circles might identify students, teachers, parents, and administrators, among others.  Alternatively, the central circle might hold the student as a central person; the radiating circles then might include the parents, teachers, other students, guidance counselors, testing agencies, etc.

After identifying these outer circles, you can pose relevant questions such as:

• What is the perspective of each constituency on the central event or person?  What matters to them?  What is their investment in this process or person?
• At what points do they interact with the central event, for the purposes of my research questions?
• What “data” might they hold, whether in terms of process or perspective, to define or address my research questions?

This process also fits in nicely with developing a logic model with the program provider, to develop an evaluation project.  Even if you are not logic model-bound, it can frame a good conversation and understanding of the research planning process and the final decisions about data collection.

Here is a simple version of this map, generalized from a program for which I developed an evaluation plan.  The green highlights indicate the data collection sources: a focus group with program volunteers and a survey of clients.

Lessons Learned: In research planning, move from the perspective of the constituency to specific research questions for the project.

Hot Tip: Combine this mapping process with the Tearless Logic Model process, to jumpstart the conversation about research plans with program staff. 

Rad Resources:  Everett C. Hughes offers the sociological eye on any and all processes we might want to research: The Sociological Eye: Selected Papers (Social Science Classics

The Tearless Logic Model

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

concept mapping · logic model · research design

Hi, I am Jindra Cekan, PhD, an independent evaluator with 25 years of international development fieldwork, at www.ValuingVoices.com.

What if we saw our true clients as project participants and wanted the return on investment of projects be maximally sustained? How would this change how we evaluate, capture, learn together?

Lesson Learned: Billions of dollars of international development assistance are spent every year and we do baseline, midterm and final evaluations on most of them.  We even sometimes evaluate sustainability using OECD’s DAC Criteria for Evaluating Development Assistance: relevance, effectiveness, efficiency, impact and sustainability.  This is terrific, but deeply insufficient. We rarely ask communities and local NGOs during or after implementation what they think about our projects, how to best sustain activities themselves and how to help them do so.

Also, very rarely do we return 3, 5, or 10 years after projects close and ask participants what is “still standing” that they managed to sustain themselves. How often do we take community members, local NGOs, or national evaluators as the leaders of evaluations of long-term self-sustainability of our projects? Based on my research 99% of international aid projects are not evaluated for sustainability or impact after project close by anyone, much less by the communities they are designed to serve.

With $1.52 trillion dollars in US and EU foreign aid being programmed for 2014–2020, our industry desperately needs feedback on what communities feel will be sustainable now, what interventions offer the likelihood of positive impact beyond the performance of the project’s planned (log-framed) activities. Shockingly, this does not exist today.

Further, such learning needs to be transparently captured and shared in open-date format for collective learning, especially at the country and implementer level. Creating feedback loops between project participants, national stakeholders, partners and donors that foster self-sustainability will foster true impact.

Hot Tip: We can start in current project evaluations. We need to ask these questions of men, women, youth, elders, the richer and poorer in communities as well as of local stakeholders. Ideally we would request national evaluators to ask (revise!) questions such as:

• How valuable have you found the project overall in terms of being able to sustain activities yourselves?
• How well were project activities transferred to local stakeholders?

o   Who is helping you sustain the project locally once it ends?

• What were the activities do you think you can least maintain yourselves?

o   What should be done to help you?

• What were activities that you wish the project had supported that build on your community’s strengths?
• Was there any result that came of the project that was surprising or unexpected?
• What else do we need to learn from you to have greater success in the future?

(Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

development evaluation · sustainability