Hi, this is Kat Athanasiades from Innovation Network. Our team is especially engaged in improving how evaluation information is communicated, and to that end we thought we would do some graphic recording at Evaluation 13!

That’s me, working on the graphic recording of an Evaluation 13 session about advocacy evaluation.

What is graphic recording? Graphic recording is organizing information in a visual way using words, symbols, and pictures. This is often done in real time, paced with the information generated in a panel session, a meeting, a focus group, etc.

Hot Tip: Graphic recording benefits a number of different audiences:

• Speakers. It isn’t always easy to remember the themes of a talk. A visual map of key points can help speakers recall ground covered and build on themes from earlier in a presentation.
• Attendees. A graphic recording of a session can help attendees better follow the conversation and understand key take-aways. In our session, one concept was discussed pessimistically by panelists—and the graphic recording helped elicit discussion from attendees around the “frownie face” that had represented that concept. The session became more participatory as attendee views were incorporated into the visualization.
• Absentees. You can’t make it to every talk or meeting you would like to attend, particularly at Evaluation 13! A graphic recording can help you quickly understand the conversation arc and key points.

What did the finished product look like? Here are a few shots of the completed graphic recording for session 706: Seeing the Forest Beyond the Trees: Learning Across the Experiences of Seven Advocacy Evaluators.

Rad Resource: Looking for inspiration? Check out the RSA Animate videos, which are high-quality visualizations of visionary talks put on by the RSA.

Hot Tip: You don’t need an art background to practice visual recording. I have some basic knowledge of composition and shapes from a couple of drawing and photography courses I took in college—quite some time ago. Really, I just started drawing in a staff meeting one day, and thus a graphic recorder was born. Graphic recorders often work off of templates to give a shape to a visualization, and the point is to populate a graphic recording with easy-to-recognize images. They do not have to be works of fine art.

Rad Resource: Wondering how to get started? You can browse Dan Roam’s Napkin Academy, or practice by incorporating graphic recording methods into your own note taking.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Graphic recording

Hi!  I’m Mallory Johnson, an Evaluation Specialist with the UC Davis Schools of Health Evaluation Unit.  Have you been frustrated by webpages that allow you to view data in a browser, but don’t have export or download functions?  Government websites in particular often post useful evaluation information in useless formats.  After all, that table is no help to your evaluation efforts stuck on a computer screen.  Data scraping (or screen scraping) allows you to extract data from constrictive webpages, kind of like scratching delicious frosting off a stale cake. Here are time-saving techniques for scraping data off the internet into Excel or CSV files.

Rad Resource: Google Spreadsheets is a free resource that allows you to import HTML tables into a spreadsheet.

1. Open a Google Spreadsheet:

2. Locate the URL from the webpage you would like to scrape:

3. Enter the following formula into the first cell..

Formula:  =importHTML(“websiteURL”,“table”,“# of table on the webpage”)

Example: =importHTML(“http://en.wikipedia.org/wiki/California_counties”,”table”,2)

Hot Tip: It may take some trial and error to locate the correct table number (the last portion of the formula). Review the HTML source code for difficult cases.  For Internet Explorer users, navigate to the page, select “view” from the toolbar, and click “source”.  Then, count the number of times you see the word “table” in the source-code.  This will give you the number of tables within the webpage.

4. Press enter, and data will magically populate the spreadsheet.  Data can be exported as Excel, CSV, or text files.

Rad Resource: Import.io (http://import.io/) is a new product that allows users to create data scrapers, spiders, and bots with an easy to use “drag and drop” interface.  The software is still in beta, but it promises to be a coding-free solution for the programming-challenged.

Rad Resource: Beautiful Soup is a Python library designed to make screen scraping projects faster than writing code from scratch.  For those with rudimentary programming knowledge, Beautiful Soup is a scraping, parsing, url-finding wonder.  Both Beautiful Soup and Python are open-source.  Tutorials are available from multiple free venues, such as Coursera and Code Academy.

Lessons Learned: Just because data is “scrape-able” doesn’t mean it’s legal to possess it.  Traditional copyright and privacy protection rules apply.  Also, these techniques should be avoided on password protected websites– even when users utilize avatars or usernames to protect their identities (programs exist with the capability to decode them, making personal information frighteningly transparent).  Most producers of web-based media monitor their websites for scraping activity.  If you wouldn’t want site-owners or users to know that you scraped their data, chances are that you shouldn’t.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data scraping · excel

Hello! I am Laura Beals and I am an internal evaluator at Jewish Family and Children’s Service (JF&CS). JF&CS is a large nonprofit social service agency located just outside of Boston, MA. As we are surrounded by 58 institutions of higher learning, we have many opportunities to leverage students and faculty in order to increase our evaluation capacity.

At Evaluation 2013, I presented with Jennifer Lowe, Director of Research at Crittenon Women’s Union, about how our organizations handle collaborations with students and faculty:

In this post, I am focusing on the process we have at JF&CS for evaluating (pun intended!) requests that come into the agency from students and faculty.

Lessons Learned:

• Requests for research can come in many forms; examples of requests include:
  • Students or faculty requesting staff assistance to distribute flyers or notify agency clients of the availability of a research project.
  • Students requesting access to agency staff or clients to observe a program session, distribute a survey, or take part in an interview for a class project.
  • Students or researchers asking for access to agency administrative or managerial staff to study nonprofit management practices.
  • Academic researchers wishing to gain access to client data or to current clients for scholarly research governed by an Institutional Review Board (IRB).
  • Practitioners or researchers wishing to gain access to clients to validate a clinical evaluative tool.
  • Nonprofit collaborations with students or faculty can have many benefits; for example:
    • Capacity: Tackle a project that might have been on the back burner
    • Knowledge: Learn more about the nonprofit and the field in which it operates
    • Inspiration: Be inspired by the methods or tools used
    • Reputation: Improve the nonprofit’s reputation through collaborations with well-known researchers or universities
    • And can also have many risks, for example:
      • Ethics: Clients may not be treated in an ethnical manner
      • Privacy: Client data privacy may be breached (e.g., HIPPA)
      • Resources: Staff or facilities may be overburdened
      • Reputation: The nonprofit’s reputation may be damaged through bad collaborations

Hot Tips:

• Research requests from students or faculty can be a great way to increase nonprofit evaluation capacity, but there should be a process in place for reviewing and approving to ensure that the benefits outweigh the risks.
• At JF&CS, students or faculty with a research request must complete an application; all requests for research are then reviewed by our internal evaluation and research department. Depending on the level of risk, varying levels of senior leadership need to approve.

Rad Resource: The handout from this session, which details this process in more detail, is available here in the AEA Public elibrary.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

collaboration · research

Hi, my name is Jason Ravitz, an Independent Research and Evaluation Professional and former Director of Research for the Buck Institute for Education, where my job included evaluating project-based learning (PBL) impacts.Here is a district-level approach developed with Melissa “Missy” Rowe, Executive Director of Assessment and Research at Grand Prairie ISD in Grand Prairie, TX.

This approach addresses key concerns about selection bias, that teachers who implement instructional reforms are systematically different from others.  The trick is to compare student performance for the same teacher and students during testing periods when the intervention was and was not used.

Cool Trick: Parentheses represent “optional” steps for added value.

1.  Make a list of participating teachers (with background and treatment data)

2.  Identify tested standards for each testing period (and the most relevant ones)

3.  Find out which teachers implement during each testing period (rating their implementation, when possible

4.  Obtain district- and teacher-level averages, for each course, for each standard and testing period (excluding teacher from district overall)

5.  Calculate differences from the district average for each teacher, testing period and standard

6.  Calculate the average of differences for each teacher, with and without the intervention (teacher-level analysis)

7.  Calculate the average of differences for all teachers (district-level analysis)

Lesson Learned: Benefits: Results can provide a rigorous measure of impacts, effectively controlling for teachers and students by comparing their performance (against themselves and the district) with and without the intervention.  In addition, the results can shape professional development and ongoing research based on findings for each standard.

Example: A few key standards are shown for geometry Teachers A and B during Testing Period 1 (neither used PBL) and Testing Period 4 (when both used PBL).

Teacher A’s students had slightly better than district average (+5%) performance during Testing Period 1, but substantially better performance (+14%) during Testing Period 4, when PBL was used. The same result was not seen for Teacher B, whose students appeared to perform slightly better without PBL.

Interestingly, Teacher A had three PBL-related professional development sessions, while Teacher B only had one.

We see several standards for which Teacher A could provide insights about using PBL effectively (e.g., the second standard on using conversions in real-world situations).  However, Teacher B was more successful using PBL to teach the first standard that is shown (properties of transformations).

Note. Averages reflect all testing periods and standards, not just the few that are visible.

Replicated for all PBL-using teachers in the district these results could be useful for decision makers and for guiding research and professional development based on cases where PBL was effectively used to teach the standards.

What do you think?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

problem-based learning · selection bias · Teacher

Hi I’m Pat Campbell, president of Campbell-Kibler Associates, Inc. A couple of years ago, Eric Jolly, president of the Science Museum of  Minnesota and I were wondering why, with so many evaluations of so many programs, we didn’t know more about “what works, for whom in what context”.  Our conclusion—rigor is not enough.  Even the most rigorous evaluations can be incomplete, or even wrong, when they don’t take into account the needs, values, issues, and goals of different subgroups.  Our solution, with the help of the National Science Foundation, was to develop BeyondRigor.org to provide folks with lots of easy to use tips to improve evaluation with diverse populations (and let’s face it, all populations are diverse).

Hot Tips:  From BeyondRigor.org:

• Whenever possible, use measures that have been tested and validated with groups similar to the groups who will be given the measures.
• Have members of the target population review affective and psychosocial measures for clarity.  Ask them what concepts they think are being measured.  If what is being measured is obvious, consider using a less obvious measure.
• Prior to the data collection, provide the observer or interviewer with as little demographic information as possible about the participants.
• When interpreting demographic differences, consider such conceptually relevant, and possibly confounding, factors as socioeconomic status, individual and family educational backgrounds, immigrant status, and place of residence.  Where possible include statistical controls.
• Make comparisons across more and less effective projects/programs. Factors common across effective projects/programs may also be common across ineffective ones.

Lesson Learned: Context is key.  Contextual factors include individual factors (e.g. age, race,ethnicity) and group factors (e.g. the local economy, available resources, location, changes in project/program leadership).  Evaluators need to:

• have knowledge and understanding of the contextual factors that are important to the evaluation;
• be aware of their own world view and the assumptions they make, especially in data analysis and interpretation;
• build relationships and trust with project/program staff and participants.

Rad Resource: BeyondRigor.org of course.  Its tips can help evaluators collect more accurate data, do more appropriate analysis and even be more apt to collect the right data.  It can also help evaluators better understand the role of context and explore some contextual factors that are important to consider in evaluations.

(Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

contextual factors · rigor

I’m Nicole Vicinanza, a Senior Research Associate at JBS International, and a member of the AEA Board.  As part of our work on the board, we seek input from AEA members and other stakeholders so that we can establish policies that help AEA as an association better succeed in our mission, attain our vision and live our values.  As we look at priority setting for the next year, it would be helpful for us to know how members and non-members connect (or don’t) with AEA.

Rad Resource: The AEA Web page is your portal for dozens of different ways to learn from, contribute to and connect with the AEA community.  If you haven’t already, visit the home page – almost every link will lead to a new way to connect with AEA.

Since you’re reading this post, I’m assuming you do connect with AEA in some way (at least through AEA365, or a colleague who forwards you AEA365 posts). However, there are lots of ways to connect – some available only to members, and some available to everyone.  For example, I connect to AEA through attending conferences, volunteering (on the Board, as a proposal reviewer for TIGs of which I’m a member, and at the conference), through LinkedIn, and (of course) through AEA365.  Other folks I know participate in local affiliates, contribute to AEA Journals or participate in the Thought Leaders Forum.

I consider AEA my primary professional “home” however many of my colleagues call other professions or disciplines their “home” but still use the AEA Library, eStudy courses or journals in their work.   Some are connected to AEA indirectly, through friends and colleagues who forward AEA365 postings, public access Coffee Break options or share AEA public statements.

Cool Trick: Sponsored by the AEA Board, enter this AEA365 drawing to win a $75 SAGE gift certificate.  To enter, all you have to do by midnight, December 10, in any time zone, is add a note to the comments section of this post that describes, 1) whether you consider AEA your professional “home” and 2) how you connect to AEA.

Hot tip: You don’t have to be a member of AEA to enter or win (but if you aren’t, wouldn’t you like to join?).  If you are receiving this via email, just click on the post’s title to return to this post on the AEA365 website and click on the word “comments.” One entry per person please (although you are welcome to comment on each other’s connections and ideas all you wish)! We will randomly select a winner from all those who add an on-topic comment by midnight December 10. The winner will receive a $75 Sage gift certificate.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor.

You’ve probably read about AEA’s LinkedIn page for fabulous free discussion about anything evaluation. Today, I want to highlight one particular discussion that has sparked a good deal of participation from a diverse group of evaluators.

Terminology has always been a sticky point for evaluators as those from different sectors (i.e. health, education, non-profits, government, etc.) have developed their own preferences and in many cases, definitions of terms.

This* discussion started by a Project Manager received 51 responses – not the longest discussion  – but nonetheless a rich and detailed investigation into these two terms. (*you will need to have a LinkedIn account to access the discussion)

Evaluators rang in on this from a variety of perspectives. Positions identified in the profiles included a:

• Research Analyst
• Prevention Specialist
• Independent Consultant
• Strategy and Planning Advisor
• Community-based Impact, Assessment and Evaluation Consultant
• Impact, Monitoring, Evaluation and Research Specialist
• Senior Public Engagement Associate
• Policy Analyst
• And several owners and presidents of research, consulting, or evaluation firms.

The discussion featured individuals who offered their own definitions of the two terms, after which several became engaged in a discussion of how these tools are used or should be used in practice.

Lesson Learned: Most commenters consider logic models and theories of change as related but distinct. Several indicate that theories of change are indeed embedded in logic models.

Here is how some commenters describe logic models:

• help identify inputs, activities and outcomes
• trace a flow of inputs through program activities to some sort of output or even on to outcome, and are usually intended as handy guides to program implementer
• visual model of how a program works
• represent the basic resource and accountability bargain between the ‘funder’ and the ‘funded’

Here is how some commenters describe theories of change:

• show how and why outcomes/activities cause change
• an attempt to make explicit the “whys” behind relationships or expected outcome
• explicit or implicit theory of how change occurs
• how one designs a program as it breaks out how and why the change pathway will happen
• work behind the scenes, and can be drawn from to assemble logic models

Rad Resources: Several commenters offer resources for exploring these concepts:

• EvalTalk
• sparksforchange  blog (my first two bullets below are from this blog)
• Purposeful Program Theory by Sue Funnell and Patria Rogers

I recommend these blog posts on the topic:

• Using a Theory of Change to Better Understand Your Program
• An Introduction to Developing Logic Models – Eight Key Elements to Include
• 52 weeks of BetterEvaluation: Week 12 Having an adequate theory of change

and this one on the topic of evaluation terminology:

• Evaluator, Watch Your Language

And finally, I must recommend Kylie Hutchinson’s tools for untangling evaluation terminology:

• A Working Typology of Evaluation Terms
• The Evaluation Glossary app

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

evaluation terminology · logic model · Theory of Change

Greetings from Washington, D.C.! My name is Kwamé A. McIntosh, member of the American Evaluation Association’s Graduate Education Diversity Internship (GEDI) 2012-2013 cohort. Today, I am eager to share with you my experience that I had as an Assistant Education Evaluator for the National Oceanic and Atmospheric Association’s (NOAA) Office of Education (OED).

As a scholar, I was given an opportunity to conduct a process and outcome evaluation of five years of statistical data about the effectiveness of two scholarship programs for undergraduate students to expand their knowledge of the atmospheric, oceanic, and/or environmental sciences. Since the students participated in their internships within the summer, I would never have the opportunity to interact or engage with the clients. This meant developing a relationship with staff members who worked directly with the population to gain a viable understanding of the program and its participants, though the staff members for this program were located at another NOAA location. Though each site is under the umbrella of the OED, I was still an evaluator who was responsible for unveiling the effectiveness of the program.  Though it took time, through responsive pro-activity versus reactivity, I gained the insight needed to provide practical deliverables while assisting in solidifying the bridge between both facilities.

Lesson #1: Trust is earned, not given. The process of gaining trust may be the difference between empowering your client or becoming the enemy.

Lessons #2: It is the evaluator’s duty to be purposeful in engagement, not the client. In Hazel Symonette’s “Walking Pathways Toward Becoming a Culturally Competent Evaluator: Boundaries, Borderlands, and Border Crossings”, she highlights the need to have multilateral self-awareness by asking “Who do those that one is seeking to communicate with and engage perceive the evaluator as being?” Many times due to social misunderstandings of the role of evaluator, clients tend to become distrustful towards us, which will ultimately result of in difficulty in conducting the evaluation and/or not gaining valuable insight to truly impact the client. This can be addressed by constantly removing “self” out of its box and being willing to reform “self” in another completely for the sake of utilizable evaluation outcomes.

collaboration · cultural responsiveness

Hello! I am Michelle Corbett, alumna of the AEA Graduate Education Diversity Internship (GEDI) program’s 2012-2013 cohort. For my internship, I worked at the Planning Council for Health & Human Services in Milwaukee, WI. A non-profit, community-based organization, the Planning Council offers planning, research, and program evaluation services to a wide range of organizations and community groups in Southeast Wisconsin.

The majority of my time was spent assisting Planning Council staff in the execution of a 3-year internal evaluation capacity building contract with a small non-profit that focuses on leadership and self-development of primarily at-risk African American and Latina girls. The Planning Council is committed to using participatory approaches in their work and during the course of my internship we engaged with not only organizational leadership but with front-line programming staff and program participants throughout the process of identifying relevant outcomes and selecting appropriate evaluation instruments and tools and piloting them. I would like to share some lessons I have learned from this experience.

Hot Tip: Do form an evaluation advisory group comprised of diverse individuals from the start. By including Board members, organizational leadership, program management, staff and participants, you will gain a more complete understanding of the organization, its programs, and its culture. This insight and the group’s ongoing guidance will help ensure that the capacity building process and its outcomes are relevant and culturally responsive.

Hot Tip: Don’t rush the process; building evaluation capacity is not a race. Take advantage of each opportunity to educate leadership and staff alike and make explicit evaluation’s relevance to the work each person does. Taking the time to do this will foster “buy-in” and increase the likelihood that newly implemented internal evaluation practices are sustained.

Hot Tip: Do be flexible and adaptive. Remember, building its internal evaluation capacity is not the only item on an organization’s daily To-Do list. Meetings may be cancelled and progress may appear to stall on occasion but you can take advantage of this time to work on other projects or, if appropriate, assist the organization in addressing the issues that are interrupting the capacity building process.

evaluation capacity · evaluation capacity building

Hello and Hola. We’re Nnenia Campbell, research assistant at the Natural Hazards Center, University of Colorado Boulder and Saúl Maldonado, research assistant at the ELLISA Project, University of California, Santa Cruz. We are recent graduates of the AEA Graduate Diversity Education Internship (GEDI). One of the GEDI goals is to: “Deepen the evaluation profession’s capacity to work in racially, ethnically and culturally diverse settings.” In the first of this two-part series, we shared a few suggestions for amplifying definitions of diversity. In this second-part, we share a few of the lessons we’ve learned that frame diversity as a measure of validity in culturally responsive evaluation.

Diversity informs validity. Cultural responsiveness is a critical feature of the evaluation process and increases the utility and truthfulness of results (Frierson, Hood & Hughes, 2002). Practicing cultural responsiveness in settings that are racially, ethnically, culturally or otherwise diverse requires considering cultural contexts as technical properties of validity. As evaluators, positioning cultural contexts as validity measures may include review procedures such as the accuracy with which programs serve diverse populations as well as how evaluators select instrumentation and how participants of diverse cultural backgrounds interpret items.

Additionally, cultural responsiveness is co-constructed between evaluators and evaluands. We must seek to identify and communicate the diverse worldviews that are represented in the evaluation process and, more importantly, how diverse beliefs, perspectives, behaviors and values can serve as “resource differences.” Such communications should be continuous, throughout initial, intermediate and final stages of evaluations.

Instrument Selection. When deciding upon which tools and instruments will be used as evaluative measures, it is important to recognize the diversity of our professional training. As much as possible, a range of recommended tools and instruments from multi-disciplinary perspectives should be shared with program participants and organizations.

Item Interpretation. When evaluative measures deemed appropriate are co-selected by evaluators and evaluands, determining the degree of item’s construct-relevance is necessary. Practicing cultural responsiveness requires being attentive to and appreciative of the possibilities for cultural variation. Including participants’ life contexts, prevailing socio-economic conditions, cultural communication and socialization styles in relation to how items are interpreted may reveal the need to, for example, modify measures’ number of items or adapt measurement processes.

Review Procedures. After selecting appropriate instruments and establishing items’ construct-relevance, inclusionary procedures to establish representation of program participants’ and organizations’ diversity is needed. Integrating the perspectives from culturally diverse panels in the interpretation of findings and presentation of results enhances the accuracy and validity of an evaluation.

Regardless of evaluation approach (i.e., participatory, utilization, systems) or method, being attentive and appreciate of diverse cultural contexts enhances both internal and external validity. To reframe diversity as a measure of validity, we recommend the following resources: Frierson, Hood and Hughes’ A guide to conducting culturally responsive evaluation as well as Basterra, Trumbull and Solano-Flores’ book, Cultural validity in assessment: Addressing linguistic and cultural diversity.

diversity