AEA365 | A Tip-a-Day by and for Evaluators

Search

Welcome to the final installment of the Design & Analysis of Experiments TIG-sponsored week of AEA365.  It’s Laura Peck of Abt Associates, here again to address some complaints about experiments.

Experiments have limited external validity

Experimental evaluation designs are often thought to trade internal validity (ability to claim cause-and-effect between program and impact) with external validity (ability to generalize results).  Although plenty of experiments do limit generalizing to their sample, there is good news from the field. Recent scholarship reveals techniques—retrospective analyses and prospective planning—that can improve generalizability. You can read more these advances in recent articles, here, here, and here.

Experiments take too long

Experimental evaluations have a bad reputation for taking too long.  Certainly there are some evaluations that track long-term outcomes and, by definition, must take a long time. That may be a criticism of any evaluation charged with considering long-term effects.  A recent push within the government is challenging the view that experiments take too long: the White House Social and Behavioral Sciences Team is helping government identify “nudge” experiments that involve tweaking processes and influencing small behaviors to affect short-term outcomes.  It is my hope that these efforts will improve our collective ability to carry out faster experimental research and extend the method to other processes and outcomes of interest.

Another reason experiments may take a long time is that enrolling a study sample takes time.  This depends on specific program circumstances, and it does not necessarily need to be the case. For example, the first round of the Benefit Offset National Demonstration enrolled about 80,000 treatment individuals into its evaluation at one time, with the treatment group getting a notification letter of the new program rules.  Such a change can be associated with large sample build up in a very short time.

Experiments cost too much

A rule of thumb is that evaluation should comprise one-tenth of a program budget. So, for a program that costs $3 million per year, $300,000 should be invested in its evaluation.  If the evaluation shows that the program is ineffective, then society will have spent $300,000 to save $3 million per year in perpetuity.  Efforts are underway to ensure that low-cost experiments become feasible in many fields, such as using administrative data, including integrating data from systems across agencies.

The Bottom Line

Experimental evaluations need not be more time-consuming or costly than other kinds of impact evaluation; and the future is bright for experimental evaluations to meet high standards regarding external validity.

This week’s-worth of posts shows that the many critiques of experiments are not damning when carefully scrutinized, thanks to recent methodological advances in the evaluation field.

Rad Resource:

For additional detail on today’s criticisms of experiments and others that this week-long blog considers, please read On the Feasibility of Extending Social Experiments to Wider Applications.

The American Evaluation Association is celebrating the Design & Analysis of Experiments TIG Week. The contributions all week come from Experiments TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, I’m Sarah Singer, Research Associate at Hezel Associates in Syracuse, NY. I am not an expert in cost analysis by a long shot, but have become increasingly aware of its potential as a component of a program evaluation, particularly for education programs. My post today offers my reflections and some things to consider, from one novice to another, to help you approach cost analysis.

My colleague Caitlin Griffin and I facilitated a discussion at Evaluation 2015 about the barriers to cost analysis in education evaluation, which include a lack of training, difficulty in quantifying educational outcomes, and a lack of perceived need from program implementers. These challenges not only hinder evaluators from being able to perform cost analyses, but may prevent us from collectively asking the right questions in order to understand if and when we should do them. Before we, as evaluators, tackle the logistics of how to do cost analysis, we need to ask ourselves if we can (and want to) convince our clients that it’s useful to them, taking into account the context and timing of each program.

Lesson Learned: I am of the opinion that cost analysis is a valuable tool for us to have in our bag of tricks. A hard look at the real costs of a program, as they relate to the outcomes, can be revealing in ways that can inform even better decision-making than if only outcomes were measured. Armed with real cost findings, a program implementer can understand the full resource requirements for a program; pursue the most cost effective program improvements; and describe a program’s impacts as they relate to cost, which is a language that policy-makers speak.

Rad Resource: For some inspiration and to see cost analysis in evaluation in action, check out the Robin Hood Foundation, located in NYC, whose aim is to fight poverty. Their approach to metrics emphasizes a commitment to funding the most impactful AND financially sound programs.

Rad Resources: The CBCSE Cost Tool Kit © 2015. This free resource guides you through the process of the ingredient method, in which you identify and add together all program costs. There are several examples to help you get started. It even does the math for you!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi! I’m Brian Yates. This is the fourth piece in a series of AEA365’s on using costs in evaluation. I started using costs as well as outcomes in my program evaluations in the mid-1970s, when I joined the faculty of the Department of Psychology at American University in Washington, DC. Today I’m still including costs in my research and consultation on mental health, substance abuse, and consumer-operated services.

Three other 365ers focused on evaluating costs, benefits, and cost-benefit of programs; there’s even more to cost-inclusive evaluation!

Lesson Learned: What if important outcomes of a program are not monetary, and cannot be converted into monetary units? Easy answer: do a cost-effectiveness analysis or a cost-utility analysis!

Cost-effectiveness analysis (CEA) describes relationships between types, amounts, and values of resources consumed by a program and the outcomes of that program — with outcomes measured in their natural units. For example, the outcome of a prevention program for seasonal depression could be measured as days free of depression. Program costs could be contrasted to these outcomes by calculating “dollars per depression-free day” or “average hours of therapy A versus therapy B per depression-free day generated.”

Hot Tip: How to compare apples and oranges. “But how can you compare costs of generating one outcome with costs of generating another? Cost per depression-free day versus cost per drug-free day?!” No problem: compare these “apples” and “oranges” by bumping the units up one notch of generality, to fruit. Diverse health program outcomes now are measured in common units of Quality-Adjusted Life Years (QALYs), with a year of living with depression as being worth substantially less than a year of living without depression. This and other forms of cost-utility analysis (CUA ) are increasingly used for health services funding.

Lessons Learned:

Insight Offered: It’s easy to dismiss using of costs in evaluation with “…shows the price of everything and the value of nothing.” Actually, cost-inclusive evaluation encompasses types and amounts of limited societal resources used to achieve outcomes measured in ways meaningful to funders and other stakeholders.

More? Yes! Lately I’ve gained better understanding of relationships between resources invested in programs and outcomes produced by programs when I work with stakeholders to also include information on program activities and clients’ biopsychosocial processes. More on that later.

Rad Resources:

Cost-effectiveness analysis (2nd edition) by Levin and McEwan.

Analyzing costs, procedures, processes, and outcomes in human services by Yates.

Want to learn more? Brian will be presenting a Professional Development workshop at Evaluation 2014 in Denver, CO. Click here for a complete listing of Professional Development workshops offered at Evaluation 2014. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

“But wait — there’s more!” Hi! I’m Brian Yates. Yes, this is the third piece in a series of “AEA365’s” on using cost data in evaluation … and not entirely inappropriate for your former AEA Treasurer to author. (I’m also Professor in the Department of Psychology at American University in Washington, DC.)

My past two 365ers (you can find them here and here) focused on evaluating costs of resources consumed by programs, and the monetary and monetizable outcomes produced by programs. With those two types of data, we can begin evaluation of the programs’ cost-benefit.

Lesson Learned – Funders should ask “Is it worth it?” and not just “How much does it cost?” The impulse to cut programs to meet budget goals actually can increase costs and bust budgets if the programs would have increased employment (and taxes paid), reduced consumers’ use of other services, or both. Evaluators can investigate this by comparing program costs to program benefits.

Hot Tip – Strategies for quantifying Value. “Is it worth it?” can be answered in different ways, including:

  • dividing benefits by costs (benefits/costs ratio),
  • subtracting costs from benefits (net benefit), and
  • measuring the time required before benefits exceed costs (Time to Return On Investment).

Each cost-benefit index describes a different aspect of a program’s cost-benefit relationship, and all can be reported in a Cost-Benefit Analysis or CBA.

Hot Tip Too – Costs, benefits, and cost-benefit relationships can be measured at several levels of specificity (such as for an individual in a program or for a new implementation of the program). Cost-benefit differences between programs as well as between consumers in the same program are important to understand, too.

Lesson Learned – Programs whose costs exceed measurable benefits still can be funds-worthy. A “news” story in humorous Daily Onion a few years back made an important point with its story, “Cost of living now outweighs benefits” (http://www.theonion.com/articles/cost-of-living-now-outweighs-benefits,1316/). Few of us would be inclined to act in a summative manner based on these “findings” (even if they were valid)! So may the value of many programs be underestimated or missed entirely when viewed through the “green eyeshade” of an exclusively pecuniary perspective.

Also, some programs provide services to which all people are legally entitled. Evaluations of these can instead ask, “What is the best way to deliver the highest quality services to the most people for the least money?”

Next time: how to include costs when evaluating nonmonetary program outcomes.

Resources: Text: Michael F. Drummond’s (2005) Methods for the Economic Evaluation of Health Care Programmes. 

CBA for environmental interventions

Want to learn more? Register for Evaluating and Improving Cost, Cost-Effectiveness, and Cost-Benefit at Evaluation 2014.

This week, we’re featuring posts by people who will be presenting Professional Development workshops at Evaluation 2014 in Denver, CO. Click here for a complete listing of Professional Development workshops offered at Evaluation 2014. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! My name is Matthew Von Hendy, I am the owner of Green Heron Information Services, which provides research and information services to evaluation professionals.  I love helping connect people with the information that they need to solve problems or make decisions.

Many evaluators are working in budget constrained environments which can make accessing the best high quality resources difficult. Fortunately, many excellent research resources are being made open access (free) or available at a reduced cost.  You just need to know where to look. I recently presented a brown bag session for the Washington Evaluators on this topic, “10 High Quality Low-Cost Research Resources for Evaluators.”

Some quick tips and suggestions:

  • Search engines. Most everyone makes extensive use of Google and it is the tool that evaluators use most often to answer research questions.  It is a very good resource but you need to be aware that Google filters your results meaning you may not be seeing important resources.  Search engines such as Duck Duck Go or Blekko can provide a way to make sure you are not missing something crucial.
  • Finding Full Text. Many evaluators are very familiar with the subset of Google that focus on academically oriented research, Google Scholar.  It is an effective tool for finding free full-text of articles.  I would suggest using the advanced search feature; you can find this by clicking on the downward pointing carat on the right hand side of the search box. Mendeley can also be a good source for finding free full-text of research articles.
  • Discounted rates. Some major research publishers offer deeply discounted rates to individual researchers for a short or limited amount of time. For instance, the American Psychology Association (APA) offers a 24 hour pass that allows access to several of their major databases for under twelve dollars.
  • Citation Managers. If you are searching and storing many citations you should not being using Word to do this function.  Many excellent free citation managers are available including Zotero, Mendeley and Endnote Web just to name a few.  Yes, it will take a little time to learn them but the payoff will be enormous in terms of time saved.
  • Databases. Just about every major research area has a high quality free database which evaluators can use:  the health sciences has PubMed, education has ERIC, and even transportation has a great resource TRID.  If you are working in the science and technology sector, you may check out my recent article on research resources in those fields.

The American Evaluation Association is celebrating Washington Evaluators (WE) Affiliate Week. The contributions all this week to aea365 come from WE Affiliate members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Jonathan Margolin, and I am a senior researcher in the Education Program at American Institutes for Research, where I work primarily in the State and Local Evaluation Center. One common challenge when evaluating the implementation of educational programs is to understand how the program is interpreted and adapted by teachers and schools. This issue is particularly challenging when the program is being implemented in dozens of sites across the country, where it is often not feasible to conduct in depth case studies or collect other implementation data. One low cost and highly efficient approach to capturing data on implementation is to provide teachers with online logs with which to record classroom activities. We used this approach in our recent evaluation of The CryptoClub, an informal program involving cryptography and mathematics (more information about the program is available here).

Lessons Learned:

  • Make the online log as simple and easy-to-use as possible. The activity leaders involved in the CryptoClub were typically operating in an afterschool setting with little time to fill out a lengthy log. To simplify things, we recommend you think about the key elements of the program to keep track of. For the CryptoClub program, the key elements were the cryptography and math topics addressed during the session. To determine these key elements, it is helpful to think forward to the evaluation report itself, and whether each element on the form would add appreciably to the report’s usefulness.
  • Track response rates and follow up with teachers. In an earlier phase of our evaluation, we used paper logs. By the time we had collected the logs, we discovered that many, if not most, of the activity leaders did not regularly complete their logs. By putting these logs online, however, we were able to track and monitor completion rates of each teacher. We were able to follow up with non-responders via email to provide a polite reminder to complete them.
  • Speaking of response rate, we realized that it was important to communicate our expectations clearly with activity leaders. We told them that they would need to complete logs for at least 80 percent of their sessions for us to have an accurate picture of their experiences. We used the data only from the activity leaders who reached this threshold.

Hot tips: For our log, we used a form in Microsoft SharePoint, but there are many highly accessible approaches to creating these logs. For example, in SurveyMonkey and other similar services, it is possible to set the survey to “kiosk” mode, to allow respondents to complete the survey multiple times. By asking the respondents to indicate the date and their program location, it is easy to generate compliance reports.

Rad Resource: See our handout from Evaluation 2013 for a step-by-step approach to developing logs that includes a picture of the online log and data summary table.

The American Evaluation Association is celebrating Chicagoland Evaluation Association (CEA) Affiliate Week. The contributions all this week to aea365 come from CEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Kelly Smith, and I’m an evaluator, policy analyst and economist at ECONorthwest, a consulting firm in Portland, Oregon. I’ve been a devoted member of OPEN (Oregon Program Evaluators Network), an AEA affiliate, since I was introduced to it in graduate school in 2007. OPEN works hard to provide our members with a variety of learning opportunities, including brown bag lunch talks, workshops, networking events, conferences, and a book club. We do this mostly by tapping into our own members’ expertise. Until recently, I’d been instrumental in planning these events, and I’d certainly been an enthusiastic participant, but I hadn’t yet felt comfortable leading one.

This past year, I overcame my reticence and agreed to give a talk about cost analysis, a topic that seems to raise fear and confusion, if not hackles, among many evaluators. The event was free and the room was packed, far exceeding our expectations. Evidently, people were hungry for knowledge about this subject. Some attendees even expressed surprise that we didn’t charge for such a valuable class (lesson learned!). Having sensed a large untapped demand, we decided to offer two sessions about cost analysis at our annual conference in the spring, and both were well attended. We got great feedback about the usefulness of this topic.

These sessions were my first experiences “teaching”, and I found it both intellectually challenging and valuable (and if I’m honest, a bit nerve-wracking). I had to study the topic to refresh my knowledge, practice public speaking and presentation skills, and think on my feet. I’m confident that I got more out the experience than the attendees.

Get involved! If you’re a member of your local affiliate, don’t hesitate to step up and contribute your own knowledge! There is a real demand for learning opportunities, and certainly not enough supply. Your colleagues will benefit from your willingness to share, your affiliate will become stronger and more active, and you’ll come out ahead, too.

It’s hard to understate the value of learning from each other as evaluators. Not only do we grow as professionals when we contribute to our colleagues and our affiliate, but we expand the value and reach of the evaluation profession as a whole.

Rad Resource: If you are interested in learning more about cost analysis in evaluation, you’ll find a copy of the slides used at the conference sessions here.

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

By Gert K Nielsen, used with permission

I am Susan Kistler, the Executive Director of the American Evaluation Association and the owner of iMeasureMedia. I usually don’t mention the latter on aea365, but today I want to be clear that I’m wearing my iMeasureMedia hat since AEA does not endorse particular products or services.

Last week, at Evaluation 2012, I gave a presentation on 25 low-cost/no-cost tech tools for Data Visualization and Reporting (thank you to the DVR TIG for its sponsorship of the session). With Paloma Faith crooning Do You Want the Truth or Something Beautiful in the background, we opened the session and took a look at tools that hopefully help us to merge truth and beauty.

Rad Resource: Download the full slidedeck from the AEA public eLibrary here.

Hot Tip: Consider downloading the pdf version where the notes are more readily accessible – the notes include the URL links for each item, cost information, and tips.

Rad Resources: So, what tools did we explore?

1. VisualJournalism.com (and Paloma Faith – Do you want the truth or something beautiful?)

2. EndlessYouTube

3. Official Seal Maker

4. Wallwisher

5. Amazing Multicolor Search Engine

6. ColorZilla

7. Office Timeline

8. amaztype

9. SnagIt

10. Intentional Data Visualization & Reporting Blog

11. Poll Everywhere

12. Pinterest

13. aea365

14. Prezi

15. Tagxedo

15. Tagxedo

Here are a few bonus items, included in the slide deck with notes:

16. CartoDB

17. Storify

18. PicturePalette

19. Color Palette Generator

20. Lovely Charts

21. fiverr

22. Pixton

23. Tableau Public

And, here are tools recommended by attendees at the session, also included in the slide deck with links and notes – and we’re soliciting aea365 posts from each one in hopes of learning more.

24. visual.ly

25. Photosynth

26. GIMP

27. amCharts

28. PDFill

Stay tuned for individual demonstrations on a number of these in the coming weeks, and thank you to all who attended the session.

The above represents my own opinions, and not necessarily that of AEA. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Susan Kistler, the American Evaluation Association’s Executive Director and regular aea365 Saturday contributor. I wanted to take today and build on Marc Smith’s post from yesterday regarding NodeXL. Marc has been kind enough to do a bit of coaching to help get us up to speed on the use of this free tool, and I’ve learned as well from earlier posts from Johanna Morariu, and Shelly Engelman and Tom McKlin.

Rad Resource: If you would like to learn a bit more about interpreting the network graph found in Marc’s post, I wrote about doing so in an AEA newsletter article, the content of which may be found here.

Rad Resource: I was honored to attend the Minnesota Evaluation Studies Institute this past week and to give an updated version of a session I have offered before with LaMarcus Bolton on 25 low-cost/no-cost tools for evaluation. It included NodeXL as well as (over) 25 other tools that you may find useful in your evaluation practice.  The slide deck may be found here in AEA’s public eLibrary, along with over 1000 other great resources!

Rad Resource: Marc mentioned the book Analyzing Social Media Networks with NodeXL that he co-authored. It is extremely useful if you are hoping to use NodeXL in your work.  I just finished it, and it answered so many of the questions I had about both Social Network Analysis (about which there are other books) and NodeXL (about which this is the only book!). For me, the book:

  1. Clarified the language of social network analysis
  2. Explained how SNA may be applied to social media with concrete examples of when and how to do so
  3. Helped me to understand how to analyze network maps and use them to provide actionable information

Hot Tip – Win a Copy of Analyzing Social Media Networks with NodeXL: In the comments for this post, share your favorite low-cost/no-cost tool for evaluation that is NOT already represented in the slide-deck and tell us why it is great. We’ll draw one response at random to win the book from among those submitted on or before April 15, and who knows, maybe your suggestion will be added to the next 25LCNC presentation as well!

Hot Tip – Coming Soon! I am working with a colleague to create a set of network maps in NodeXL that may be used as templates. Each will include a sample map, information on how to interpret what is there, and a step-by-step guide for downloading it and customizing it to apply to your own social media analysis project.

The above represents my own opinions and not necessarily those of the American Evaluation Association.

· ·

My name is Susan Kistler. I am the American Evaluation Association’s Executive Director and I contribute each Saturday’s aea365 post. At AEA’s annual conference in Anaheim earlier this month LaMarcus Bolton and I gave a demonstration of low-cost/no-cost tools for evaluators. We shared great tools that we’ve used, ones recommended by colleagues via aea365, and also got great suggestions from the audience.

Rad Resources: The PearlTree below has links to over 40 low-cost, no-cost tools.

Hot Tip – To get the most from the PearlTree: Click back to the aea365 website  and find the November 19 post if you are reading this via email (yes! It’s worth the click), then click on any of the Pearls to connect to that resource. you’ll notice that some of the pearls have little trees on them – that shows that they are nodes for subtopics. For instance, click on the pearl labeled “Free Photo Sites” to find links to multiple free photo sites as well as to two previous aea365 entries about free photo sites. Use the little [ -……..*……..+ ] slider at the bottom right to change the size of the pearls for easier reading.

Want to see this PearlTree full size? You can view it here on the PearlTrees website (you may need first to close the PearlTrees encouragement to download its iPad/iPhone ap).

Hot Tip: Many of the tools were recommended by the audience (check out the ‘Attendee Recommendations’ pearl node). We seem to have misplaced one of the flipchart pages with audience suggestions – if yours isn’t up there, OR if you just have a great low-cost/no-cost tool to recommend, add it to the comments please and we’ll grow this resource. We’ll also be reaching out to colleagues to get aea365 posts related to some of the tools for which we don’t yet have fuller narratives. See one up there without a linked aea365 post and that you use regularly? We’d love to hear from you!

Hot Tip: PearlTrees make a fun, green, easy, way to share materials related to a presentation or to a project. Joining PearlTrees is free and it is super easy to use.

Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Older posts >>

Archives

To top