AEA365 | A Tip-a-Day by and for Evaluators

TAG | measurement

Hello! We are Silvana Bialosiewicz and Kelly Murphy from Claremont Graduate University. Working as Senior Research Associates at the Claremont Evaluation Center, we have often been tasked with analyzing large quantitative databases and drawing conclusions about program effectiveness based on our results. Today, we’d like to share some hot tips about pesky sources of bias that may be lurking in your data and provide some rad resources about how to uncover this bias.

Hot Tip: Conduct tests of Measurement Invariance on your evaluation surveys

We all know that the accuracy of the self-report measures we use to assess multi-dimensional constructs (e.g., self-esteem, organizational commitment) in our evaluations is contingent upon the reliability and validity of our measures, but did you know that these measurement properties are not always generalizable to the different populations and program contexts we find in our evaluations? For example, have you ever wondered…

  • If program participants from different cultures or socioeconomic backgrounds are interpreting your survey questions in the same way?
  • If a participant’s gender, age, or literacy level affects the way they respond to your survey?
  • If participating in the program changes the way participants think about your survey questions?

Answering questions such as these in a statistically rigorous manner helps us ensure that the comparisons we make (either across time or across groups) represent true differences in our constructs of interest!

Measurement Invariance:

What is it?

Measurement Invariance is the statistical property of a measurement that indicates that the same underlying construct is being measured across groups or across time.

How do we know if we have it?

When the relationship between manifest indicator variables (scale items, subscales, etc.) and the underlying construct are the same across groups or across time.

How do we test for it?

Because measurement invariance is too dense to cover in a single blog post, we have put together some rad resources to help you learn more about measurement invariance and the steps to assess measurement invariance.

Rad Resource #1: Based on our AEA13 demonstration session, we have put together an extensive resource packet for practitioners who are interested in learning more about measurement invariance and how to test for it.

Rad Resource #2: If you’re interested in learning more about the software used to assess measurement invariance, here is a link to a discussion thread on the strengths and weaknesses of available software.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi everyone!  I’m Yvonne M. Watson, a Program Analyst in U.S. EPA’s Evaluation Support Division (ESD) and Chair of AEA’s EPE TIG.  I’m primarily responsible for managing the division’s internal evaluation capacity building efforts. I read Moneyball: The Art of Winning an Unfair Game by Michael Lewis to understand the importance of a culture that values performance measurement.  The book focuses on the life of Billy Beane, former general manager of the Oakland A’s baseball team and his efforts to use unconventional stats to select members of his team.  Though I did not inherit my grandmother’s passion for baseball, this book helped me gain some important insights that can be applied to the measurement and evaluation of our programs.

Lessons Learned:

  1. Leadership and Using Performance Data and Evaluation Results.  Billy Beane used baseball stats to make decisions about the players that were drafted, traded or let go.  He also demonstrated the critical role of leadership in convincing organizational members to try a new measurement approach.  Our organizations have to take the next step in moving from merely collecting information to routinely analyzing performance data and using evaluation results to inform the decisions we make about our programs. Equally important is the task of asking the right questions and collecting the right data to accurately communicate performance.  These data can either tell a story of fact, fiction, drama, poetry, or comedy.
  2. Data Quality.  Michael Lewis notes that “…inadequate data led the people who ran major league baseball teams to misjudge their players and mismanage their games.”  Similarly, inaccurate and poor quality performance data and evaluation results can lead program managers and decision-makers to misjudge, mismanage, over-value or under-value programs.  Investing in the infrastructure to support good quality performance data and evidence needed for evaluation is crucial.
  3. Insiders and Outsiders. In baseball there are insiders (players, managers) and outsiders (statisticians).  Our organizations and programs can benefit from program insiders who have a wealth of subject matter expertise and knowledge about a program’s goals, clients and underlying assumptions and outsiders – analysts, evaluators etc. external to the program who bring a fresh  perspective at how to measure or evaluate a program.  These individuals can help broaden our thinking about our program’s design, measures, processes and procedures.
  4. Thinking Differently.  Billy Beane exhibited a willingness change things that no longer worked.  In short, he learned to adapt.  Likewise, our organizations – if they are to succeed, must adapt.  We must challenge ourselves not to rely on what we “know or see” but be willing to dig deeper, challenge the status quo and look at our problems, programs and solutions differently.

Rad Resource: Moneyball: The Art of Winning an Unfair Game by Michael Lewis.

The American Evaluation Association is celebrating Environmental Program Evaluation Week with our colleagues in AEA’s Environmental Program Evaluation Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi, I’m Jade Caines. I have taught prekindergarten through college grades for over 10 years and have also worked on numerous evaluation studies in the southern, western, and eastern parts of the U.S. I recently received a Ph.D. in Educational Research, Measurement, and Evaluation. Currently, I am a postdoctoral research fellow at University of Pennsylvania, where my research area includes the validity of scales and instruments used within education evaluations.

Lessons Learned:

I have been in many situations where people wanted to know whether or not something “worked.” Folks would slap together some questions, put it on Survey Monkey, and get some people to respond. But since my involvement in the Educational Measurement field, I have learned so much about survey design. The most critical lesson I’ve learned in creating surveys used within education evaluations is the importance of defining exactly what it is that is being measured. Often I’ve had conversations with clients where the “what” is taken for granted. I have learned to spend significant time probing, asking pointed questions about what they really want to measure. Eventually, clients realize that they may not be so sure about the one “thing” they want to measure. Questions, Questions, and more Questions have helped in defining a construct (that “thing”) and then operationalizing it (creating situations where that “thing” is represented) for survey design. A construct can be defined as the theoretical object of our interest in the respondent. For example, character in students may be a construct that evaluators may want to measure as a part of a character evaluation grant.

Hot Tips:

  • Name the “thing” you want to measure (e.g. perseverance in the classroom).
  • Create a list of situations/experiences where that “thing” is represented (e.g. a student resubmits a writing assignment three times for a better grade). This would be a list of indicators, or evidence that a certain “thing” exists.
  • Create a list of situations/experiences where the opposite of that “thing” is represented (e.g. a student chooses not to submit an assignment, despite multiple deadline extensions from the teacher).
  • Then decide how to represent these situations that span a continuum of that “thing” on a measurement tool (e.g. a survey).

Cool Trick:

Draw a vertical line where the top of the line represents a high amount of that “thing” and the bottom represents a low amount. Try to create a survey that has several questions in the high, medium, and low sections of that line.

Rad Resource:

Constructing Measures: An Item Response Modeling Approach by Mark Wilson (2005). The first 2 chapters are the most relevant.

Hot Tip: Take a minute and thank a teacher this week!

The American Evaluation Association is celebrating Educational Evaluation Week with our colleagues in the PreK-12 Educational Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our EdEval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

I am Phil Halbrook and I would like to share with you information about TRASI – a great tool to help evaluators seeking to measure social impact. The issue of measuring social impact is a hot button one right now and the TRASI database and website is extremely useful.

Rad Resource – TRASI: TRASI is “Tools and Resources for Assessing Social Impact” and it is from the Foundation Center. TRASI has over 150 indexed tools, each with a description of what it does and what entity developed it. Full disclosure – all of the resources there aren’t tools per se, they include best practice guides, methodological guides, and what we might more traditionally consider to be a tool. TRASI also includes a blog and discussion groups where you can discuss the resources available there or broader issues related to assessing social impact.

Rad Resources – Examples from the TRASI Database: So, what’s there?

Campaign Champions Data Collection Tool from the Annie E Casey Foundation: “This is a tool that measures strengthened base of public support. This form tracks and measures the number of “champions” (people who take actions to advance the public will) engaged and the actions these champions undertake as a part of the Born Learning campaign.”

Criteria for Philanthropy at Its Best: “This best practice provides guidelines on values…, effectiveness …, ethics …, and commitment … criteria for evaluating philanthropy.”

Interrupted Time Series Designs [Guide]: “This is an observational method that measures impact of education programs in cases where data before the implementation of the program is available. This method compares the data from before the implementation to the same data afterwards to tease out a trend in achievement.”

And 100+ others

Hot Tip – TRASI on Twitter: You can follow TRASI on Twitter at @FCAtlanta. This apst week, on November 14th, they held their first ever tweet chat on social impact assessment using the hashtag #socimp. If you are on Twitter, the conversation is still going – just search for the hashtag. Not on Twitter? No problem. This link http://hashtags.org/socimp will take you access to a record of the discussion to date with the #socimp hashtag – check it out for insights and resources.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

I’m John Dearborne and I’ve taken to reading thought leaders in measurement in Public Relations in order to improve my evaluation practice, in particular around measuring communications. Today, I’d like to share resources related to identifying measurement standards in social media.

Hot Tip – KD Paine’s PR Measurement Blog: Katie Delahaye Paine is an insightful PR Consultant who gets to the heart of measuring what matters. She focuses heavily on measuring communications success, in particular around social media and supplements her ongoing guidance by sharing great resources (a bit like aea365).

Rad Resource – CASRO Guidelines for Social Media Research: Earlier this month, Paine shared the guidelines from the Council of American Survey Research Organizations (CASRO) for research on social media. The guide was useful for its glossary in particular and discussion of ethical research in this emerging field of study.

Rad Resource – The Barcelona Declaration of Measurement Principles: This document from 2010 established a starting set of principles on which most other efforts appear to be building. It was developed at an annual Summit organized by the International Association for Measurement and Evaluation of Communication (AMEC) and the Institute for Public Relations (IPR).

Lesson Learned – New Coalition Forms to Develop Global Standards in Social Media Measurement: AMEC and IPR have just this fall joined with the Council of PR Firms (CPRF) to form a partnership to try to identify global standards. They are just beginning the conversation, but building on the Barcelona Declaration.

Rad Resource – Michaelson & Stacks Article on Standardization in Public Relations Measurement and Evaluation: Reading Intelligent Measurement, another great blog focusing on evaluation of communications, lead me to the Michaelson and Stacks article from Public Relations Journal which lays out succinctly the challenges of identifying common, agreed-upon measures across social media. The focus on measurement and types of measures applicable to social media helped me to think critically about what it means to measure engagement, communications, advocacy, and the like.

Rad Resource – The State of Setting Social Media Measurement Standards: Returning back to KD Paine, this article from The Measurement Standard (a PR Newsletter from KD Paine & Partners) tries to frame the issue and explore the many groups, often with agendas, with an interest in setting social media measurement standards. Interestingly, evaluators and social science researchers aren’t listed. She herself is bringing together “the conclave” an ostensibly nonpartisan group to attempt to identify standards.

Lesson Learned: This issue of social media measurement is definitely in flux with many parties working both collaboratively and potentially at cross-purposes in a race to establish measurement standards. Who will win, Betamax or VHS?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

My name is Susan Kistler. I am the Executive Director of the American Evaluation Association and author of aea365’s Saturday posts. Today, I wanted to call to your attention the Organization for Economic Co-operation and Development (OECD) Better Life Index (BLI).

OECD launched the BLI in May, describing it as a “new, interactive index that will let people measure and compare their lives in a way that goes beyond traditional GDP [gross domestic product].” The BLI compares 34 countries on 11 dimensions – housing, income, jobs, community, education, environment, governance, health, life satisfaction, safety, and work-life balance.

Rad Resource: Use the BLI at http://www.oecdbetterlifeindex.org/. Go ahead, click the link, and explore. The interactive format made me want to learn more about my own country, how it compared to other countries, and the basis for the ratings. Be sure to click on a specific country to find out about what went into the ratings, and to create your own index by giving different weightings to each dimension to see what happens to the graphic.

Lesson Learned: Exploring the BLI made me consider critical questions, in particular,

  • What measures really matter to stakeholders? An article from the designers, “Designing Your Better Life Index from a Methodological Perspective” expanded my understanding of the decisions that went into indicator selection.
  • How can we report on measures in ways that allows stakeholders to prioritize what is most important for them?
  • Would reporting similar to the BLI be feasible with resources that are more modest than those of the OECD and what tools might we use to make that happen? (consider adding your ideas via comments)

Lesson Learned: Those interested in data visualization may find the BLI valuable as a case study. It is sleek, customizable, and intuitive. The design has garnered considerable attention, and generally very positive reviews for both its accuracy and aesthetic.

Rad Resource: Mortiz Stefaner, one of the designers, talks through the design decisions and variations via this great video. http://ow.ly/5kOs5

Rad Resource: Bryan Connor, blogger at The Why Axis, (a must-read blog for those into thoughtful analysis of data visualization), critiqued the Better Life Index earlier this week. http://thewhyaxis.info/oecd/

Lesson Learned: The BLI is certainly not perfect. Limited only to 34, primarily developed, countries, a large portion of the world is left out. The designers note that a major limitation was identifying the data needed for a country to be included. Other observers feel that the 11 dimensions still cannot fully capture what is truly important to a populace, such as social networks that sustain relationships, and freedom of speech. A few of the critiques may be found here: http://ow.ly/5mD9S

The above is my opinion and does not necessarily represent that of AEA. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hello! This post is by Susan Hahn and Guy Sharrock. Susan works as a Monitoring and Evaluation (M&E) consultant to Catholic Relief Services (CRS), a Baltimore-based international humanitarian assistance and development agency, with whom Guy is a Senior M&E Advisor. Evaluations are an important part of our project activities, but they must be linked with good monitoring systems. This Tip is focused on how to establish a robust project M&E system.

Don’t forget: Reviewing data collected and analyzed in the project’s monitoring system gives the evaluator important information on project management, progress and achievements on lower level indicators. This information can be the basis of a scope of work for the evaluation.

CRS has developed an approach to setting up an M&E system for projects, called SMILER: Simple Measurement of Indicators for Learning and Evidence-based Reporting. While M&E is rarely thought of as simple, SMILER breaks down the development of an M&E system into easily understood parts. SMILER is an innovative, comprehensive, coherent and practical approach to making an M&E system operational; the objectives and their indicators are linked to a system to collect, analyze and report on data. The approach enables staff to turn the M&E planning documents in the proposal into a useful M&E system that can benefit all staff. The resulting system is documented in a project M&E Operating Manual.

The graphic below shows the documents included in the SMILER M&E system.

Hot Tips:

  • Set up your project M&E system in the early stage of project implementation.
  • Set aside a week to collaborate with a few project and M&E staff in developing a first draft of the SMILER M&E system; these staff should be very familiar with the details of the project.
  • Use all steps in the SMILER process for all projects; the size of the M&E system will depend on the size of the project.
  • Review the SMILER draft with other agency and partner staff who will be implementing the project and contributing to M&E. These stakeholders will have valuable comments and can help finalize the M&E system for use.
  • Develop a communication and training plan to roll out the M&E system, and review its operation on a regular basis.
  • Delegate a staff member to maintain the M&E Operating Manual that you should treat as a “living’ resource.

Rad Resource: the CRS Guide to creating a SMILER M&E system can be found at:

http://www.crsprogramquality.org/publications/2011/1/17/propack-iii-english.html

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

My name is Emily Warn and I’m a senior partner at SocialQuarry, a social media consulting group that plans and analyzes online networks and social media for non-profits, foundations, and public agencies.  Today I’m going to be sharing a hot tip about using social media in evaluations.

This past week Facebook surpassed Google in US visitors, proof—if you needed one—that social networking sites are rapidly transforming our offline friendship, family, and work circles to online communities.

It’s obvious to anyone who has tried to ignore ads on Facebook and Twitter that social media is changing how for-profit companies advertise, sell products , and build their brands.  A plethora of tools exist—it seems like new ones are announced every day—to measure a company’s return on their investment (ROI) in social media.  For-profit companies can measure the success of social media campaigns, search engine optimization efforts, customer conversions from browsing to buying, mentions on Twitter, etc.

Not-for-profit organizations are also investing in social media; most are dabbling in Facebook and Twitter, but they lack the tools to measure their (ROI) because, for the most part, their reasons for using social media are very different than those for-profit uses. For example, tools developed to measure the success of advertising campaigns to sell handbags, don’t always work to measure the success of advocacy campaigns to change policies.  Plus, people who participate in advocacy campaigns are passionate about their cause and more likely to check related websites and social networking sites with a regularity that for-profits can only dream of.

Hot Tip: Here are some of the reasons why social networks can be a holy grail for many non-profits.  Non-profits can use social networks to help them:

  • Increase capacity by using network to pool and share resources
  • Coordinate groups working on a common issue
  • Generate ideas and tap expertise to develop grant and advocacy strategies
  • Raise money for capital campaigns and causes
  • Increase a donor base and engage new donors
  • Identify leaders who can expedite learning and coordinate actions across a network.

Hot Tip: Using for-profit tools to measure non-profit outcomes requires defining fundamentally different key progress indicators (KPI). For example, commercial companies can use web analytics tools to measure engagement with customers. They can define a number of units sold as a KPI for an ad campaign and measure progress against that goal by analyzing how many customers clicked-through to their site and bought a product.  Further analysis can reveal on which pages customers abandoned  the process of stepping through an online shopping cart.  Improving those pages improves customer engagement.

Non-profits could define a KPI for engagement as the number of network members who participated in a in collective actions? Instead of stepping through  a shopping cart, web analytics tools could measure how many people stepped through a process to send an email to a legislature, or signed up for a newsletter to stay informed about an advocacy campaign or stay connected with key people or organizations.

I hope this gives you some ideas about how you can incorporate some online tools in your own evaluations!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · ·

Archives

To top