AEA365

Poster Week: Karen Zannini Bull on Making Good Evaluation Decisions

Susan Kistler — Fri, 10 Sep 2010 08:00:04 +0000

I am Karen Zannini Bull, the Assistant Director of Distance Learning at Onondaga Community college and doctoral student at Syracuse University in Instructional Design, Development and Evaluation. My interests reside in Evaluation Theory and what comprises a good evaluation decision. With that in mind, I wanted to share lessons learned and a great resource for evaluators.

Lesson Learned: Many factors contribute to making a good, sound, quality evaluative decision. There is no single formula or recipe to conduct a successful evaluation. Each evaluation decision differs based on many factors including the goals, mission and vision of the organization, the stakeholders involved, the resources allocated (grant monies or otherwise), and the evaluators conducting the evaluation.

Sub Lesson Learned: Even though an evaluator may conduct many evaluations, this does not mean the evaluator will necessarily make the same decision twice. An evaluation conducted for a local school district yielded suggestions for ways in which the district could save thousands of dollars without cutting a single staff member. But would this same evaluation yield different results if the stakeholders had not emphasized the importance of retaining all staff members? Each situation is unique and each group of stakeholders varies. These factors influence change with regard to the variables and data considered when the evaluator makes a decision. With each variable variation, an alternate decision may be made.

Sub Lesson Learned: Never underestimate the power of outside forces such as time and cost. These two components can have a major impact on decision making. It is possible that an evaluator may make a different decision if given just one more week to collect data or two more days to look for themes across participants. What if an evaluator was given more funds to assist with manpower necessary for the given evaluation? Or what if, the client demanded results prior to the agreed upon deadline?

Sub Lesson Learned: Experience counts. If a novice evaluator and experienced evaluator conduct the same evaluation, with the same participants, the same context and stakeholders, side-by-side, the conclusions drawn in the end may be very different from one another. An experienced evaluator has vast knowledge of what works and doesn’t work, experience identifying themes that are subtle and an intuition about practicing evaluation.

Resource: Provided above was a synopsis of broader themes and issues that are apparent when conducting evaluations and making decisions. A great resource to discover more about evaluation practice and the nuances of decision making is Evaluation in Action: Interviews with expert evaluators by Fitzpatrick, Christie and Mark (SAGE, 2008). In short, “evaluation practice, as any professional practice, is concerned with subtleties, nuances, or larger shades of difference in how evaluators behave during a study” (p. 355).

Want to learn more from Karen or have a chat with her about her work? Attend the poster exhibition this November at Evaluation 2010!

Poster Week: Anna Douglas on Using Asynchronous Discussion Groups for Evaluation

Susan Kistler — Thu, 09 Sep 2010 08:00:31 +0000

Hi. My name is Anna Douglas and I’m a PhD student in Evaluation, Measurement, and Applied Research Methods at Purdue University. I’ve been using asynchronous discussion groups online to evaluate a new learning tool.

Online asynchronous discussion groups are discussions through an online media where participants are responding to each other, but not simultaneously. While using these groups will not work for all evaluands, there are several situations in which the method makes sense. If you are interested in collecting qualitative data from a diverse group of individuals without being constrained by time or location, this evaluation tool is particularly useful.

Similar to traditional focus groups, participants respond to comments from other participants, and the evaluator can also ask follow-up questions. However, because the discussion is happening in an asynchronous manner, participants have time to think about the question and others’ responses before posting their response. They are able to develop their thoughts more fully, rather than responding immediately after the question is asked. The groups also provide an opportunity for participants to hear feedback from members who may be reluctant to share information in a face-to-face group setting.

One word of caution, while other disciplines have researched the use of online discussion groups (as an instructional tool), there does not appear to be much research on using these groups as part of the evaluative process. Until its efficacy as an evaluation tool has been established, it would be good to use asynchronous discussion groups as one source of evidence, not your only source.

Hot Tip: When setting up an asynchronous discussion group, it pays to work with the online media that is already in place, if possible. For example, most universities have online communication tools, such as WebCT and Blackboard. A webpage could also be established for participants, set up similarly to discussions on blogs. Social network sites could be utilized as well (isn’t Facebook just a lot of asynchronous discussions?!).

Hot Tip: Think about what rules should be in place for your evaluation needs and clearly state them for participants. For example, does it make sense for participants to have anonymity or be known? (there is some research indicating that anonymity brings out more bad behavior in online discussions, so participants may come across more strongly than if in person.) How long will you give participants to discuss? Should participants read each other’s posts and respond directly to each other, or just answer the question asked?

Rad Resource: If setting up a website for the discussion, Gravatar can be used to identify users on websites. The Gravatar website has tutorials on setting this up. The Genuine Evaluation blog utilizes this in their discussions as does aea365.

Anna will be presenting her work on using asynchronous discussion groups as part of the poster exhibition at AEA’s annual conference this November in San Antonio. Join us at Evaluation 2010 to connect further wish Anna and over 1000 other presenters!

Poster Week: Shobha Mittal on Seeking a Higher Survey Response Rate

Susan Kistler — Wed, 08 Sep 2010 08:00:51 +0000

My name is Shobha Mittal and I am an independent consultant. Seeking a higher response rate on the surveys has always been tricky in the field of evaluation.

Lessons Learned: Although there are many good references available to help evaluators impact the response rates, I have found that thinking like a respondent before seeking their response improves the likelihood of success. From a respondent’s perspective there are two major concerns (Besides of course having the inclination or time to respond). First is the technological comfort of responding to a questionnaire electronically versus the hardcopy response, and the second is the fear of consequence which may follow an honest response.

The response seeker must address the respondent’s need prior to expecting them to accommodate his/hers.

Hot Tip: Using a mixed mode survey strategy and providing both the paper survey method for survey response with an additional option of an electronic response may address both concerns of the respondents. Some respondents may not be assured by the confidentiality of the survey process and sometimes perceive the fear of being identified through their email IDs. The paper survey option will encourage such respondents to use the paper survey instead of a no response. Conversely, some respondents may find dealing with paper and pencil difficult with the technological advancement and may prefer using the electronic version instead of a no response. Providing a mixed mode response method is likely to help respondents take the survey than to ignore it. The mixed mode methodology requires extra time, efforts and resources. However, such costs may not outweigh the benefits.

Shobha will be presenting her work on mixed methods survey research as part of the poster exhibition at AEA’s annual conference this November in San Antonio. Join us at Evaluation 2010 to connect further wish Shobha and over 1000 other presenters!

Poster Week: Kristy Jang on Resources for Data Collection

Susan Kistler — Tue, 07 Sep 2010 08:00:23 +0000

My name is Kristy Jang, and I am a Master’s student at the University of British Columbia, Canada. I am interested in evaluating educational programs in developing countries and professional development programs in higher education. Since last fall, I have had the pleasure to help evaluate a graduate-level professional development program which is based on a group-mentoring model. As part of our evaluation, we are looking at the development of social networks among stakeholders (e.g., trainees, mentors) and trainee achievements. In collecting data on these topics, we have encountered some helpful resources:

Rad Resource: First, Lime Survey is a free online survey development tool: www.limesurvey.org. You can collect data via e-mails with a link to the online survey, and the responses are automatically compiled within the server. You can export the data in different file formats (e.g., Excel, SPSS) and obtain data summary statistics and graphs. With regards to collecting social networks data, the most useful feature was “array_filter” function, which allowed us to ask questions with response choices that were filtered out from a previous question. For example, we first asked, “Which of the following people do you interact with?” Then, the respondent’s choices became the possible answer choices of the subsequent questions such as, “Whom do you ask for help when you have a challenging problem in your research?” and “Who do you talk to when you have a new innovative idea?” The social networks data were analyzed through UCINET software, which allowed us to visualize interactions among stakeholders as sociograms in three-dimensional space.

Rad Resource: Next, Google Docs is a helpful resource for simpler survey questions: www.gmail.com. Although it does not have complex features such as the “array_filter, it is more user-friendly with 71 design choices. Thus, it is more suitable for beginners and helpful for increasing response rates. Moreover, Google Docs Forms helps enhance communications among the evaluators as documents can be stored within the server and jointly edited by anyone who has access to the g-mail account.

Rad Resource: Adobe Acrobat 9 Pro software helps you collect information that could be better presented in a table rather than a survey format (e.g., a list of workshops and conferences respondents attended, including date, title, place, and the respondents’ roles). The software works similarly as the other online survey development tools. It sends out an e-mail with a link to the document, and when responses are submitted, it automatically compiles them in your computer as a separate file. Make sure that the PDF file you created is set to be writable, allowing respondents to save data in the document using Acrobat Reader – from the “Advanced” on the menu bar click on “Extend features in Acrobat Reader.”

Want to learn more about Kristy’s work? Join us at the American Evaluation Association’s Annual Conference, Evaluation 2010, in San Antonio this November and check out the poster exhibition on Wednesday evening.

Poster Week: Kai Chi Yam on Using Exploratory Factor Analysis in Evaluation

Susan Kistler — Mon, 06 Sep 2010 08:00:29 +0000

My name is Kai Chi Yam, and I am a graduate student at Washington State University. I recently had an opportunity to evaluate a university-based mentoring program. Exploratory Factor Analysis (EFA) was used to provide score validity evidence through the examination of internal structure of study-specific measures. Such evidence can increase the utility of the instruments in evaluation, and ultimately increase the creditability and utility of evaluation results. While working with my colleagues, I found that some evaluators may be familiar with EFA, but most follow software default settings with little consideration of different procedures that can affect the results of EFA. Here are some general guidelines for appropriate use of EFA.

Hot Tip: Follow these steps when conducting an EFA:

1. Data screening: Check Chi Square (p > .05), Kaiser-Meyer-Olkin measure of sampling adequacy (MSA > .70), Bartlett’s test of sphericity, univariate and multivariate normality to determine if the data is appropriate for factor analytic procedures.

2. Extraction methods: Use principal component analysis if the rationale of EFA is purely data reduction. Use principal axis factoring if the rationale of EFA is to extract latent variables.

Note: Principal axis factoring is preferred because it takes measurement error into account.

3. Rotation methods: Use direct oblimin or promax rotation when factors are non-orthogonal. Use varimax rotation when factors are orthogonal.

Note: In most evaluations, especially when the measurements are psychological in nature, evaluators should assume factors to be non-orthogonal and proceed with direct oblimin or promax rotation.

4. Criteria for retaining factors: Decide the number of factors based on at least two criteria (e.g., scree plot, parallel analysis, conceptual meaningfulness of the factor) other than Kaiser’s rule of eigenvalue > 1.

5. Interpretation: Name each factor with theoretical (i.e., supporting theories) and statistical justifications (i.e., factor loadings). A general rule of thumb for acceptable factor loading is .40 or above.

Note: All interpretations must be theoretically justifiable! Don’t base your judgment solely on factor loadings.

Rad Resource: Making Sense of Factor Analysis: The Use of Factor Analysis for Instrument Development in Health Care Research. (ISBN: 0761919503) This introductory textbook is easy to follow and requires minimum knowledge in statistics and math.

Exploratory factor analysis can be a useful tool in evaluation when study-specific measures are employed. Please note that these are general guidelines, not definitive rules. Evaluators should consult with methodologists, textbooks, or journal articles before attempting EFA.

Want to learn more about Kat Chi’s work using EFA? Join us at the American Evaluation Association’s Annual Conference, Evaluation 2010, in San Antonio this November and check out the poster exhibition on Wednesday evening.

Poster Week: Virginia Dick on Using Geographic Information Systems (GIS) in Evaluation

Susan Kistler — Sun, 05 Sep 2010 08:00:24 +0000

My name is Virginia Dick and I am currently public service evaluation faculty at the Carl Vinson Institute of Government at the University of Georgia. Most of my work focuses on assisting state and local government agencies, and other university divisions, with evaluation of programs, policies and systems.

As part of my role I often find myself working with a wide range of individuals with different backgrounds, perspectives, purposes, and information assessment styles. It has been important to find ways to help different groups examine and understand relevant evaluation data using a wide range of mechanisms.

Most recently, I have begun working with our state child welfare agency to use GIS (Geographic Information Systems) methods to examine child welfare client characteristics and outcomes spatially through mapping. Often key stakeholders (community members, agency leadership, and social work students) have expressed new and interesting perspectives and interpretations of the data when it is portrayed via mapping rather than in traditional charts and tables.

Rad Resource: ESRI (http://www.esri.com/) often provides free training and educational opportunities to work with their mapping software and may be available through some universities.

There are many open source software options out there, some of which I am currently working with at the University of Georgia Information Technology Outreach Service to explore with my current project. A list of open source options is available at: http://gislounge.com/open-source-gis-applications/

Hot Tip: When working with a group reviewing the data and relationships between variables, start with a few layers and options on the map and slowly build and add additional components as the individuals start to become more comfortable talking about the relationships between the different variables.

Hot Tip: By looking at census tracts as units it allows groups to discuss the relationship between variables without having to dig down to the individual street address level which can become much more complicated when compiling the maps. Often analysis at the census tract level can be most beneficial to communities and government agencies rather than the individual street address level.

Hot Tip: Let the stakeholders generate the ideas and discussion among themselves to get the richest information about the perceived relationship between variables. This is particularly useful when looking at small units such as counties or smaller (with the mapping done at the census tract or block level).

Want to learn more about Virginia’s work using GIS? Come to the poster exhibition on Wednesday evening in San Antonio this November for AEA’s Annual Conference.

Susan Kistler on Data Visualization Part II

Susan Kistler — Sat, 04 Sep 2010 08:10:30 +0000

My name is Susan Kistler, the American Evaluation Association’s Executive Director, and I contribute each Saturday’s post to aea365. One hot topic for me this year is data visualization – representing data in ways that are accurate, accessible, and appealing. My very first aea365 post identified resources for those with interests in data visualization – including the classic Periodic Table of Visualization Methods. Today, I want to provide an update.

Hot Tip: Stephanie Evergreen (who gave Great Tips for Graphic Design on July 16) is working to bring together evaluators, and those working in related disciplines, who have an interest in data visualization and reporting in hopes of forming an AEA Topical Interest Group. The group would strive to build capacity and expand the knowledge base in the evaluation field in order to expand stakeholder understanding, improve interpretation, and increase use of evaluation results. As a starting point, Stephanie is building an emailing list of interested individuals and if you would like to be on the list, add a comment to this post (click through back to the website if you received this via email). She’ll also be hosting an informal meeting at Evaluation 2010!

Rad Resources: I attended a presentation in Boston given by the wonderful team at juice analytics as part of their Viva Visualization tour. The presentation was free, definitely worth the 90-minutes of my time, and gave me great ideas for improving my own reports. They’ll be coming to Washington on September 16 if you are in the area. If you aren’t in the DC area, and even if you are, you can learn from their blog – some of the best content can be found on their visitor’s guide and you can subscribe from that page as well. A couple of my favorites? Check out the post on Lightweight data exploration in Excel (under Excel Tricks) to make super-easy inline bars and Stimulus Bill Explorer (under demos) to see an interactive tree map in action.

Rad Resource: The vizthink group on LinkedIn (you’ll need to join LinkedIn if you aren’t a member but it is free) is a great place to learn from others, post questions, and gather feedback and suggestions.

Hot Tip: Nancy Duarte, author of slide:ology, a book recommended by John Nash in his April 9 aea365 post on Creating Outstanding Presentation Slides. Nancy has a new book coming out – resonate: Present Visual Stories that Transform Audiences due on September 28. She has also created a series of short new videos on creating outstanding presentations, some of which may be found individually on Amazon, but as a set they were uploaded just this week to Vimeo for free viewing.

The above represents my own opinions and not necessarily those of AEA. This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Nicole Jackson on Improving Interview Techniques During Formative Evaluations

Susan Kistler — Fri, 03 Sep 2010 08:00:45 +0000

Hello, my name is Nicole Jackson. I am both an adjunct faculty in the Human Resource Management Certificate program at U.C. Berkeley Extension and a doctoral candidate in Policy, Organization, Measurement, and Evaluation at U.C. Berkeley’s Graduate School of Education. From my previous and current work, I discovered that interviewing is both an art and a science especially when it is used in more formative evaluations. Although considered important, interviews are prone to researcher bias that can impact data collection and reporting. Below I offer some tips to help mitigate forms of research bias during interviews.

Hot Tip #1: Understand how different interview formats may alter findings. The two general categories of interview formats include invidual versus panel interviews and unstructured versus structured interview scripts. Individual or one-on-one interviews as well as unstructured or loose ended-scripts are the most prone to researcher bias. Both of these formats lend easily to loss of control due to different personality types that can affect information collection. Where possible, try to use multiple interviewers or a small panel with a structured interview script to help mitigate and triangulate real-time interview data. Structured interview scripts should always focus on the critical research questions during an evaluation project.

Hot Tip #2: Tailor question types according to personality type and experience level. A variety of question types exist to help evaluators navigate difficult and shy personality types as well as those participants with more or less knowledge and experience. Where possible try to use more open-ended, situational questions with follow-up probes for more shy personalities and those participants with more knowledge and experience. For more difficult personalities, begin with more close-ended (e.g., yes/no) questions and then transition to open-ended question prompts in order to maintain control and focus during the interview.

Hot Tip #3: Never underestimate the role of the interview environment. Nothing can be as frustrating as a distracting interview environment. Always conduct interviews in a quiet, private location with good lighting, appropriate room temperature, and minimum distraction. Have water ready to go to place participants at ease. When using recording technology, always consider Murphy’s Law and have extra notepads and recorders ready on hand. Test all recording equipment during the first two minutes of the interview as a safe-guard.

Hot Tip #4: Be mindful of both verbal and non-verbal language. Experts on interviewing claim that non-verbal communication is just as important as verbal behavior in evaluating the trustworthiness of data. Be aware of how your own body language and those of your participants can alter data collection and assessment. Never use closed poses such as crossed arms while interviewing, which is a sign of defensive behavior. Also, be mindful that non-verbal behaivor is culturally influenced.

Nicole will be conducting a roundtable at evaluation 2010 on improving methods of inquiry to incorporate diverse views and perspectives. Join Nicole and over 2500 colleagues at AEA’s Annual conference this November in San Antonio.

Monica Hargraves on Supporting Evaluation Practice in Organizations

sgrant — Thu, 02 Sep 2010 08:00:57 +0000

My name is Monica Hargraves and I work with Cooperative Extension associations across New York State as part of an evaluation capacity building effort in the Cornell Office for Research on Evaluation (CORE). My work with Extension is shaped, in part, by insights we gained through a Concept Mapping research project we did in late 2008. We wanted to explore, from practitioners’ perspectives, what factors contribute to supporting evaluation practice in an organization.

We used Concept Mapping software from Concept Systems, Inc. to gather ideas in response to this prompt: “One specific thing an Extension organization can do to support the practice of evaluation is …” Contributors included county-based educators and Executive Directors, as well as state-level Extension administrators and Cornell staff. The raw ideas were pared down to a working set of 80, and then participants sorted the ideas into clusters and rated them on two criteria: potential for making a difference, and relative difficulty.

The individual ideas become points on a “Cluster Map” that gives a visual representation of how participants conceptualized the patterns and themes in ideas (see below. For information on the Concept Systems technology and the statistical techniques that underlie it, see www.conceptsystems.com.) The ratings are useful for thinking strategically about what to do give priority to when trying to improve and sustain evaluation practice in organizations.

Rad Resource: For more detail on the study, including a handout with the individual idea statements and their ratings on potential difference, see http://core.human.cornell.edu/AEA_Conference.cfm#2008

Cluster Map of Ideas in Response to the Prompt: “One specific thing an Extension organization can do to support the practice of evaluation is …”

Lessons Learned:

Technical assistance and training are not enough! The top-rated cluster in terms of potential for making a difference was “Communicate the Value of Evaluation.” The ideas there included educating organization leaders, staff, and volunteers on the importance of evaluation (not the how-to), using evaluation results well and demonstrating how they lead to better programming, having an evaluation champion in-house, making evaluation results easy to understand and user-friendly.
Communication is important. Communication should be used to motivate evaluation and build organizational commitment to it, and as a practical tool for sharing what works, fostering collaborations, and saving time.
Leadership and Structure matter. The second and third most important clusters were “Set Expectations and Requirements” and “Integrate into Organization Structure”. Respondents wanted clarity and consistency, and to have evaluation woven into a wide range of organization functions and practices.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Michelle Jay on Lessons Learned in Acquiring Stakeholder Feedback

Susan Kistler — Wed, 01 Sep 2010 08:00:06 +0000

My name is Michelle Jay and I am an Assistant Professor at the University of South Carolina. I am an independent evaluator and also an evaluation consultant with Evaluation, Assessment and Policy Connections (EvAP) in the School of Education at UNC-Chapel Hill. Currently I serve with Rita O’Sullivan as Directors of AEA’s Graduate Education Diversity Internship (GEDI) program.

Lessons Learned: A few years ago, EvAP served as the external evaluators for a federally-funded Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) state-wide grant housed at University of North Carolina (UNC) General Administration. Part of our work involved assisting project coordinators in 20 North Carolina counties to collect student-level data required for their Annual Performance Review reports as well as for program monitoring, assessment, and improvement. For various reasons, project coordinators experienced numerous difficulties in obtaining the necessary data from their Student Information Management Systems (SIMS) administrators at both the school and district levels. As collaborative evaluators, we viewed the SIMS administrators not only as “keepers of the keys” to the “data kingdom,” but also as potentially vested program stakeholders whose input and “buy-in” had not yet been sought.

Consequently, in an effort to “think outside the box,” the EvAP team seized an opportunity to help foster better relationships between our program coordinators and their SIMS administrators. We discovered that the administrators often attended an annual conference each year for school personnel. The EvAP team sought permission to attend the conference where we sponsored a boxed luncheon for the SIMS administrators. During the lunch, we provided them with an overview of the GEAR UP program and its goals, described our role as the evaluators, and explained in detail how they could contribute to the success of their districts’ program by providing the important data needed by their district’s program coordinator.

The effects of the luncheon were immediate. Program coordinators who had previously experienced difficulty getting data had it on their desks later that week. Over the course of the year, the quality and quantity of the data the EvAP team obtained from the coordinators increased dramatically. We were extremely pleased that the collaborative evaluation strategies that guided our work had served us well in an unanticipated fashion.

Hot Tip: The data needs of the programs we serve as evaluators can sometimes seem daunting. In this case, we learned that fixing “the problem” was less a data-related matter that it was a “marketing” issue. SIMS administrators, and other keepers-of-the-data, have multiple responsibilities and are under tremendous pressure to serve multiple constituencies. Sometimes, getting their support and cooperation are merely a matter of making sure they are aware of your particular program, the kinds of data you require, and the frequency of your needs. Oh, and to know that they are appreciated doesn’t hurt either.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.