My name is Michelle Jay and I am an Assistant Professor at the University of South Carolina. I am an independent evaluator and also an evaluation consultant with Evaluation, Assessment and Policy Connections (EvAP) in the School of Education at UNC-Chapel Hill. Currently I serve with Rita O’Sullivan as Directors of AEA’s Graduate Education Diversity Internship (GEDI) program.

Lessons Learned: A few years ago, EvAP served as the external evaluators for a federally-funded Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) state-wide grant housed at University of North Carolina (UNC) General Administration. Part of our work involved assisting project coordinators in 20 North Carolina counties to collect student-level data required for their Annual Performance Review reports as well as for program monitoring, assessment, and improvement. For various reasons, project coordinators experienced numerous difficulties in obtaining the necessary data from their Student Information Management Systems (SIMS) administrators at both the school and district levels. As collaborative evaluators, we viewed the SIMS administrators not only as “keepers of the keys” to the “data kingdom,” but also as potentially vested program stakeholders whose input and “buy-in” had not yet been sought.

Consequently, in an effort to “think outside the box,” the EvAP team seized an opportunity to help foster better relationships between our program coordinators and their SIMS administrators. We discovered that the administrators often attended an annual conference each year for school personnel. The EvAP team sought permission to attend the conference where we sponsored a boxed luncheon for the SIMS administrators. During the lunch, we provided them with an overview of the GEAR UP program and its goals, described our role as the evaluators, and explained in detail how they could contribute to the success of their districts’ program by providing the important data needed by their district’s program coordinator.

The effects of the luncheon were immediate. Program coordinators who had previously experienced difficulty getting data had it on their desks later that week. Over the course of the year, the quality and quantity of the data the EvAP team obtained from the coordinators increased dramatically. We were extremely pleased that the collaborative evaluation strategies that guided our work had served us well in an unanticipated fashion.

Hot Tip: The data needs of the programs we serve as evaluators can sometimes seem daunting. In this case, we learned that fixing “the problem” was less a data-related matter that it was a “marketing” issue. SIMS administrators, and other keepers-of-the-data, have multiple responsibilities and are under tremendous pressure to serve multiple constituencies. Sometimes, getting their support and cooperation are merely a matter of making sure they are aware of your particular program, the kinds of data you require, and the frequency of your needs. Oh, and to know that they are appreciated doesn’t hurt either.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Administration, Evaluation planning, project management

Hi, my name is Susan Geier and I am a doctoral student at Purdue University studying research methods, measurement and evaluation. I employ a participatory evaluation approach with the GEMscholar project and have learned much from the Native American college students and the dedicated program staff.

Lessons Learned: I would like to share my three R’s for participatory evaluation:

1. Build Rapport: In addition to conducting formal interviews and assessments, I interacted informally with the students and mentors when time allowed, during meals and in between activities. I spent time learning about Native American history and culture from the project team and students.

2. Demonstrate Relevance: I discussed with the stakeholders and participants possible benefits of the evaluation process and their unique roles in the improvement and success of the program components. For example, when the students expressed interest in helping future GEMscholars, a peer-mentoring option was added to the program. Consequently, students began to see the evaluation process as a mechanism for sharing their experiences and suggestions instead of an outside critique of their lives and activities.

3. Maintain Responsiveness: I provided the stakeholders with information in a timely and accessible format. Often these were oral reports followed by brief documents outlining the changes discussed. We had conversations about those issues that could not be resolved in a timely matter and possible effects on the program. In turn, the project team made ongoing changes, adding components where needed and modifying those elements that were not serving the objectives of the program. Assessments were modified if needed and the process continued.

Hot Tip: Journaling is a useful technique to capture real time reactions to interventions. This is particularly important when working with groups who are being introduced to unfamiliar and/or uncomfortable experiences as part of an intervention. I worked closely with the researcher and program coordinator to develop pertinent guiding questions for the students’ and mentors’ daily reflection journals. This is also a good time to develop an analysis rubric if applicable. Journals can be hand written or online (I provide a link to an online journal using Qualtrics). The journal entries provide a project team with valuable insights about how the program elements are perceived by all involved.

If you want to learn more from Susan, check out the Poster Exhibition on the program for Evaluation 2010, November 10-13 in San Antonio.

College Access Programs, Native American, participatory

Greetings, My name is Mehmet Dali Ozturk. I am the Assistant Vice President of Research, Evaluation and Development at Arizona State University Office of Education Partnerships (VPEP), an office that works with P-20, public and private sector partners to enhance the academic performance of students in high need communities.

Along with my colleagues Brian Garbarini and Kerry Lawton, I have been working to develop sound and reliable evaluations to assess educational partnerships and their ability to promote systemic change. One of my ongoing projects has been the evaluation of ASYouth, a program developed to provide a holistic support system to the University, schools, and parents so that disadvantaged children have the opportunity to participate in university-based summer enrichment activities.

Based on this experience, we offer the following advice to evaluators working on university-based outreach programs:

Hot Tip: Create a Multi-Disciplinary Evaluation Team

Although most University-led summer enrichment programs are directed towards similar goals, the activities often focus on a multitude of subjects ranging from drama, music and art to intensive math and science courses. Given this, evaluation teams that recruit individuals with expertise in a variety of academic subjects are well-equipped to develop evaluation designs and assessment tools appropriate to these programs.

Hot Tip: Ensure Linguistic and Cultural Relevance

Evaluations should be developed and conducted by evaluation teams that possess cultural competency to the target population. This allows for the development of culturally sensitive assessment materials that can be translated into the heritage language of the program participants at a fraction of the cost of hiring outside consultants. In addition, when survey methods are used, culturally-appropriate measures will result in higher initial response rates. The need for fewer follow-ups can greatly reduce the cost of successful evaluations.

Hot Tip: Embed Evaluation into Program Design

Due to limited resources, evaluation expertise, and/or capacity, many summer enrichment programs do not include rigorous evaluation components. In these cases, evaluation is merely an afterthought, making it very difficult to ensure valid data collection or implement a design with appropriate controls.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources, and to consider attending CAP-sponsored sessions this November at Evaluation 2010.

cultural competency, program design, summer outreach programs

My name is Mika Yoder Yamashita. I am the qualitative evaluation lead for the Center for Educational Policy and Practice at Academy for Educational Development. Our Center has been conducting process and outcome evaluations of the federally funded program, Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP).  This program aims at increasing college access among disadvantaged students.  As we are evaluating programs implemented in several sites, we are beginning to explore the possibility of conducting a multi-site evaluation. Today I will share my Center’s thoughts on how we can effectively approach conducting a multi-site evaluation that uses qualitative data to understand the process of program implementation. Then I will share how we use the literature to guide our data collection and analysis.

Our evaluation utilizes a similar approach to cluster evaluation (W.K. Kellogg Foundation, 1998). We draw upon Davidson’s (2000) approach to build hypotheses and theories of which strategies seem to work in different contexts.  The end goal of our cluster evaluation is to provide the client with a refined understanding of how programs are implemented at the different sites.

Cluster evaluation presents us with the following challenge: How to effectively collect and analyze qualitative data in a limited time to generate information on program implementation. To help us to guide qualitative data collection and analysis, we draw on a literature review.

Hot Tip: Start with literature review to create statements of what is known about how a program works and why it works. Bound a literature review according to the availability of time and evaluation questions. Document keywords, search engines, and decision regarding which articles are reviewed in order to create a search path for others. Create literature review protocols that consist of specific questions.  The reviewers write answers as they review each article. The evaluation team members review two to three summaries together to refine literature review questions and the degree of description to be included. We use qualitative data analysis software for easy management and retrieval of literature summaries. With this information, we draw diagrams to help us articulate what the literature reveals about how a program works and in what context. Using diagrams helps to share ideas across evaluation team members who are not involved in literature review.  Finally, create a statement of how and why the program works in what context and compare these statements with the data from the multiple sites.

Resources: Davidson, E.J., (2000). Ascertaining causality in theory-based evaluation. New Directions for Evaluation, 87, 17-26.*

W. K. Kellogg Foundation (1998). W.K. Kellogg Foundation Evaluation Handbook. Battle Creek, Michigan: Author. Retrieved from: http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

*AEA members have free online access to all back content from New Direction for Evaluation. Log on to the AEA website and navigate to the journals to access this or other archived articles.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Mika, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

literature review, time management

My name is Jack Mills; I’m a full-time independent evaluator with projects in K-12 and higher education. I took my first course in program evaluation in 1976. After a career in healthcare administration, I started work as a full-time evaluator in 2001. The field had expanded tremendously in those 25 years. As a time traveler, the biggest change I noticed was the bewildering plethora of writing on theory in evaluation. Surely this must be as daunting for students and newcomers to the field as it was for me.

Rad Resource: My rad resource is like the sign on the wall at an art museum exhibit—that little bit of explanation that puts the works of art into a context, taking away some of the initial confusion about what it all means. Stewart Donaldson and Mark Lipsey’s 2006 article explains that there are three essential types of theory in evaluation: 1) the theory of what makes for a good evaluation; 2) the program theory that ties together assumptions that program operators make about their clients, program interventions and the desired outcomes; and 3) social science theory that attempts to go beyond time and place in order to explain why people act or think in certain ways.

As an example, we used theory to evaluate a training program designed to prepare ethnically diverse undergraduates for advanced careers in science. Beyond coming up with a body count of how many students advanced to graduate school, we wanted to see if the program had engendered a climate that might have impacted their plans. In this case, the program theory is that students need a combination of mentoring, research experience, and support to be prepared to move to the next level. The social science view is that students also need to develop a sense of self-efficacy and the expectation that advanced training will lead to worthwhile outcomes, such as the opportunity to use one’s research to help others. If the social science theory has merit, a training program designed to maximize self-efficacy and outcome expectations would be more effective than one that only places students in labs and assigns them mentors. An astute program manager might look at the literature on the sources of self-efficacy and engineer the program to reinforce opportunities that engender it.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Jack, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

theory

My name is Kirsten Rewey and I am a Senior Research and Evaluation Associate at ACET, Inc. Two years ago ACET was selected to provide evaluation services for a federally-funded Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) implemented by the Minnesota Office of Higher Education (OHE). As part of the services, OHE asked ACET to create a research plan to determine the impact of GEAR UP on students’ academic preparedness using Minnesota’s statewide academic test (Minnesota’s No Child Left Behind assessment).

Hot Tip: ACET used the following approach and finds that it facilitates staff development and client selection of a research design:

1. Identify data available from the client: In order to develop a solid research plan, ACET needed to know what data OHE/GEAR UP were currently collecting. One of our first planning meetings was to discuss the types of data available in the GEAR UP database, the format of the data, and retrieving the data. During the meeting ACET obtained the list of variables maintained in the database, their descriptions, and the timeline for data entry and retrieval.
2. Identify data available from other sources. Once ACET knew the data that would be available from GEAR UP, other sources of data needed to be identified and catalogued. From previous work, ACET staff knew there was a substantial amount of school-level, public-record data available from the Minnesota Department of Education. For example, the Minnesota Department of Education publishes demographic information for the state’s public schools on their website including number of students enrolled, number eligible for free or reduced-price meals, and number who have limited English proficiency. For some of the variables, demographic data are available by grade-level and ACET can hone in on a specific grade of interest. Other data, such as individual demographic data and test score results, are only available pending approval from the district or the state. In order to obtain this data OHE and ACET wrote a formal application to the district to receive selected data from their databases.
3. Create an array of research options and present them to the client. After ACET identified available data, the staff created an array of research options for the client. ACET typically creates a variety of designs that vary in research/experimental control, types of conclusions which can and cannot be drawn, and cost. The options are presented to the client in a matrix with a brief description of the design, the advantages, and challenges for each. And because clients have an array of research options and are alerted to the advantages and challenges of each design, they can select the research design that best meets their needs and budget.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Kirsten, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

collaboration, Evaluation planning, GEAR UP

My name is Sandra Eames, and I am a faculty member at Austin Community College and an independent evaluation consultant.

For the last several years, I have been the lead evaluator on two projects from completely different disciplines.  One of the programs is an urban career and technical education program and the other is an underage drinking prevention initiative.  Both programs are grant funded, yet; they require very different evaluation strategies because of the reportable measures that the funding source requires.  Despite the obvious differences within these two programs’ such as deliverables and target population, they still have similar evaluation properties and needs. The evaluation design for both initiatives was based on a utilization-focused (UF) approach which has universal applicability because it promotes the theory that program evaluation should make an impact that empowers stakeholders to make data grounded choices (Patton, 1997).

Hot Tip: UF evaluators want their work to be useful for program improvement, and increase the chances of stakeholders utilizing their data-driven recommendations.  Following the UF approach could avoid the chance of your work going on a shelf or in a drawer somewhere.  Including stakeholders in the early decision making steps is crucial to this approach.

Hot Tip: Begin a partnership with your client early on that will lay the groundwork for a participatory relationship and it is this type of relationship that will ensure that the stakeholder utilizes the evaluation. What good has all your hard work done if your recommendations are not used for future decision-making? This style helps to get buy-in which is needed in the evaluation’s early stages.  Learn as much as you can about the subject and intervention that they are proposing and be flexible.  Joining early can often prevent wasted time and efforts especially if the client wants feedback on the intervention before they begin implementation.

Hot Tip: Quiz the client early as to what they do and do not want evaluated, help them to determine priorities especially if they are under a tight budget or short on time for implementation of strategies.  Part of your job as evaluator is to educate the client on the steps that are needed to plan a useful evaluation. Inform the client that you report all findings both good and bad upfront might prevent some confusion come final report time.  I have had a number of clients who thought that the final report should only include the positive findings and that the negative findings should go to the place were negative findings live.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Sandra, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

design, stakeholders, utilization-focused evaluation

My name is Kathryn Hill and I work with the state of Minnesota’s GEAR UP college access initiative as the Evaluation and Research Manager. GEAR UP is the acronym for the Gaining Early Awareness and Readiness for Undergraduate Programs federal grant program, designed to increase the number of low-income students who are prepared to enter and succeed in postsecondary education. GEAR UP provides six-year grants to states and partnerships to provide services at high-poverty middle and high schools.

I collaborated with external evaluators to develop a framework for a cost-benefit study. I learned a lot from this evaluation management experience, and I have compiled a few tips.

Hot Tip- Start with a thorough literature review: To develop a cost analysis framework, you need evidence to support proposed outcomes and to attribute economic value for these proposed outcomes. It is possible to rely on existing research to determine a projected “effect” for the program when building a framework, but you should have results from your own rigorous evaluation before proceeding with the cost study.

Hot Tip- Use evidence to articulate the program theory: A logic model is a common starting point for communicating program theory, and it is essential for cost analysis.Using program theory, an evaluator can move toward a clear identification of the different program components. These program components guide the economic calculations of “inputs”. Financial calculations of program expenses are viewed from program staff and participant perspectives. Both are important for determining the cost of a program. If it takes a lot of time for a staff member to develop/deliver a program component that is utilized by only a small number of students, the cost per participant will be high for that specific component. This brings us back to the “effect” issue, because you may want to know what proportion can be attributed to each of your program components.

Hot Tip- Find a way to document/describe what is actually happening in the program: The external evaluators developed an interactive format for interviewing all program staff. Staff members found the process interesting; some even thought it was fun!

Hot Tip- Think “program components” rather than “accounting categories” when recording expenses: For example, our program provides college visits, and the cost includes transportation expenses. However, that budget category usually has transportation expenses for EVERYTHING, including field trips, summer programs, etc. You will save yourself many headaches if you set up detailed sub-codes for expense records.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio, to learn more from Kathryn.

cost analysis

My name is Susan Kistler and I am AEA’s Executive Director. Although I normally contribute each Saturday’s aea365 post, I am very excited to hand off those duties this week to the Chair of the College Access Programs TIG. One of AEA’s newest TIGs, CAP is building a strong group and program for the conference and they are going to be sponsoring the coming week on aea365. All week long you’ll see great contributions from our CAP colleagues!

My name is Rita O’Sullivan. I teach at the University of North Carolina at Chapel Hill and am the Executive Director of EvAP (Evaluation, Assessment, & Policy Connections). Within AEA, I also serve as the Chair of the College Access Programs TIG.

Evaluating college access programs can be challenging: a) Program participation can differ greatly among students in the same program; b) Measuring the ultimate desired program outcome (i.e. advancement to college) can be difficult, as student data become harder to gather after they leave high school.

Lessons Learned – Tracking Program Participation: Often there are many different opportunities for students to participate in a college access program. For example, they may enroll in tutoring programs and/or take part in organized college visits. They may be part of an intensive Freshman Academy program that meets daily for a year or only attend a one hour career counseling session. Evaluators need to remember that outcomes are usually proportional to program participation, so they need to estimate participation levels. They also need to maintain a balance in terms of the evaluation resources that will be used to gather data about program participation. One college access program had its program coordinators spending 20% of their time (one day per week)collecting and entering program participation data. With all this effort, however, participant turnover made it impossible to draw any conclusions about the relationship of program participation to program outcomes. On the other hand, asking students annually about program participation can result in serious underestimates. A possible compromise might be to ask students quarterly to identify the program-related activities in which they have participated.

Lessons Learned – Tracking Students’ Entrance and Persistence into College: Most college access programs operate through local public school districts in middle schools and/or high schools. Gathering data from high school seniors just before graduation about their “intentions” for the up-coming Fall is common practice in many places. Unfortunately, this practice usually overestimates the desired outcomes. Yet a more accurate alternative can be much more difficult to pursue. The National Student Clearing House keeps such data but charges for its services. Beyond that there’s growing evidence that getting to college is just a first, albeit important, step in finishing college. It’s incumbent on the evaluator to understand the college-going patterns within a given state context, so that a reasonable estimate of college-going can be made. In some states the vast majority of students attend public colleges and universities, so that forging partnerships with these colleges can yield extremely useful estimates by which to measure program outcome accomplishments and even persistence rates. Where this isn’t the case, other strategies, such as the National Student Clearing House, need to be explored.

College Access Programs, persistence

My name is Sarah Hug and I am a Research Associate with the Alliance for Technology, Learning and Society (ATLAS) at the University of Colorado at Boulder. I am going to give a few tips regarding “pipeline evaluation” for programs aimed at increasing enrollment and diversity in science, technology, engineering, and math (STEM) fields.

Pipeline programs aim to change the career trajectories of young people in the long term. These goals for youth are often beyond the scope of the evaluation and program timelines- for example, an academic science program targeting middle school students will not be able to collect career data for at least five years, when students have graduated high school. What should  an evaluator study in the meantime?

1. Student aspirations and interest: Evaluators can focus on student interest in the STEM fields, and their changing or continuing aspirations for STEM careers.
2. Student knowledge of the fields: Knowledge about careers in technical areas is essential for advancement in STEM, particularly for underrepresented and under-resourced students. Evaluators can focus efforts on program participants’ change in career awareness at all academic stages. Some elements of career awareness evaluators might measure include:  knowledge of the depth and breadth of science careers, knowledge regarding what scientists do, and familiarity with the level of education needed to attain specific careers.
3. Students’ “next- step behaviors”: Students’ college and career readiness can be influenced by early academic experiences. Discover what students might do at each level of the academic pipeline to further their STEM career readiness. This is often context and even school district specific-for example, are there clubs or camps students might join to advance their careers? What high school course choices could indicate students’ preparation for STEM careers? It is particularly important to consult program directors and school partners to gather ideas for measuring “next-step behaviors”. Comparing your program findings to local or national data for similar groups is essential for showing program impact.

Rad Resource: The National Center for Educational Statistics http://nces.ed.gov/ provides national data on student enrollment, graduation, and academic behavior at all levels of the education pipeline. Check out “The Condition of Education 2010” report, and data exploration tools to understand how students are progressing at different stages of the pipeline.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

careers, Geographic information systems, pipeline