Hi, I am Christina Freyman, a Director in the Center for Innovation Strategy and Policy at SRI International. Today I am writing about non-survey sources of data for evaluation – specifically related to the evaluation of research programs. Evaluations of programs inevitably suffer from biases related to missing or unreliable data, particularly when the goal is to measure a program’s impact on human capital. Assessing human capital often relies on survey-based research, which is prone to a number of biases. In addition, it is extremely costly to identify and survey individuals who did not participate in the program being evaluated (non-participants) to obtain a comparison group. You can overcome these challenges through the novel application of a text-analytic, non-survey-based approach.
Hot Tip: Apply text analytics to measure skills from participant and non-participant resumes.
Resumes are freely posted on resume sites and allow researchers to create robust comparison groups. Structured resumes can be parsed and ingested into a Mongo database with software code. Skills can be extracted from the resumes based on a skill list, and skills can be manually coded as related to the topic of the evaluation. For example, we have algorithmically tagged resumes with energy efficiency skills. Using the listed skills, you can identify jobs that were energy efficiency jobs on a large scale.
For the comparison groups, you can use an automated webcrawler to obtain resumes (with the permission of the website). For example, we have investigated three analysis groups in a recent evaluation: a random sample of participant resumes (N=109), a comparison cohort of non-participant resumes drawn from individuals attending the same school programs during the same time period (N=301), and a comparison group of energy efficiency job holders (N=867). Using algorithms, we were then able to quickly and efficiently determine the following:
- Participants had more energy efficiency skills than comparison groups.
- Participants entered an energy efficiency job faster than the energy comparison group.
- Participants spend a larger portion of their subsequent career in an energy efficiency job than energy comparison group.
Lessons Learned:
Evaluations of education and skill-building programs are frequently conducted using surveys, which have significant shortcomings when attempting to understand skills and career trajectories. Using a readily available data source that provides machine-readable resumes, we developed and employed a methodology that produced a more reliable and detailed understanding of the impact of a program on the skills and careers of energy-focused professionals.
The American Evaluation Association is celebrating Research, Technology and Development (RTD) TIG Week with our colleagues in the Research, Technology and Development Topical Interest Group. The contributions all this week to aea365 come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.