Hello. I’m Marjorie McRae, a Principal with Research Into Action ,Inc., a firm that specializes in process evaluation and market research for energy efficiency and renewable energy programs.
Repeated process evaluations and market research studies are great tools for supporting continuous improvement of efficiency and renewable programs. Yet some program administrators think, “It’s too early to do an evaluation. We are still figuring things out.” Or, “Things are working fine.” Or, “Nothing much has changed since the first evaluation. Why do another?” Sometimes evaluators wonder how to promote an evaluation, or how to deliver something of value that is not a repeat of past findings.
Hot tips:
- View the efficiency program as having a life cycle akin to infancy, childhood, maturity, and possibly less effectual old age (signaling a need for retirement).
- View the market response as following a diffusion of innovation cycle, a framework pioneered by Everett Rogers.
Thus, both the program implementation and the market response have their own progression of stages that the evaluator needs to explore. In addition, the program and the market interact and push each other along their life cycles; evaluators should explore this interaction and response.
Here are questions the evaluator might explore at different junctures of a program.
Questions relevant pre-launch and in the first six months:
- How can metrics be defined for measuring goal attainment? Does data collection support the metrics?
- Does the program logic support goal attainment?
- Is the program being implemented as designed?
- Any implementation difficulties?
- Are key market actors responding as anticipated?
Questions relevant at one year:
- What are the goal measurements? Is progress on track to attain longer term goals?
- What are the process steps? How long do they take? How many people are involved?
- What are the “sticking points”, opportunities for improving efficiency?
- What are the experiences of key market actors? Is the word getting out about the program? How easy is it for them to take action?
- What are the opportunities for improving effectiveness?
Questions relevant in subsequent years: (All questions at previous junctions may be relevant, as indicated by previous findings or when no prior evaluation done.) Plus an in-depth look at:
- Adequacy of assumptions about market response
- Reasons for non participation among end users and trade allies
- Barriers to repeat participation and larger projects among end users and trade allies
- Need for new interventions (training, marketing, networking, qualifying actions, etc.)
The American Evaluation Association is celebrating Earthweek with our colleagues in the Environmental Program Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.