Hi, my name is Nicole Vicinanza. I am a member of the Government Evaluation TIG and a Senior Research Associate with JBS International. In my work with JBS I conduct and review evaluations, and provide technical assistance in evaluation for federally funded programs. Many of these have federal regulations or guidance regarding evaluation with which they must comply.
Lessons Learned: In order to help determine whether an evaluation effort (either proposed or completed) meets requirements in place for that program, my colleagues and I review the guidance or regulations for the specific program and create a checklist. We then review planned or completed evaluations for that program against the checklist. This allows us to check our own work and to provide clear feedback to program staff, internal evaluators and/or government staff overseeing the programs. Some tips and resources we’ve found in the process of doing this include:
Hot Tip: Break your checklist items down into small, separate pieces. For example if a program is required to have external evaluation using a comparison group once every three years that addresses at least a full year of program data, your checklist would have one item for “program conducted evaluation” one for “evaluation was conducted by an external group”, one item for “evaluation included a comparison group”, and so forth.
Hot Tip: Consult with program oversight staff at the funding agency on what will meet the requirements. Some guidance is vague or left open to interpretation. For example “sufficient sample size” may be statistically determined in some cases (i.e. a sample will yield a margin of error of +/-5% at the 95% confidence level) and in other cases a minimum number is determined based on other factors.
Rad Resource: A good place to search for the rules applicable to your federal program. If your program’s rules are here, a keyword search can find them. http://www.gpo.gov/fdsys/ This searches the Federal Register; United States Code, Compilation of Presidential Documents, Congressional Bills, Congressional Documents, Congressional Hearings, and many other documents.
Definition – Federal Register: “Published by the Office of the Federal Register, National Archives and Records Administration (NARA), the Federal Register is the official daily publication for rules, proposed rules, and notices of Federal agencies and organizations, as well as executive orders and other presidential documents.” (from http://www.gpoaccess.gov/fr/ )
Definition – The Code of Federal Regulations: “The Code of Federal Regulations (CFR) is the codification of the general and permanent rules published in the Federal Register by the executive departments and agencies of the Federal Government. It is divided into 50 titles that represent broad areas subject to Federal regulation. Each volume of the CFR is updated once each calendar year and is issued on a quarterly basis.” (from http://www.gpoaccess.gov/cfr/ )
The American Evaluation Association is celebrating GOV TIG Week with our colleagues in the Government Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our GOV TIG members and you can learn more about their work via the Government TIG sessions at AEA’s annual conference. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.