AEA365 | A Tip-a-Day by and for Evaluators

TAG | gatekeepers

I am David J. Bernstein, and I am a Senior Study Director with Westat, an employee-owned research and evaluation company in Rockville, Maryland. I was an inaugural member of AEA, and was the founder and first Chair of the Government Evaluation Topical Interest Group.

Westat was hired by the U.S. Department of Education’s Rehabilitation Services Administration (RSA) to conduct an evaluation of the Helen Keller National Center for Deaf-Blind Youths and Adults (HKNC). HKNC is a national rehabilitation program serving youth and adults who are deaf-blind founded by an Act of Congress in 1967, and operates under a grant from RSA, which is HKNC’s largest funding source.

The Westat evaluation was the first evaluation of HKNC in over 20 years, although HKNC submits performance measures and annual reports to RSA. RSA wanted to make sure that the evaluation included interviews with Deaf-Blind individuals who had taken vocational rehabilitation and independent living courses on the HKNC campus in Sands Point, New York. After meeting with HKNC management and teaching staff, it became clear that communication issues would be a challenge given the myriad of ways that Deaf-Blind individuals communicate. Westat and RSA agreed that in-person interviews with Deaf-Blind individuals would help keep the interviews simple, intuitive, and make sure that this critical stakeholder group was comfortable and willing to participate.

Hot Tips:

  • Make use of gatekeepers and experts-in-residence. Principle Three encourages simple and intuitive design of materials to address users’ level of experience and language skills. For the HKNC Evaluation, interview guides went through multiple reviews, including review by experts in Deaf-Blind communication not associated with HKNC. Ultimately, it was HKNC staff that provided a critical final review to simplify the instruments since HKNC was familiar with the wide variety of communication skills of their former students.
  • Plan ahead in regards to location and communication. Principle Seven calls for appropriate space to make anyone involved in data collection comfortable, including transportation accessibility and provision of interpreters, if needed. For the HKNC evaluation, interview participants were randomly selected who were within a reasonable distance of locations near HKNC regional offices. Westat worked with HKNC partners and HKNC regional representatives with whom interviewees were familiar. In the Los Angeles area, we brought the interviews to the interviewees, selecting locations that were as close as possible to where former HKNC students lived. Most importantly, Westat worked with HKNC to identify the Deaf-Blind individuals’ communication abilities and preferences, and had two interpreters on site for interviews. In one case we used a participant’s iPad with large print enabled to communicate interview questions.


The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · · · ·

My name is Robert McCowen and I am a doctoral fellow in Western Michigan University’s Interdisciplinary Ph.D. in Evaluation. I served as a session scribe at Evaluation 2010, and attended session number 651, Introduction to Evaluation and Public Policy. My evaluation interests focus on education, and a great deal of modern educational policy flows from the top down—so it only makes sense to find out as much as possible about how policy is made, and how evaluators can make sure their voices are heard.

Lessons Learned: George Grob, the presenter, has a long history of involvement with evaluation and government. Among his many past positions is a 15-year term as the Director of the Inspector General’s Office of Evaluation and Inspections. He had a number of wise statements for evaluators:

  • “Home runs” do happen in government, but that’s not how games are won. Rejoice if your work finds instrumental use in legislation or regulation, but don’t make it your only goal.
  • Get to know the gatekeepers in government, whether at the federal and state level. Work with them, listen to them, keep them informed, be willing to respect their schedules, and you’ll have a much easier time making sure your reports get to where they can do the most good.
  • Know the relevant body of work when you deal with policymakers. Assume they know everything important about the topics they deal with (because they might), and strive to do the same.
  • When writing reports, you have maybe two pages to catch the eye and make a case for your conclusions. Make sure your best evidence and most compelling findings are obvious to readers.
  • Be as professional as possible, including making sure your integrity and independence are unimpeachable—but be careful to keep lines of communication and cooperation open with major policymakers and other stakeholders.

Great Resource: Mr. Grob’s presentation is an excellent resource for any evaluator who is new to dealing with government, and can be found here at the AEA public eLibrary.

At AEA’s 2010 Annual Conference, session scribes took notes at over 30 sessions and we’ll be sharing their work throughout the winter on aea365. This week’s scribing posts were done by the students in Western Michigan University’s Interdisciplinary PhD program. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· ·


To top