Improve your evaluations through informal peer review by Bronwyn Mauldin

I’m Bronwyn Mauldin, Director of Research and Evaluation at the Los Angeles County Arts Commission. I’m going to share the informal peer review process we use to improve the quality of our work.

Even if you’re not writing for an academic journal you want to make sure your methods are rigorous, your findings watertight, your final report lucid and clear. How can you get an objective assessment prior to print if your report doesn’t go through peer review? Ask an external colleague who works in the same field or uses similar methods to read it and give you feedback. In fact, ask two or three of them. Here at the LA County Arts Commission we’ve established a practice of doing this for every research or evaluation report we publish. It’s a simple idea we’ve found to be remarkably beneficial.

This practice is especially useful for those of us who work in that area some call “gray literature” published by nonprofits, foundations, government or other non-academic institutions. While we may have the advantage of working closely with practitioners and subject-matter experts, we have less access to the kind of meticulous critique available in the academy.

Rad Resource: Your colleagues. Identify three or four experts outside of your organization, then ask them to review your report and comment on it. Provide guiding questions so they’ll pay attention to your key issues, but be open to whatever else they find. Be sure to credit your reviewers in the final report.

Lesson Learned: People can be remarkably generous with their time and expertise. We’ve sent reports to reviewers that run to 70 pages or more, and others that were loaded with charts and graphs. Most people we’ve asked delivered thoughtful, thorough feedback.

Lesson Learned: Timing and communication are critical. Reach out to potential reviewers to get their commitment early in the writing phase. Send them the finished report when the text and charts are complete (but before the design phase). Give reviewers enough time for their review based on the length and complexity of the report, and a clear deadline. It might take a reminder or two, but most people eventually come through.

Cool Trick: Don’t limit yourself to colleagues you know. Contact the top experts in your field – both academics and others. This is also a great way to raise your profile with experts you’d like to get to know.

Independent evaluators who want to use informal peer review will probably need to let the institution you’re working for know what you’re planning in advance. Invite them to recommend experts to serve as reviewers.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.