Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hi, my name is Sonia Chen. I’m a Principal Research Advisor at the Ministry of Health in New Zealand. In the last five years, I have transitioned from being a researcher and evaluator to overseeing research projects as a research commissioner and contract manager for the New Zealand government.
A crucial aspect of my role involves reviewing draft research reports and associated materials. During the review process, I have noticed some common issues.
Sometimes assumptions were presented as facts, sometimes survey questions lack the needed depth to provide the specific knowledge needed, sometimes as interesting as the findings are there is a misalignment between research aims and findings, and sometimes reporting structure and presentation of results (especially tables and charts) needed to be rearranged to effectively deliver important insight.
I would like to share my experience in a series of articles as a way of giving back to the sector. For the purpose here, ‘research’ includes evaluation.
Part 1 – The value of early draft reviews
As a funder, I consider early draft reviews as an opportunity to:
1. Bridge the gap between authors and stakeholders
Although the research contract would have included project aims and research questions, researcher can still have a different sense of interest and importance during analysis and reporting. For example, researcher might provide descriptive analysis for each factor examined, funders might be more interested in a selected few that they know the least about and want to know more about them.
Researcher and funder often have different writing styles and terminologies. What considers clear to one may not be so to the other. Here is an example from a funder relating to this: “There is a theme of quite complex language along with very wordy text blocks. This reduces the readability of the report”.
2. Align research aims with findings
At times I see reports including all the data analysis that’s been done, or the summary section talks about what the study has discovered but did not link the findings to the research questions. Funders asked those research questions with a reason and expect a clear answer to them for decision making. This could involve synthesis, triangulation, and cross referencing of individual findings, for example.
3. Allow time for meaningful discussions and necessary adjustments
During early draft review, these differences and expectations can be highlighted and discussed, so the final report can be presented in a way that speaks to the funder’s audience while captures important information.
In my experience, projects that go through early draft review process delivers faster and more fit-for-purpose final reports. To demonstrate the value-add of early draft review, here is an example from a longitudinal report (fake data used). On the left is the original chart from the early draft report. On the right is the modified chart after the review. Both contain the same data but it is much easier to see the information from the chart on the right. “I prefer the modified chart. Because there are many drink options, having a distinct colour for each is easier to follow. Presenting the sorted preferences at age 22 made the chart feel more anchored” (one funder’s comment).
Lessons Learned
Researchers’ responses to feedback vary. Some welcome early input to improve while others may perceive it as a challenge to their work. As a reviewer, I have learned to adapt my communication style to different researchers, including sharing the above information to aid understanding, without compromising on funder requirements.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.