Hello! My name is Melissa Day, Ph.D., and I’m a scientific analyst with Clarivate’s Web of Science consulting team. I wrangle mostly publication-based data for government, corporate, and academic clients.
For many government research grants, scientific publications are one indicator of success and impact. Funders often want to identify and promote grantee results, especially highly-cited publications. The pandemic has affected every part of research generation, however, with uneven implications for who and what was published. See below for time gaps to consider for scientific grant evaluations in 2020 and beyond.
Hot Tips
- Award to manuscript: it commonly takes three years to produce a publication after the start of an award, in my evaluation experience (and according to Boyack and Jordan, 2011). Agencies and evaluators should consider how the pandemic affected their projects, including whether their grantees were able to work from home without lab or resource access, if work from home expenses were allowable, whether there were gaps in data collection (e.g. fieldwork cancelled, data collection interrupted), and whether graduate students or postdoctoral fellows experienced funding interruptions or left the project. Note that the research output of women and minority researchers has been disproportionately affected by the pandemic (e.g. Vincent-Lamarre et al.).
- Manuscript to publication: the peer-review process accounts for some of the time lag between award and publication. Publishers and review teams have worked hard to reduce acceptance times for COVID-relevant papers to as little as 7 days during 2020, but often this process can take 100 days or more for non-COVID papers (Palayew et al., 2020). Publishers have experienced staffing and reviewer shortages, and delays to convert proofing jobs to work-from-home, so evaluators should anticipate that papers may take longer to publish.
- Publication to citation: Citation-based metrics are often used to represent the scientific impact of a publication. Citing papers are subject to the same time lags, however, so it may take years for a publication to garner many citations. The use of normalized citation metrics – which compare the citations a paper has received to the average citations that papers from the same subject area and year received – can help provide context.
- Post-award publication: grantees often provide a comprehensive list of publications with their final report, but they are not required to update that list after grant close. Given that a significant fraction of research output may occur post-award, agencies and evaluators should actively monitor for new publications (e.g. setting up alerts for award numbers appearing in funding text) so they have a more complete picture of their funded impact.
The American Evaluation Association is hosting Gov’t Eval TIG Week with our colleagues in the Government Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Gov’t Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Melissa,
It is terrible that so much work was put on hold because of COVID-19. I had not thought about the evaluations or research that had to be modified or completely disregarded because of the pandemic.