Keith Child on Evaluation Research Quality

I am Keith Child, a Senior Research Advisor to the Committee on Sustainability Assessment.

The debate around appropriate criteria for measuring research quality has taken a new turn as development donors apply collective pressure on development agencies to prove that they can bringing about positive change for intended beneficiaries.  It is within this research for development (R4D) context that traditional deliberative (e.g., peer review) and analytic (e.g., bibliometric) approaches to evaluating research merit are themselves not measuring up.  In part this is because the design and evaluation of research has been the exclusive preserve of scientists who tend to judge research quality according to science values like internal and external validity, research design and implementation, replicability and so on, rather than on research use, uptake and impact.  Within the scientific community these latter criteria are seen largely as “somebody else’s problem”.  The message from the donor community, on the other hand, is adamant: science and scientific values “can no longer be considered a largely academic enterprise divorced from societal concerns about social goals”.

Rad Resource: To reconcile these sometimes-conflicting perspectives, the Canadian International Development Research Centre (IDRC) has recently developed an alternative approach, called Research Quality Plus (RQ+).  While the RQ+ framework consists of three core components, worth noting here are the  four dimensions and subdimensions for assessing research quality:

1. Research Integrity

2. Research Legitimacy

2.1 Addressing potentially negative consequences

2.2 Gender-responsiveness

2.3 Inclusiveness

2.4 Engagement with local knowledge

3. Research Importance

3.1 Originality

3.2 Relevance

4. Positioning for use

4.1 Knowledge accessibility and sharing

4.2 Timeliness and actionability

Dimensions 1 and 3 are typically examined as part of a research quality framework.  Dimension 2, with its emphasis on gender, inclusiveness and local knowledge is less the preserve of scientists, but certainly a core idea in R4D settings.  It is the fourth dimension, however, that makes the RQ+ approach so novel for evaluation research quality.

The “positioning for use” criteria attempts to measure the extent to which research has been positioned to increase the probability of its use.  Significantly, research influence (e.g., bibliometric or scientometric analysis, reputational studies, etc.) and actual development impact are not part of the assessment criteria.  Instead, dimension 4.1 focuses on the extent to which research products are targeted to specific users, conveyed in a manner that is intelligible to intended beneficiaries and is appropriate for the socio-economic conditions of their context.  Dimension 4.2 focus on the intended user setting at a particular time and the extent to which researchers have internalized this in their planning.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.