AEA365 | A Tip-a-Day by and for Evaluators

CAT | Translational Research

We are internal evaluators Cath Kane, of Weill Cornell Clinical & Translational Science Center, and Jan Hogle, at the University of Wisconsin-Madison Institute for Clinical & Translational Research. The 62 Clinical and Translational Science Award (CTSA) hubs — along with community-based and industry partners — collaborate to advance translational research from scientific discoveries to improved patient care by providing academic homes, funding, services, resources, tools, and training.

Lesson Learned: How do CTSA hubs define “translational?” Translation is the process of turning observations from the laboratory, clinic, and/or community into interventions that improve individual and public health. Translational science focuses on understanding the principles underlying each step of that process.

Hot Tip: Ask “What are we evaluating?” Internal evaluators determine whether programs are efficiently managed, effective in meeting objectives, and ultimately impacting the process and quality of biomedical research — involving multiple variables in complex systems. Case studies use qualitative data to complement quantitative data on research and training productivity. Analysis of multiple case studies identifies factors that facilitate or impede successful translation.

Retrospective or prospective? Retrospective analyses use an abundance of data to deeply study known successes. Prospective case studies can identify factors that influence translation in real time. Questions might include:

  • What does ‘successful translation’ mean?
  • Which operational process markers are most important?
  • What are the ideal duration metrics?
  • How does collaboration and the individual CTSA hub move translation along over time?
  • How can we better support the translational process?

The retrospective case study below tells a story of key operational markers in the development of Gleevec, used to treat leukemia. Public data such as publications, FDA approval and patenting events, and mortality data were overlaid with interview data from key informants.

Kane & HogleLessons learned:

  1. An individual face-to-face interview is indispensable for obtaining an insider’s view of the research process.
  2. Record the interview and take notes. Often the research details are unfamiliar to the interviewer. Hand-written notes alone may not capture the interview accurately.
  3. Ask investigators for their interpretation of “translation.” People have different ideas about the meaning of translation. What are the key moments or markers in translation?
  4. Case studies can serve many needs from short summary “stories” for public relations and newsletters, to more rigorous approaches paired with quantitative data for decision making by internal and external stakeholders.

Perhaps the most important lesson learned is the value CTSA evaluation teams bring to their hubs, not only for long term objectives, but also in the shorter term through contributions on a daily basis to programming adjustments and course corrections using a mixed methods approach to understanding complex change.

The American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

We, Arthur Blank and Julie Rainwater, are pleased to introduce a new Translational Research Evaluation Special Interest Group (SIG).  The SIG is part of the Association for Clinical and Translational Science (ACTS) and provides a forum for all aspects of evaluation related to clinical and translational science. The ACTS SIG recognizes organizations involved in translational science as diverse and their “evaluators” may not necessarily identify themselves as professionals in the field of evaluation.  Thus, our membership includes administrators, faculty, clinicians, librarians, biomedical scientists, and other stakeholders in translational research and workforce development.  The SIG, like its AEA partner TIG, offers its members the opportunity to share mutual interests, evaluation expertise, resources, and materials. Our work is closely coordinated with the AEA TIG and the diverse membership across both groups provides access to a valuable practice community that can share experiences and challenges.

We are off to a great start.  The ACTS Translational Science 2015 meeting in Washington DC in April hosted a first-ever “evaluation” track sponsored by the SIG.  Judging by the high attendance and lively discussion at the two evaluation-relevant panels, this track is likely to be a feature of subsequent annual meetings.

Lesson Learned: We walked away from this meeting with a few lessons to guide us moving forward.  The first panel, “Classifying Publications along the Translational Science Spectrum:  A Machine Learning Approach,” provided an opportunity for us to learn state-of-the-art approaches for how Clinical and Translational Science Award (CTSA) organizations analyze publications to understand the progress of discoveries through the stages of translation to implementation.  The second panel, “The Role of Evaluation in Translational Science Organizations,” was a unique opportunity for us to hear what current leaders of the CTSA Domain Task Forces and NIH National Center for Advancing Translational Sciences (NCATS) representatives think about the evolving role of evaluation.  The discussion about the future of CTSA evaluation was beneficial to all of us, including NCATS, as we consider how evaluation can help move the translational research enterprise in the right direction.

Over the next few months we will transition to new SIG leadership and start planning for the ACTS Translational Science 2016 conference. We are looking forward to building on the 2015 meeting, as well as the opportunity to gather at the AEA conference in November 2015.  (See you in Chicago!)

Rad Resource: For those interested in joining the Association for Clinical and Translational Science (ACTS), as well as learning about the various activities those engaged in translational science are involved with visit the ACTS web site.

The American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Bill Trochim and Arthur Blank here and we are delighted to introduce AEA’s newest Topical Interest Group (TIG) – the Translational Research Evaluation TIG – and this week’s theme.

So, let’s start with – what is “translational research” (TR) and why is it so important? There are lots of definitions of TR. We prefer a broad and encompassing definition along these lines “the systematic effort to move research from initial discovery to practice and ultimately to impacts on our lives.” In biomedical research, some refer to TR as “bench-to-bedside” suggesting that it links basic laboratory work to the practice of clinical medicine. Others (like us) tend to describe TR more sweepingly as “innovation-to-impact”, emphasizing the entire research-practice continuum from initial new ideas to their ultimate application and effect on society. In one sense, TR is very new, and one of the hottest topics in contemporary research. But, in another sense, it is as old as the research-practice distinction itself.

If research and practice were well integrated and functioning efficiently together, the emphasis on TR would be unnecessary. In most biomedical and applied social policy areas, research takes too long to influence practice; one well-known estimate is that it takes on average 17 years for a biomedical discovery to influence practice (and that is likely an underestimate). Some of this time is undoubtedly due to the inherent difficulties of translation. But there is considerable evidence to suggest that much of this time lapse may be due to other factors.

Lesson Learned: In many fields, the problems being studied are complex and require multidisciplinary and transdisciplinary approaches. But researchers have not been trained in collaborative and team science methods that might enhance such work. In many fields, researchers develop innovations without considering the world of practice, only to find out later that their ideas won’t work in the real world. Learning how to involve the practice community as integral participants throughout the research development process could help avoid such costly errors. In many instances, research and practice realms are poorly managed and full of inefficiencies. For instance, we know that the process of reviewing and starting a single biomedical clinical trial can take years and involve hundreds of steps (many of them unnecessary or duplicative). If we learned more about how to manage the research enterprise better – something like a “science of science” or a “science of science management” – we might see significant progress.

Rad Resources: This week you’ll be introduced to some of the members of our new TIG and to the kinds of issues we are addressing.

Rad Resources: The National Institute of Health’s Clinical and Translational Science Award program provides support to professionals engaging in this work.  NIH also offers additional information about translational science in biomedicine.

The American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

<< Latest posts

Archives

To top