AEA365 | A Tip-a-Day by and for Evaluators

TAG | content analysis

We are Shannon L. Griswold, Ph.D., a scientific research evaluator and member of AEA’s Research Technology and Development TIG, Alexandra Medina-Borja, Ph.D., Associate Professor of Industrial Engineering at University of Puerto Rico-Mayaguez, and Kostas Triantis, Ph.D., Professor of Systems Engineering at Virginia Tech. We are thinking about new ways to envision and evaluate impacts from discovery-based scientific research. Tracing dollars spent on funding research in universities to societal impacts is very difficult due to the long time lag between experimentation and commercialization, and the serendipitous nature of discovery.

Lesson Learned: Even though we can’t predict every outcome of scientific research, we can apply a general framework that allows us to envision the complex system of scientific discovery and identify areas of inquiry that could lead to major breakthroughs.

Hot Tip: Gather your research community and ask them to think backwards from societal needs (e.g., in transportation research this might be a solution for traffic congestion). This can be HARD for fundamental researchers; they are accustomed to letting curiosity drive their research questions. From societal needs, ask them to map several enabling technologies that could meet that need. Enabling technologies should be things that could solve that need but that don’t exist yet (e.g., teleportation). Finally, from enabling technologies, ask your research community to map out knowledge gaps. These are the things that we don’t know yet, which prevent us from developing enabling technologies (e.g., how do you convert all the mass in a human body into energy without blowing things up? How do you reassemble that energy at the destination into a human body?). It can be helpful to frame knowledge gaps as questions.

Hot Tip: Use societal needs, enabling technologies, and knowledge gaps to perform a content analysis of your research portfolio. How many of the topics are already funded? How many topics are not yet represented in the portfolio? This analysis should be performed in the context of a portfolio framework, which may help you envision the scope of your funding program’s discipline and relation to other funding streams.

Rad Resource: When mapping societal needs, enabling technologies, and knowledge gaps, it can be helpful to place them in a hierarchical framework to track their relationships. In this diagram, dotted lines show the direction in which the logic framework is generated, working backwards from societal needs. Solid arrows show the flow of scientific knowledge, from discoveries (knowledge gaps) to technologies that meet societal needs.

Logic tree generic color v2

Rad Resource: The flow of knowledge and information in the scientific process is rarely linear. It is probably more accurately represented as a “ripple effect”. We can predict some discoveries and technologies (darker polygons), but others are emergent, and knowledge flows in all directions.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello!  I am Crystal Hurd, a K-12 educator and an adjunct professor at East Tennessee State University and Northeast State Community College.  Last year, I embarked on an unorthodox dissertation topic: Transformational Leadership in the Life and Works of C.S. Lewis.  No one had ever applied leadership theory to Lewis.  In response, many replied, the guy who wrote Narnia is a leader?

My research aims were two-fold:

  1. to study Lewis as a leader
  2. to illustrate how he fostered transformational traits in his characters

Using transformational leadership theory as my theoretical framework, I completed 15 months of intense document review, interviews with Lewis scholars, and observational data obtained from Belfast and Oxford.  The theory states that leaders should raise morality and motivation and ultimately create leaders.  My goal was to study Lewis’ influence to transform others.

Hot Tips:

  • Use research questions to guide the analysis. Always have a conceptual or theoretical structure when commencing research.  Bernard Bass’ four transformational traits, guided the trajectory of my reading. Several questions determined if the work would be included :
    • Did the fictional or nonfiction work:
      • mention themes of leadership, authority, or power?
      • contain Lewis’s perspective on leadership?
      • contain expectations of leadership in his time?
  • Let the interviewee lead the response. Emailed asynchronous text-based interviews allowed multinational participation. Contributors answered at their leisure and produced personally satisfying responses that differed in length and depth. This allowed for fuller analysis of relevant themes.
  • Triangulate with observation data. By documenting surroundings, annotating conversations, and taking pictures and video I was able to capture the climate and context.  My observational data illustrated how Lewis’s legacy is still influencing culture today, nearly 50 years after his death.
  • Take copious and concise notes. Annotating the massive Lewis bibliography was no small feat.  I captured my notes in one notebook with titles in the margin and pertinent information following.  I used highlighters to categorize important quotes and concepts.  Today, I continue to add to the notebook.
  • Do not dismiss topics thought to be irrelevant.  Exhaustive searches reap the greatest rewards.  Great gems appeared to address issues unrelated to my research topic, however, a deeper examination revealed interesting parallels.
  • Print out transcripts and highlight theory-based themes.  Interviewees were given four questions via email concerning Lewis’s impact on their lives and how his leadership is still effective.  I conducted content analysis to find support that Lewis was an effective leader. I then analyzed Lewis’ leadership traits inspiration in others.
  • Be patient and confident. In the beginning, I endured many a strange look when describing my research to others. I refused to be discouraged when people suggested I do a “more conventional” topic.  Follow your instinct.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hello, we are Rashon Lane, Alberta Mirambeau from the Centers for Disease Control and Prevention, and Steve Sullivan from Cloudburst Consulting and we work together on an evaluation aimed at assessing the uptake, use and impact of national public health hypertension recommendations by the Institute of Medicine (IOM). If you’ve ever wondered how to assess if public health programs are shifting their priorities to address evidence-based recommendations you might consider a methodology we used entitled alignment scoring analysis. In short, an alignment scoring analysis is a type of content analysis wherein narrative descriptions of organizational activities are analyzed to determine whether they support specific goals or strategies.  We conducted a pre-post alignment scoring analysis of state health department work plans to objectively determine if their project portfolios align with nationally recommended priorities.

Lessons Learned:

  • Conduct pre-post content analysis. During our content analysis we coded state work plan activities as aligned, mis-aligned or neutral to the IOM recommendations.  As a result, we were able to share with program stakeholders that many state health departments were able to adjust their prevention priorities within 18 months to reflect national priorities.  If you are working on an evaluation to assess changes in priorities over time, you might consider conducting a similar pre-post content analysis to determine the degree to which public health programs align with priorities and how these priorities change over time.
  • Use stringent criteria. Use stringent criteria to consider activities as aligned, mis-aligned or neutral for more accurate coding.

Hot Tips:

  • Use a database. Use a database to facilitate the review of documents being analyzed and to speed reporting.  If you plan to use multiple reviewers, be sure to keep track of which reviewer coded a document so you can check inter-rater reliability and improve training on your coding protocol.
  • Use alignment scoring. Use alignment scoring analysis results to provide recommendations to program stakeholders on how they might shift priorities that are NOT aligned with national recommendations that have proven to be effective.

Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· · · ·

Archives

To top