I’m Maurya West Meiers. I work at the World Bank as a Senior Evaluation Officer and am coauthor of A Guide to Assessing Needs: Essential Tools for Collecting Information, Making Decisions, and Achieving Development Results (free World Bank book).
As the global community experiences COVID-19, there is much value in looking backwards as we plan ahead for our future activities such as programs, needs assessments, and evaluations. So today I’m writing with some thoughts and resources on conducting after action reviews (AARs) during and after a crisis, such as COVID-19.
I’m reminded of this great quote from Rahm Emanuel, former mayor of Chicago, former Democratic congressman from Illinois, and former chief of staff to President Barack Obama from 2009 to 2010.
If you are planning future needs, don’t let a crisis, such as COVID, go to waste. You and your organization are making many important decisions during this time. Collect real-time information during the crisis to inform those things that you can do differently or better than before. Do it now, or as soon as the crisis begins to subside. Transition from “playing defense” to proactively learning and apply strategic insights to reflect on what went right, what went wrong, why, what we can do better, and so on. AARs focus on problem-solving and decision-making, hopefully to lead to organizational learning and actions that will have better performance and results in the future.
AARs were developed and popularized by the US Army in the 1970s, building on early approaches from the US military. Over the years, all sorts of organizations have adopted and adapted AARs for their own uses, with Shell Oil being one well known example. AARs usually respond to a general question (“How did we do?”) by answering these (and other) more specific questions:
- What were the intended results?
- What were the actual results?
- What caused the results?
- What will we sustain or improve?
In carrying out your AARs, there are a range of quantitative and qualitative approaches for you to use to support the review – such as interviews, surveys, focus groups, administrative data reviews, and among many others. Below I present some resources to guide you in developing AARs and to learn how other agencies use AARs. Be sure to consider implementing AARs so you can learn during and from a crisis.
Rad Resources:
- Foundations of the After Action Review Process, Institute for Defense Analyses (1999)
- Learning in the Thick of It, Harvard Business Review (2005)
- BetterEvaluation’s page on AARs
- Adaptation of the US Army’s After-Action Review for Simulation Debriefing in Healthcare, Simulation in Healthcare, The Journal of the Society for Simulation in Healthcare (2013)
The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Maurya West Meiers,
I wanted to begin by saying that I thoroughly enjoyed reading through your article and found it to be very impactful. Your article is highly relevant and tailored to our global community’s current circumstances. As you addressed in your article, the COVID-19 pandemic has introduced a degree of uncertainty, impacting many professional fields, including the field of evaluation. However, you highlight how the pandemic has introduced a time of opportunity. The idea that where there is chaos, there is opportunity, resonated with me and introduces a new perspective to the learning opportunities various industries can have as a result of these unique times.
Your insights regarding organizations collecting data during a crisis, specifically the COVID-19 pandemic, to be used for an after action review, were insightful and clearly outlined the value of reflective practices during these unprecedented times. As society’s current state leaves many questions regarding what is to come, implementing collecting data and implementing after action reviews enables us to prepare for an improved future in a post-pandemic world.
As an educator, I read your article with the education field in mind. I began to consider the reflective opportunities the pandemic could give our current education practices. With the future instructional practices still being an uncertain topic, I feel that after action reviews could be implemented to assess the effectiveness of the current remote learning practices.
Instructional and assessment practices, from primary through post-secondary, have been impacted by the current COVID-19 pandemic. The transition to remote learning was sudden and unexpected. We know little of how to make the impacts, procedures and policies of remote learning effective in supporting student learning, as it is a foreign practice for many.
Initially, remote learning was seen as a temporary solution to ensure learning continued during these times. However, discussions surrounding remote learning have begun to shift, addressing how the current approach to education is becoming a more permanent fixture in the field. As we see more and more programs extending their implementation of remote learning, it makes you wonder if remote learning will be a long-term component of our education system. Therefore, it is vital that we work to obtain data throughout this time, data we can use to evaluate and reflect on our current remote learning practices in after action reviews.
As stated in your article, reviewing data we collect throughout the pandemic will help us answer the question How did we do? The data we collect during this crucial time will allow us to assess components of remote learning that worked and ones that did not. The conclusions, determined by an after action review, regarding our current remote learning practices, could enable us to evaluate our programs, enhancing future remote learning. Reflecting on our current data could help us solve challenges such as attendance and equal accessibility to technological devices. This information is valuable as remote learning is beginning to be seen as an ongoing educational practice that will remain a staple in the education field until we have a means of treating or preventing COVID-19.
I thoroughly enjoyed reading through your article. It has made me reflect on the opportunities available to the education field in these unprecedented times. It is motivating to think about the options the field of education has to maximize this period of time and how we can enhance the newly introduced instructional practices for future use.
Thank you,
Sarah Losch
Sarah,
Your ideas were spot-on about applying AARs and also, more generally, being intentional about continuous learning while carrying out activities such as remote learning. I agree that there is so much that can be learned about remote learning (and other topics) during this time. If you are an educator doing remote learning, capture your observations – write them down or videotape them, even if only for your own reflections later, if not possibly to share your ideas with researchers on the topic. It seems we’re all part of a number of natural experiments these days. I saw an interesting article in Science about this topic: https://www.sciencemag.org/news/2020/04/social-scientists-scramble-study-pandemic-real-time. Thanks for your insights – Maurya
Hi Maurya,
First off, thank you for your article, a reminder of something positive that we can take from a negative situation. My name is Brittney, I am currently enrolled in a course called Program Inquiry and Evaluation, for my Masters degree. Evaluation is a very new subject for me and although I think I have barely scratched the surface, I have enjoyed learning the intricacies of the subject.
In a way, I sensed an underlining of hope in your article. I appreciate your call to not let COVID go to waste. It is an experience we can learn from and a push for evaluators to think outside the box and take advantage of the opportunity at hand. As you mentioned, the After- Action Review (AAR) is a relevant tool, requiring a focus on problem-solving and decision-making, two things many business and organizations have had to do a lot of the last few months. It is a time to be vigilant and gather data that will benefit both stakeholders and programs. In months and years to come, with the help of data collection and AARs, we will see the full extent of the impacts of the pandemic.
Your Rad Resources supported your points and allowed myself, the reader, to dive in a little deeper. I took a brief look at the Harvard Business Review, entitled, Learning in the Thick of It. Whether in the midst of the COVID-19 pandemic or part of the U.S. Army’s Opposing Force, there are moments where the daily norms do not follow the same path they used to. An AAR is valuable in both circumstances, being able to reflect back on the problems, and with data, support decision making.
A message I can take away from your article is that the pandemic may have created a “new normal” for the time being, but evaluation doesn’t stop, in fact, it aids in the process of learning to understand the full effect of what we did go through, what we are going through, and how actions can be better in the future.
– Brittney
Brittney,
Thanks very much for your excellent comments and for taking the time to provide feedback. I fully agree with your comment “evaluation doesn’t stop.” When I’m teaching a course to new evaluators, I always ask, “when does evaluation start?” It starts at the beginning (of a policy, program, project, etc.)….and runs through the life of the activity, ideally. People so often think about an evaluation as being an end-of-activity event (and it can be that in some cases). But we should be thinking about evaluation efforts (of varying types) happening through the life of a project, program or policy….and also the time after a project, program or policy ends — to see what difference it made. An AAR might be a tool used – in the right circumstances and applied the right ways – that an evaluator can keep in his/her toolbox. Good luck with your studies, Brittney. – Maurya