My name is Charmagne Campbell-Patton and I am an independent evaluation consultant. About a year ago, I made the transition from my role as program manager and internal evaluator at an education nonprofit, to an external evaluation consultant. I continued working with my former employer as a client and, in my naiveté, I thought the transition would be relatively straightforward. I figured that since I knew the inner workings of the organization and had strong relationships with most staff members, it would be easy to continue to conduct useful evaluation.
Lessons Learned: My first mistake was failing to recognize and address that as the program manager, I used to be the primary intended user of the evaluation results. When I made the transition to an external consultant, I needed to be much more intentional about designing evaluations that met the needs of the new intended users.
Hot Tip: Be aware of how your position affects use. The personal factor is different in different relationships – internal and external.
Lesson Learned: Process use is different internally and externally. As a staff member, I used to be able to identify opportunities for process use in an ongoing and informal way. As an external consultant, however, I again had to be much more intentional about identifying opportunities and planning for process use.
Hot Tip: External evaluators need to be intentional about seeking opportunities to support evaluative thinking across the organization through more formalized process use.
Cool Trick: One way to engage staff is a reflective practice exercise. Bring staff together to reflect on the question: “What are things you know you should be doing but aren’t?” This question gets people thinking about potential personal barriers to using information. That sets the stage for discussing barriers to evaluation use organizationally. Next identify enabling factors that support and enhance use, and ways to overcome barriers to use.
It’s also worth noting that despite some of the challenges noted above, the transition from internal to external also gave me a new perspective on evaluation use. Once I recognized some of the barriers to use as an external consultant, I was actually able to use my position to promote use more effectively than I did while internal. The added distance gave me some leverage that I lacked as a staff member to call attention to opportunities and challenges to evaluation use across the organization.
Rad Resources: Essentials of Utilization-Focused Evaluation, Michael Quinn Patton, Sage (2012).
Consulting Start-Up and Management, Gail Barrington, Sage (2012).
Using Reflective Practice for Developmental Evaluation, Charmagne Campbell-Patton, AEA365 March 2015.
The American Evaluation Association is celebrating Evaluation Use (Eval Use) Topical Interest Group Week. The contributions all this week to aea365 come from our Eval Use TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello,
My name is Emily Carroll, and I am currently enrolled as a student in a Professional Masters of Education with Queen’s University.
I am taking a course entitled “Program Inquiry and Evaluation”, in which we are learning about evaluation methods and applying our understandings to the construction of a practical evaluation plan.
I was particularly interested in your post “Internal vs. External Evaluators”, as I am undergoing a similar, albeit less formal, transition from the role of “internal evaluator” to “external evaluator”.
At my place of work, we conduct evaluations of our pre-Ontario Secondary School Literacy Test (OSSLT) Preparation Program. This program is aimed at providing opportunities for student development in the area of literacy, before taking a mandatory standardized test.If students are not successful on this test, they are not able to receive an Ontario Secondary School Diploma. With the results of the test, the planning team reflect on the programming practices and how we may approach things similarly or differently in the upcoming year. Originally I was on an evaluation team as a facilitator, who helped to develop, plan and implement the program. As the Facilitator, I was also the primary intended user of the evaluation results. This year will be my first year being asked to assist in the evaluation of the program, external to being directly involved in the planning and implementation process.
In your post, you mention “being aware of how your position affects use”. I am wondering what challenges you may have faced with regards to the “personal factor” and its implications for use. How did you address them?
Also, I may not have realized how identifying opportunities for process use was less of a challenge as an internal evaluator and was wondering what strategies you suggest for being “more intentional about identifying opportunities and planning for process use”. You suggest engaging staff in reflective practice, which I see as very valuable, but did you orchestrate the questions they should be reflecting on? Did you have scheduled meetings for reflective practices? How did you stimulate productive, positive and informative reflection? What techniques did you find helpful?
Thank you for taking the time to read this e-mail. As I am new to the theories around program evaluation, I am very interested in how I can play a more effective and efficient role as an evaluator within this new position.
Emily Carroll
Hello Charmange,
My name is Monika Wheeless and I am a student at A&M University-Central Texas.
I enjoyed your Blog because you showed the positive and negative side of working as a consultant coming from internal to external. You would be of an advantage for the inner workings of that company but I see your dilemma of having to put your mind on the external factor but still remembering your experience of the internal factor. I liked the tip on asking the staff their opinions by engaging them in the thought process of what they should be doing.
Thank you for your advise and tips.
Hello Charmagne,
My name is Nick Hanukov and I am currently taking a Masters course on program evaluation and I have found your article on “Considerations for Evaluation Use” to be very interesting and relatable. You have highlighted some important distinctions between internal and external evaluators that I will consider when conduction my own program evaluations. The question of who the intended users are and how the evaluation process can be best organized to suit their specific needs will be one of my main considerations. I think these perspectives are very important to keep in mind when creating and conducting program evaluations if we are to maximize utilization. Utilization has been a central theme in our course and we are currently looking into questions related to how we can insure that our evaluations not only meet the needs of users but that they are used as effectively as possible. I agree with you that one of the important factors to keep in mind is how to establish exactly who the intended users are in order to maximize the effects of the program evaluation. I think you make a great point that as an external evaluator, you have to be much more proactive in ensuring your evaluation meets the needs of intended users, who are not yourself. I am interested in your opinions on specific strategies that address issues around collaborative evaluation that emphasizes utilization. Having been an evaluator both from a managerial and consultant perspective, how do you feel about collaborative evaluation? Should more emphasis be placed on collaborative evaluation from the perspective of an external vs. internal evaluator? Do you have any strategies on how program evaluators can ensure they are meeting the needs of users from both an internal and external perspective? I appreciate your ideas on this topic. Thank you.
Nick Hanukov
Hello Charmange! I find your journey from internal to external evaluator interesting. I’m curious to know if working with your former employer has caused any ethical dilemmas? For instance, because you understand the inner workings of the company, has that knowledge put pressure on you to provide an unbiased evaluations? Working closely with any group of people will overtime create a bond and having the responsibility of now being in an objective role cannot be an easy position. Good luck to you and your future endeavors!