We are Rodney Hopson, former (2012) AEA President and Professor of Evaluation (Educational Psychology: QUERIES) at the University of Illinois-Urbana Champaign, and Aneta Cram, Ph.D. student at Victoria University of Wellington in New Zealand and AEA Program Chair for the Indigenous Peoples in Evaluation Topical Interest Group.
In a recently published article for Evaluation Matters – He Take T? Te Aromatawai, the journal associated with the Aotearoa New Zealand Evaluation Association (ANZEA), we reflect on the challenges and lessons learned in developing program theory, the role of culturally responsive evaluation (CRE), and the program evaluation standards (PES) that are employed or need to be employed during the development of program theory. See our article here, collaborated with Marvin Powell, Asia Williams, and Akashi Kaul, all from George Mason University.
Lessons Learned:
- While we evaluators recognize that program theory is integral to the evaluation process, navigating the development of program theory is much more difficult, especially for evaluators who concern themselves with balancing stakeholder engagement, dialogue, and reflection through a culturally responsive lens. We reflect on PES (U2: Attention to Stakeholders, F3: Contextual Viability, P1: Responsive and Inclusive Orientation, P2: Formal Agreements, A4: Explicit Program and Context Descriptions) as important considerations we realized in working with stakeholder groups to facilitate our evaluation process and lessons learned.
- The negotiation of role and expectation by the evaluation team was a challenge that raised curiosity and frustration among stakeholders. Even with a Memorandum of Understanding, evaluator roles and expectations are and can be questioned throughout an evaluation. We reflect on PES (U3: Negotiated Purposes, U4: Explicit Values, F2: Practical Procedures, P6: Conflicts of Interests, A6: Sound Designs and Analyses) to assist us in recognizing and documenting larger power dynamics that undergirded our evaluation exercise, the strategic priorities of the organization, and the subtle issues among staff members at the table.
Hot Tips:
- Engage early and engage regularly with clients! Building trust and being seen is essential to good evaluation practice.
- Even though it is fairly intuitive, do not underestimate the value of relationships and building a strong foundation with clients and other stakeholder groups early on in the evaluation process.
- We, as evaluators, are constantly learning. Regardless of the stage in the evaluation process, regular reflection is key to ensure that we are being responsive to the stakeholders and culture that underlie the evaluand/program.
Rad Resources:
- The third edition of the Program Evaluation Standards is a core resource for evaluators. A checklist developed by Western Michigan University Evaluation Center may prove useful for those looking for a thinner and practical resource.
- The Center for Culturally Responsive Evaluation and Assessment, an international community of scholars/practitioners, serves as a portal for organizations and individuals seeking to better understand and apply cultural responsiveness. Send an abstract to the CREA VI by February 7, 2020 for its conference in Chicago, IL this fall.
This week, we’re diving the Program Evaluation Standards. Articles will (re)introduce you to the Standards and the Joint Committee on Standards for Educational Evaluation (JCSEE), the organization responsible for developing, reviewing, and approving evaluation standards in North America. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Dear Rodney and Aneta:
My name is Shona Robbins, and I am currently a Professional Masters of Education student from Ontario Canada. I have recently come across your blog, and I wanted to identify some of the items that you touched on within your piece. However, before I do so, I should mention that I am currently enrolled in a course that focuses on Program Inquiry and Evaluation. Your blog caught my eye due to the role of culturally responsiveness evaluation that you have identified. Living in Canada, we are extremely multicultural, and being new to the concept of program evaluation, you definitely gave me something else to consider during the evaluation process.
Furthermore, as stated in your “Lessons Learned” section, communicating with stakeholders is a crucial part of the evaluation; however, also considering the cultural similarities, differences, and expectations of those stakeholders must also be considered in the process.
Moving on to your “Hot Tips” section, you discuss communicating early and regularly with clients. One of the courses that I currently teach is specific to business writing and correspondence. One of the main points that I encourage my students to always consider is the idea of “Goodwill” and positive communication. I want my students to realize that the main purpose of communication may be to inform, educate, or even persuade their audience; however, the idea of building a positive relationship right from the start is as crucially important as delivering important information to their designated audience. Additionally, we also spend a great deal of time consider who our audience is and what their needs, wants and motivators are, so I am glad to see that my ideas have mirrored your identified concepts as well.
Lastly, I wanted to thank you for including some additional “Rad Resources” that will further my understanding of not only program evaluation but also culturally responsive evaluation.
Thank you.