Hello, we are Katherine Tibbetts and Wendy Kekahio, and both program evaluators doing work within the field of education in Hawai`i. Our work involves using indigenous ways of teaching and learning to inform culturally relevant and responsive ways of conducting research and evaluation studies.  One of our recent projects involved working with Hawaiian-focused charter schools to assess the impact of participation in professional development programs.

The Collaborative Inquiry (CI) project was designed to be culturally relevant and responsive–representing the values of relevance, rigor, respectful relationships, and reciprocity (for more information see, among others, Tibbetts, Faircloth, Villegas and Wheeler (2008)), The CI project extended the conventional purposes of evaluation to prove or improve, by employing a meta-action-research strategy to support the transfer of knowledge and skills learned at the training and assess their impact on teaching and learning. To do this, all participating teachers were required to conduct collaborative inquiry projects. They were encouraged to do their projects in small groups. The charter school teachers’ projects were supported by faculty contracted from a local college of education and culminated in a Ho`ike (demonstration of knowledge or skills).

Hot Tip: Supporting the Inquiry Projects. If you are interested in replicating this approach, it is important to provide ongoing support and scaffolding for the inquiry projects. The simplified action research curriculum and tools provided by the college of education faculty brought what were previously largely abstract concepts to life for the charter school teachers. Multiple “touch points” throughout school year, including visits to the charter schools enabled the college of education faculty to provide advice on the feasibility of project plans to identify potential sources of data that were tailored to each action research project, and helped sustain the momentum of the projects

Hot Tip: Assessing the Impact. As evaluators, our primary challenge was to synthesize information across a wide variety of projects. In the first year, there were 8 projects conducted in 3 different schools with topics spanning nutrition education, behavior management, mathematics, and writing. We approached the analysis as a multiple case study (based loosely on Stake, 2008) and ultimately created a rubric based on the CI project objectives and standards of inquiry. This allowed us to assess and summarize the quality of the inquiry projects.

Rad Resources:

Deloria Jr., V., & Wildcat, D.R. (2001). Power and place: Indian education in America. Golden, CO: Fulcrum Resources.

Hood, S., Hopson, R. K., & Frierson, H. T. (2005). The role of culture and cultural context: a mandate for inclusion, the discovery of truth and understanding in evaluative theory and practice. Greenwich, CT: IAP

Smith, L. T. (1999). Decolonizing methodologies: research and indigenous peoples. London: Zed Books.

Stake, R.E. (2006). Multiple case study analysis. New York: Guilford Press.

Thompson-Robinson, M., Hopson, R., & SenGupta, S. (Eds.). (2004). In Search of Cultural Competence in Evaluation (Vol. 102). Fairhaven, MA: Wiley Periodicals.

Tibbetts, K. A., Faircloth, S., Villegas, M., & Wheeler, L. (2008). Section III: Indigenizing accountability and assessment. In M. K. P. A. Benham (Ed.), Indigenous Educational Models for Contemporary Practice:  In Our Mothers Voice II. New York: Routledge.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

collaboration, collaborative inquiry, indigenous learning

Hi, my name is Susan Geier and I am a doctoral student at Purdue University studying research methods, measurement and evaluation. I employ a participatory evaluation approach with the GEMscholar project and have learned much from the Native American college students and the dedicated program staff.

Lessons Learned: I would like to share my three R’s for participatory evaluation:

1. Build Rapport: In addition to conducting formal interviews and assessments, I interacted informally with the students and mentors when time allowed, during meals and in between activities. I spent time learning about Native American history and culture from the project team and students.

2. Demonstrate Relevance: I discussed with the stakeholders and participants possible benefits of the evaluation process and their unique roles in the improvement and success of the program components. For example, when the students expressed interest in helping future GEMscholars, a peer-mentoring option was added to the program. Consequently, students began to see the evaluation process as a mechanism for sharing their experiences and suggestions instead of an outside critique of their lives and activities.

3. Maintain Responsiveness: I provided the stakeholders with information in a timely and accessible format. Often these were oral reports followed by brief documents outlining the changes discussed. We had conversations about those issues that could not be resolved in a timely matter and possible effects on the program. In turn, the project team made ongoing changes, adding components where needed and modifying those elements that were not serving the objectives of the program. Assessments were modified if needed and the process continued.

Hot Tip: Journaling is a useful technique to capture real time reactions to interventions. This is particularly important when working with groups who are being introduced to unfamiliar and/or uncomfortable experiences as part of an intervention. I worked closely with the researcher and program coordinator to develop pertinent guiding questions for the students’ and mentors’ daily reflection journals. This is also a good time to develop an analysis rubric if applicable. Journals can be hand written or online (I provide a link to an online journal using Qualtrics). The journal entries provide a project team with valuable insights about how the program elements are perceived by all involved.

If you want to learn more from Susan, check out the Poster Exhibition on the program for Evaluation 2010, November 10-13 in San Antonio.

College Access Programs, Native American, participatory

My name is Gerri Spilka and I am the Executive Director of the OMG Center for Collaborative Learning – a consulting firm that assists and assesses change through evaluation, strategic research and planning, grants management and capacity building. Among our many projects, OMG serves as the lead management team on the Robert Wood Johnson Foundation Evaluation Fellowship Program and it is about that program that I will share information today.

The skill set needed to interpret effectively public health, social service, community and organizational social patterns depends on the authentic and diverse experiences within program evaluation teams. Programs and services that are often evaluated traditionally work with underserved and disadvantaged communities, which are themselves becoming more diverse. However, the program evaluation field is becoming less diverse due to a lack of training and placement opportunities for emerging professionals from graduate programs who come from underserved and disadvantaged communities. This creates a problem for the field where, more often than not, high-quality evaluations reflect a process that incorporates diverse perspectives. The RWJF Evaluation Fellowship Program works to ameliorate this disparity.

Resource: The RWJF Evaluation Fellowship Program seeks to extend the skills of participants from diverse backgrounds to include culturally responsive evaluation practices. The Fellows program supports two cohorts annually on parallel paths:

• The Emerging Professionals Program is aimed at those who have graduated from a master’s or doctoral degree program within the last three years and provides a one-year placement, extensive training opportunities, and a support stipend
• The Retooling Professionals Program is focused on mid- or senior-level nonprofit professionals who remain in their current position, but receive additional training to extend their skills to evaluation. Their organization receives a $5,000 financial award and the professional receives a travel stipend.

Both cohorts receive customized training, facilitated by some of today’s thought leaders in field, as well as mentorship, and practical field-based guidance.

Hot Tip: While the deadline from participating in the Retooling Professionals program has passed, the deadline for applying for the Emerging Professionals program is coming up on July 31, 2010. The Emerging Professionals application may be downloaded online from http://www.rwjf-evaluationfellows.org/emerging-fellows.

Resource: The RWJF Evaluation Fellowship Program just released a video entitled Evaluation With a Diversity Lens. This extended discourse among a range of professionals explores the value of diversity in the evaluation profession and may be downloaded for free.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, diversity

My name is Rodney Hopson. I am a faculty member in the Department of Educational Foundations and Leadership in the School of Education at Duquesne University. For the last several years (with a host of great colleagues in AEA, with Torres Consulting Group and OMG Center for Collaborative Learning with the support of the National Science Foundation and the Robert Wood Johnson Foundation, I have been developing internship and fellowship programs to support graduate and post-graduate students of color and from traditionally underrepresented communities. Embedded in their practice is the understanding and development of Culturally Responsive Evaluation (CRE) approaches and practices in these same communities.

Rad Resource: Adapting the elements of evaluation practice into one that reflects a culturally responsive one, Karen Kirkhart, social work professor at Syracuse University, and I have presented at the AEA/CDC Summer Institute for the last few years. We use the Frierson, Hood, & Hughes chapter in the 2002 National Science Foundation User-Friendly Handbook for Project Evaluation to guide a deeper understanding of how to integrate CRE throughout the evaluation.

Hot Tip: Situate CRE within elements or a framework of evaluation. Whether you use the CDC Evaluation Steps, or another framework to describe steps in evaluation, the key is to embed CRE throughout. For instance, at each stage of evaluation, CRE should be present from the time we prepare for the evaluation to the dissemination and use of results. Below is the CRE framework (with appreciation for the support of Elizabeth Kahl, Syracuse University, who assisted us on graphic and technical design elements), adapted from Frierson, et.al, 2002:

Rad Resource: Learn more about fellowship and internship programs in culturally responsive evaluation through the RWJF Evaluation Fellowship website or AEA’s Graduate Education Diversity Internship website.

Rad Resource: Develop a better theoretical and practical explanation of CRE by reading about its origins, its aim, and its movement as reflected in chapters in any of the following books and special issues:

• The Role of Culture and Cultural Context in Evaluation (Hood, Hopson, and Frierson Eds., Information Age Publishing)
• The SAGE International Handbook of Educational Evaluation (Ryan and Cousins, Eds, SAGE.)
• Handbook of Social Research Ethics (Mertens and Ginsberg, Eds., SAGE)
• SAGE Handbook of Evaluation (Shaw, Greene, and Mark, Eds., SAGE.)
• The Handbook of Leadership Development Evaluation (Hannum, Martineau and Reinelt, Eds., Jossey-Bass)
• In Search of Cultural Competence in Evaluation (Thompson-Robinson, Hopson, Sen Gupta, Eds, Jossey-Bass)
• Responsive Evaluation (Greene and Abma, Eds., Jossey-Bass)

Want to hear more from Rodney Hopson? He is serving as this week’s Thought Leader on the AEA Thought Leader Discussion Series. Learn more at http://www.eval.org/thought_leaders.asp. This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, culture