Hello, we are Katherine Tibbetts and Wendy Kekahio, and both program evaluators doing work within the field of education in Hawai`i. Our work involves using indigenous ways of teaching and learning to inform culturally relevant and responsive ways of conducting research and evaluation studies.  One of our recent projects involved working with Hawaiian-focused charter schools to assess the impact of participation in professional development programs.

The Collaborative Inquiry (CI) project was designed to be culturally relevant and responsive–representing the values of relevance, rigor, respectful relationships, and reciprocity (for more information see, among others, Tibbetts, Faircloth, Villegas and Wheeler (2008)), The CI project extended the conventional purposes of evaluation to prove or improve, by employing a meta-action-research strategy to support the transfer of knowledge and skills learned at the training and assess their impact on teaching and learning. To do this, all participating teachers were required to conduct collaborative inquiry projects. They were encouraged to do their projects in small groups. The charter school teachers’ projects were supported by faculty contracted from a local college of education and culminated in a Ho`ike (demonstration of knowledge or skills).

Hot Tip: Supporting the Inquiry Projects. If you are interested in replicating this approach, it is important to provide ongoing support and scaffolding for the inquiry projects. The simplified action research curriculum and tools provided by the college of education faculty brought what were previously largely abstract concepts to life for the charter school teachers. Multiple “touch points” throughout school year, including visits to the charter schools enabled the college of education faculty to provide advice on the feasibility of project plans to identify potential sources of data that were tailored to each action research project, and helped sustain the momentum of the projects

Hot Tip: Assessing the Impact. As evaluators, our primary challenge was to synthesize information across a wide variety of projects. In the first year, there were 8 projects conducted in 3 different schools with topics spanning nutrition education, behavior management, mathematics, and writing. We approached the analysis as a multiple case study (based loosely on Stake, 2008) and ultimately created a rubric based on the CI project objectives and standards of inquiry. This allowed us to assess and summarize the quality of the inquiry projects.

Rad Resources:

Deloria Jr., V., & Wildcat, D.R. (2001). Power and place: Indian education in America. Golden, CO: Fulcrum Resources.

Hood, S., Hopson, R. K., & Frierson, H. T. (2005). The role of culture and cultural context: a mandate for inclusion, the discovery of truth and understanding in evaluative theory and practice. Greenwich, CT: IAP

Smith, L. T. (1999). Decolonizing methodologies: research and indigenous peoples. London: Zed Books.

Stake, R.E. (2006). Multiple case study analysis. New York: Guilford Press.

Thompson-Robinson, M., Hopson, R., & SenGupta, S. (Eds.). (2004). In Search of Cultural Competence in Evaluation (Vol. 102). Fairhaven, MA: Wiley Periodicals.

Tibbetts, K. A., Faircloth, S., Villegas, M., & Wheeler, L. (2008). Section III: Indigenizing accountability and assessment. In M. K. P. A. Benham (Ed.), Indigenous Educational Models for Contemporary Practice:  In Our Mothers Voice II. New York: Routledge.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

collaboration, collaborative inquiry, indigenous learning

My name is Gerri Spilka and I am the Executive Director of the OMG Center for Collaborative Learning – a consulting firm that assists and assesses change through evaluation, strategic research and planning, grants management and capacity building. Among our many projects, OMG serves as the lead management team on the Robert Wood Johnson Foundation Evaluation Fellowship Program and it is about that program that I will share information today.

The skill set needed to interpret effectively public health, social service, community and organizational social patterns depends on the authentic and diverse experiences within program evaluation teams. Programs and services that are often evaluated traditionally work with underserved and disadvantaged communities, which are themselves becoming more diverse. However, the program evaluation field is becoming less diverse due to a lack of training and placement opportunities for emerging professionals from graduate programs who come from underserved and disadvantaged communities. This creates a problem for the field where, more often than not, high-quality evaluations reflect a process that incorporates diverse perspectives. The RWJF Evaluation Fellowship Program works to ameliorate this disparity.

Resource: The RWJF Evaluation Fellowship Program seeks to extend the skills of participants from diverse backgrounds to include culturally responsive evaluation practices. The Fellows program supports two cohorts annually on parallel paths:

• The Emerging Professionals Program is aimed at those who have graduated from a master’s or doctoral degree program within the last three years and provides a one-year placement, extensive training opportunities, and a support stipend
• The Retooling Professionals Program is focused on mid- or senior-level nonprofit professionals who remain in their current position, but receive additional training to extend their skills to evaluation. Their organization receives a $5,000 financial award and the professional receives a travel stipend.

Both cohorts receive customized training, facilitated by some of today’s thought leaders in field, as well as mentorship, and practical field-based guidance.

Hot Tip: While the deadline from participating in the Retooling Professionals program has passed, the deadline for applying for the Emerging Professionals program is coming up on July 31, 2010. The Emerging Professionals application may be downloaded online from http://www.rwjf-evaluationfellows.org/emerging-fellows.

Resource: The RWJF Evaluation Fellowship Program just released a video entitled Evaluation With a Diversity Lens. This extended discourse among a range of professionals explores the value of diversity in the evaluation profession and may be downloaded for free.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, diversity

I’m Tamara Bertrand Jones and I evaluate programs and services in Student Affairs at Florida State University. I’ve always been intrigued by the ways that different people develop the skills they need in order to be successful in their chosen profession. Fundamental to the success of budding evaluators is the important interaction with successful evaluation role models.

AEA has many programs that promote the exchange of knowledge between novice evaluators and more experienced professionals, including the Pipeline Program, the Graduate Education Diversity Internship Program, and the Multi-Ethnic Issues in Evaluation TIG sponsored mentor lunch held during the annual conference. The programs provide formal ways to link mentors and mentees.

Developing mentoring relationships with other evaluators not only helps to provide the evaluation exposure needed, but can also be a source of personal and professional support for both the mentor and the mentee. For graduate students or new evaluators not already engaged in a mentoring relationship, here are some tips that to help you take the first step in developing a mutually beneficial relationship.

Hot Tip: Develop a list of characteristics you desire in a mentor. Do you want someone who has the practical evaluation skills, communication skills, and/or professional standing you can aspire to?

Hot Tip: Seek mentors through relationships developed by attending professional association meetings and conferences, professional development workshops, and other gatherings where evaluation is a focus. These connections serve as a means to hear about opportunities in the field. In addition, these networks facilitate sharing, collaboration, professional visibility, and skill development.

Hot Tip: After you have developed your list of characteristics and have identified potential mentors, reach out to them. Send an introductory email; schedule a meeting to establish a personal connection. At the meeting, discuss your future goals and career plans, and how you envision the relationship benefitting both you and the mentor. Remember that mentoring relationships work both ways.

Hot Tip: Be clear about expectations of your mentor. Communicate to confirm that his/her expectations align with yours. Do not expect one mentor to meet all of your needs. If, after you have discussed expectations and one mentor cannot provide everything on the list, continue to pursue the relationship with your newly revised expectations. Take the time to find additional mentors that can meet your remaining needs.

Hot Tip: Be open to mentors from a different gender or ethnicity. Just as one mentor cannot meet all of your needs, mentors of different ethnic backgrounds or genders bring different aspects to the relationship based on their lived experiences. These experiences may be different from your own and can add a depth of perspective that you may not have previously considered. This heightened awareness can only serve to improve your evaluation skills.

mentoring, novice

My name is Jenny Jones, and I am an Associate Professor in the school of social work at Virginia Commonwealth University. I have spent a considerable amount of time engaging in evaluation research with numerous community based organizations. The majority of my work is done in communities of color, specifically within the African American community and my work almost always has a social justice nature to it. And, while this work brings me much joy, partnering with community based organizations to do this work can be very challenging. Sometimes the challenge can be so great it can make you question why you do this work.

Hot Tips: Last year while working with a community organization that provides asset building services to families at risk for homelessness, I was reminded of the core principles of evaluation that keeps me grounded in this work.

1. Ethical behavior: Always be honest, respectful, and true to your craft, regardless of what the organization may ask of you. This sometimes requires you as the evaluator to do some value clarifying, so that you do not try to make your values that of the agency.
2. Interpersonal skills: The ability to connect with the community that you are working with is critical. This requires communicating in a way that makes others feel valued. Particularly, for communities of color it is important that you as the evaluator speak and behave in a way that is respectful of their culture.
3. Flexibility: Always allow yourself to be flexible in the process. Being flexible requires that you let go of your agenda and learn to work within the agency’s agenda.
4. Set clear expectations: Tell the organization up front what you expect of them in thus process, and what you can and cannot do. Never promise what you cannot deliver.

Use humor: When all else fails, use humor! Trust me it works.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

CBO, community based organizations

My name is Rodney Hopson. I am a faculty member in the Department of Educational Foundations and Leadership in the School of Education at Duquesne University. For the last several years (with a host of great colleagues in AEA, with Torres Consulting Group and OMG Center for Collaborative Learning with the support of the National Science Foundation and the Robert Wood Johnson Foundation, I have been developing internship and fellowship programs to support graduate and post-graduate students of color and from traditionally underrepresented communities. Embedded in their practice is the understanding and development of Culturally Responsive Evaluation (CRE) approaches and practices in these same communities.

Rad Resource: Adapting the elements of evaluation practice into one that reflects a culturally responsive one, Karen Kirkhart, social work professor at Syracuse University, and I have presented at the AEA/CDC Summer Institute for the last few years. We use the Frierson, Hood, & Hughes chapter in the 2002 National Science Foundation User-Friendly Handbook for Project Evaluation to guide a deeper understanding of how to integrate CRE throughout the evaluation.

Hot Tip: Situate CRE within elements or a framework of evaluation. Whether you use the CDC Evaluation Steps, or another framework to describe steps in evaluation, the key is to embed CRE throughout. For instance, at each stage of evaluation, CRE should be present from the time we prepare for the evaluation to the dissemination and use of results. Below is the CRE framework (with appreciation for the support of Elizabeth Kahl, Syracuse University, who assisted us on graphic and technical design elements), adapted from Frierson, et.al, 2002:

Rad Resource: Learn more about fellowship and internship programs in culturally responsive evaluation through the RWJF Evaluation Fellowship website or AEA’s Graduate Education Diversity Internship website.

Rad Resource: Develop a better theoretical and practical explanation of CRE by reading about its origins, its aim, and its movement as reflected in chapters in any of the following books and special issues:

• The Role of Culture and Cultural Context in Evaluation (Hood, Hopson, and Frierson Eds., Information Age Publishing)
• The SAGE International Handbook of Educational Evaluation (Ryan and Cousins, Eds, SAGE.)
• Handbook of Social Research Ethics (Mertens and Ginsberg, Eds., SAGE)
• SAGE Handbook of Evaluation (Shaw, Greene, and Mark, Eds., SAGE.)
• The Handbook of Leadership Development Evaluation (Hannum, Martineau and Reinelt, Eds., Jossey-Bass)
• In Search of Cultural Competence in Evaluation (Thompson-Robinson, Hopson, Sen Gupta, Eds, Jossey-Bass)
• Responsive Evaluation (Greene and Abma, Eds., Jossey-Bass)

Want to hear more from Rodney Hopson? He is serving as this week’s Thought Leader on the AEA Thought Leader Discussion Series. Learn more at http://www.eval.org/thought_leaders.asp. This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, culture