Hello, we are Katherine Tibbetts and Wendy Kekahio, and both program evaluators doing work within the field of education in Hawai`i. Our work involves using indigenous ways of teaching and learning to inform culturally relevant and responsive ways of conducting research and evaluation studies.  One of our recent projects involved working with Hawaiian-focused charter schools to assess the impact of participation in professional development programs.

The Collaborative Inquiry (CI) project was designed to be culturally relevant and responsive–representing the values of relevance, rigor, respectful relationships, and reciprocity (for more information see, among others, Tibbetts, Faircloth, Villegas and Wheeler (2008)), The CI project extended the conventional purposes of evaluation to prove or improve, by employing a meta-action-research strategy to support the transfer of knowledge and skills learned at the training and assess their impact on teaching and learning. To do this, all participating teachers were required to conduct collaborative inquiry projects. They were encouraged to do their projects in small groups. The charter school teachers’ projects were supported by faculty contracted from a local college of education and culminated in a Ho`ike (demonstration of knowledge or skills).

Hot Tip: Supporting the Inquiry Projects. If you are interested in replicating this approach, it is important to provide ongoing support and scaffolding for the inquiry projects. The simplified action research curriculum and tools provided by the college of education faculty brought what were previously largely abstract concepts to life for the charter school teachers. Multiple “touch points” throughout school year, including visits to the charter schools enabled the college of education faculty to provide advice on the feasibility of project plans to identify potential sources of data that were tailored to each action research project, and helped sustain the momentum of the projects

Hot Tip: Assessing the Impact. As evaluators, our primary challenge was to synthesize information across a wide variety of projects. In the first year, there were 8 projects conducted in 3 different schools with topics spanning nutrition education, behavior management, mathematics, and writing. We approached the analysis as a multiple case study (based loosely on Stake, 2008) and ultimately created a rubric based on the CI project objectives and standards of inquiry. This allowed us to assess and summarize the quality of the inquiry projects.

Rad Resources:

Deloria Jr., V., & Wildcat, D.R. (2001). Power and place: Indian education in America. Golden, CO: Fulcrum Resources.

Hood, S., Hopson, R. K., & Frierson, H. T. (2005). The role of culture and cultural context: a mandate for inclusion, the discovery of truth and understanding in evaluative theory and practice. Greenwich, CT: IAP

Smith, L. T. (1999). Decolonizing methodologies: research and indigenous peoples. London: Zed Books.

Stake, R.E. (2006). Multiple case study analysis. New York: Guilford Press.

Thompson-Robinson, M., Hopson, R., & SenGupta, S. (Eds.). (2004). In Search of Cultural Competence in Evaluation (Vol. 102). Fairhaven, MA: Wiley Periodicals.

Tibbetts, K. A., Faircloth, S., Villegas, M., & Wheeler, L. (2008). Section III: Indigenizing accountability and assessment. In M. K. P. A. Benham (Ed.), Indigenous Educational Models for Contemporary Practice:  In Our Mothers Voice II. New York: Routledge.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

collaboration, collaborative inquiry, indigenous learning

My name is Deshonna Collier-Goubil and I am a young scholar (newly minted PhD) who has had the wonderful opportunity to collaborate with seasoned evaluators and practitioners throughout my graduate education. Most recently I collaborated with a group of practitioners and evaluators to contribute a chapter in a newly released volume specifically written for young scholars. Our book chapter highlights the importance of collaborating with practitioners.

Lessons Learned: Collaboration has many benefits for both evaluators and practitioners. For the evaluator, receiving assistance from practitioners may help to decrease barriers to rich data, the quality of research can be improved with collaboration, and the overall research process can be improved by adding the input and assistance of frontline workers. For practitioners, evaluators can aide in obtaining research funding, clarifying research goals and expectations, and can highlight the need for institutional change or can put sound research behind an excelling program.

Research collaborations can be transformative for both the evaluator and the practitioner. One should approach a collaboration with deliberation, willing to both teach and learn. An array of cognitive, technical, and interpersonal skills are needed to develop and maintain effective collaboration. Having a firm grasp on communication, trust, honesty, respect, commitment, and flexibility can make or break a collaborative relationship.

Keep in mind, however, that just as benefits exist barriers may also arise during a collaborative effort. Evaluators should try to be as open and honest as possible with practitioners in initial negotiations to attempt to eliminate issues popping up in the future. Discussing division of labor, purpose of the collaboration, timelines for completion, how research will be conducted, data ownership, and how results will be communicated and disseminated for example can eliminate misunderstandings about these things in the future. Other barriers to be cognizant of are blurred roles, divergent perspectives, differences in degree of institutional support, competing and conflicting goals, and communicating difficult results. Despite experiencing barriers, evaluators should embrace adversity and persist in the collaborative relationship. Overcoming these barriers can strengthen the collaboration.

Overall, in a model collaboration, evaluators and practitioners develop shared goals, with consensus on a few key practice and research standards. The investment of time, resources, effort, flexibility, and the willingness to think outside of the box are required. Members of the collaboration learn to enter each other’s world and appreciate the others perspective. This is where the true learning begins.

Would you like to discuss evaluator-practitioner collaboration more with Deshonna and her colleagues? She’ll be contributing to a roundtable on the topic this November at Evaluation 2010, AEA’s Annual Conference.

collaboration, practitioners

My name is Kirsten Rewey and I am a Senior Research and Evaluation Associate at ACET, Inc. Two years ago ACET was selected to provide evaluation services for a federally-funded Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) implemented by the Minnesota Office of Higher Education (OHE). As part of the services, OHE asked ACET to create a research plan to determine the impact of GEAR UP on students’ academic preparedness using Minnesota’s statewide academic test (Minnesota’s No Child Left Behind assessment).

Hot Tip: ACET used the following approach and finds that it facilitates staff development and client selection of a research design:

1. Identify data available from the client: In order to develop a solid research plan, ACET needed to know what data OHE/GEAR UP were currently collecting. One of our first planning meetings was to discuss the types of data available in the GEAR UP database, the format of the data, and retrieving the data. During the meeting ACET obtained the list of variables maintained in the database, their descriptions, and the timeline for data entry and retrieval.
2. Identify data available from other sources. Once ACET knew the data that would be available from GEAR UP, other sources of data needed to be identified and catalogued. From previous work, ACET staff knew there was a substantial amount of school-level, public-record data available from the Minnesota Department of Education. For example, the Minnesota Department of Education publishes demographic information for the state’s public schools on their website including number of students enrolled, number eligible for free or reduced-price meals, and number who have limited English proficiency. For some of the variables, demographic data are available by grade-level and ACET can hone in on a specific grade of interest. Other data, such as individual demographic data and test score results, are only available pending approval from the district or the state. In order to obtain this data OHE and ACET wrote a formal application to the district to receive selected data from their databases.
3. Create an array of research options and present them to the client. After ACET identified available data, the staff created an array of research options for the client. ACET typically creates a variety of designs that vary in research/experimental control, types of conclusions which can and cannot be drawn, and cost. The options are presented to the client in a matrix with a brief description of the design, the advantages, and challenges for each. And because clients have an array of research options and are alerted to the advantages and challenges of each design, they can select the research design that best meets their needs and budget.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Kirsten, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

collaboration, Evaluation planning, GEAR UP

I’m Cheri Hoffman, a program evaluator with Centerstone Research Institute in Nashville, Tennessee, evaluating two System of Care grants for children’s mental health. The values of a System of Care demand that the entire process, including the evaluation, be youth-guided. We recently trained selected members of the program’s youth council in research and program evaluation skills using the “Stepping Stones” curriculum by Youth in Focus. The youth then created a performance aimed at “Stomping Out the Stigma of Mental Illness” and measured the impact that performance had on people’s perceptions of youth with a mental health diagnosis using a pre- and post-survey and a focus group.

The week-long experience was eye opening on many levels. The quality of the work that was produced far exceeded our expectations in many ways. Here are some of the lessons we learned:

Hot Tip – Trust the Process: Young people are capable of much more than we tend to believe. As we taught the Stepping Stones curriculum, the youth went much deeper much faster than we expected. They were able to identify root causes (lack of early screening, discrimination, the need for education, not enough parent and youth involvement) of the issue they focused on (stigma of mental illness). As a result the youth left the week with not only new skills and a successful research project under their belts, but also with a clear direction for future youth-led action research and evaluation.

Hot Tip – Be Flexible: The young people we worked with have been diagnosed with a mental illness at some point in their lives. Many of them had ADD or ADHD. Putting nine youth with attention difficulties in a room for 8 hours a day might not have been the best way to approach this task (although it worked)! It demanded that we be flexible with the way we approached the subject matter. Make sure the techniques you use are relevant for your youth. If something isn’t working, scrap it and try a different approach.

Rad Resource: Youth in Focus has an excellent curriculum for training young people in youth-led action research and program evaluation. Find out more at www.youthinfocus.net

Rad Resource: The Institute for Participatory Action Research and Design did a project with youth called Echoes of Brown that was the inspiration for the project and performance that we created. For more information, you can email them at parinstitute@gmail.com or see more about Echoes of Brown at http://web.gc.cuny.edu/che/projectmf.htm

Rad Resource: Kim Sabo Flores has the best book out there on the topic, called Youth Participatory Evaluation:  Strategies for Engaging Young People.*

*Youth Participatory Evaluation is published by Jossey-Bass, an AEA publishing partner. AEA members receive 20% off on this title when ordered directly from the publisher – just sign in to the AEA website at http://eval.org/ and select Publications Discount Codes from the Members Only menu.

collaboration, mental health, youth

My name is Sally Honeycutt, and I am the Evaluation Team Lead for the Emory Prevention Research Center (EPRC).  The EPRC primarily works in rural Southwest Georgia and focuses on strategies to make rural home and neighborhood environments more supportive of cancer prevention behaviors, including physical activity and healthy eating.  We also provide technical assistance to enhance the capacity of community organizations to plan, conduct, and evaluate evidence-based cancer prevention programs.

When we begin working with a new organization, one of our first steps is to develop a program logic model.  A logic model is a graphic representation of how a program is supposed to work.  It describes the relationship between a program’s inputs, activities, outputs, and outcomes.  People who are unfamiliar with logic models are often intimidated by them, but logic models are actually a great, common-sense way to clarify and explain your program.  In developing a logic model, keep these tips in mind:

• Logic model development should always be a collaborative task, drawing on the input of multiple stakeholders with different perspectives on a program.  (Big sticky-notes are a handy tool for group logic model work.)
• A logic model should be logical.  All of your links (A à B) should make sense and be plausible, not wild leaps of faith.  Links should be supported by research, examples from other programs, or previous experience with your population.
• Your logic model should be complete, with no dead-ends.  Starting with your inputs, you should be able to follow every possible path and wind up at a long-term outcome.  Starting with each of your outcomes, you should be able to track back to an initial input.
• Your logic model should be a living document that you revisit and revise as the program changes and matures over time.

Rad Resource: “Enhancing Program Performance with Logic Models”  is a short online course from the University of Wisconsin-Extension.  The course is a clear, user-friendly introduction to logic models.  http://www.uwex.edu/ces/lmcourse/#

Rad Resource: The Logic Model Development Guide from the W.K. Kellogg Foundation provides detailed, practical guidance in developing and using logic models, including forms and templates that you can adapt and use on your own project.  http://www.wkkf.org/knowledge-center/resources/2010/Logic-Model-Development-Guide.aspx

Rad Resource: The University of Kansas Community Tool Box has a great section on logic models, including many examples of logic models that use a variety of graphical formats.  http://ctb.ku.edu/tools//section_1877.htm

Rad Resource: You can also download our handouts from our session at the 2010 AEA/CDC Summer Evaluation Institute on Logic Models as a Platform for Program Evaluation Planning, Implementation, and Use of Findings in the AEA Public eLibrary.  http://tinyurl.com/247vlxb

collaboration, institute, logic models

My name is Beverly Parsons and I’m the executive director of InSites,  a non-profit research, evaluation, and planning organization. We use a systems orientation and conduct evaluations related to education, social services, community change, and health. I’m an AEA board member. I have a tip about how to build evaluation capacity through a type of Community of Practice.

Hot Tip: Consider using Communities of Learning, Inquiry, and Practice (CLIPs) to build evaluation capacity and develop a culture of inquiry across an organization.

CLIPs (a type of Community of Practice) are informal, dynamic groups of organizational members who learn together about their professional practice. They gather and analyze data about a question of importance to them. CLIP members learn an evaluative inquiry process with three basic steps: (1) design the inquiry; (2) collect data; and (3) make meaning and shape practice. The process has some special features to create continual renewal in the organization.  At Bakersfield College where we developed this process under a National Science Foundation grant, the CLIP members are faculty and staff. They focus their inquiries on student learning and success.

Typically, each CLIP consists of three to seven people with one person as the group facilitator. An overall CLIP Guide supports the work of multiple CLIPs at the organizational level, builds strategic linkages among the CLIPs, and connects the whole process appropriately to the organization’s other processes and initiatives. CLIPs support, and are supported by, the broader organization’s goals. CLIPs are adaptable for use in a variety of settings.

Hot Tip: The following features of CLIPs are especially important:

• Within general parameters including a focus on the organization’s core mission, CLIPs have the freedom to select their own members and topics; set their schedules; determine their budget allocations; and tailor the inquiry process. This freedom builds internal motivation among participants and helps ensure use of results.

• The CLIPs simultaneously focus on collaboration and inquiry, building a synergy that motivates completion of their investigation.

• The CLIPs use guiding principles that create an energizing learning environment and promote a natural flow from inquiry to change in practice. The CLIP members are learning at all stages of the inquiry process and readying themselves for a natural shift in practice.

Rad Resources: An overview video and modules about the CLIP process are free through InSites at www.insites.org/clip. Also an article, Evaluative Inquiry in Complex Times, that addresses the link to complexity science is available at http://www.insites.org/clip/clip_reports.html .

Feel free to contact me if I can be of assistance (bparsons@insites.org). I love working with the CLIP process. Perhaps part of the reason is it’s the only time I got a standing ovation from faculty (CLIP members) for my work related to evaluation!

CLIPs, collaboration, communities of practice, learning communities

My name is Cassandra O’Neill and I’ve been a consultant for the past 10 years. I’m a member of a network of consultants and coaches called Wholonomy Consulting.  I’m also the President-Elect for the Arizona Evaluation Network and a member of the AEA Local Affiliate Council. A theme in my work is using effective engagement for high impact collaboration.  I have several resources to share with others interested in increasing the effectiveness and impact of collaborations.

Hot Tip: Collaboration is one of those phrases like strategic planning – that can mean something different to every person. Collaboration is talked about all the time without any clarification or definition. One of the most useful things to do with a group that is collaborating or wants to – is to explore the mental models that people have about collaboration and related ideas such as resources.

I have a pretty quick and easy way to do this which helps groups identify the different beliefs and assumptions about collaboration that are in the room. Why is this so important?  To answer this question, I am going to tell you a story about something that happened recently. I just gave a workshop on high impact presenting. In this workshop I demonstrated about 12 different ways to present information without lecturing or telling people the information.  At the end of the workshop I was reading the Appreciative Reviews and someone wrote – I didn’t learn anything that I can use in presenting.  I realized that this person’s mental model of presenting as lecturing was so strong, that nothing that was experienced in the two hours could be connected to presenting — since it wasn’t lecture. Think about the missed learning opportunities!

An example of how important exploring mental models, beliefs and assumptions can be for effective collaborations — is the impact of the beliefs that people have about resources.  When I ask people to identify a belief they have about resources, there is usually one of two responses shared.  Either the belief that resources are unlimited and abundant is shared or the belief that resources are scarce and limited. You can see how knowing people’s beliefs about this topic influence everything else that is involved in collaboration.  This exploration of beliefs may allow people who believe resources are scarce to see that this may be a limiting belief rather than a fact. See a link to my blog for tips on how to do this with a group.

http://bit.ly/mentalmodels

And you can use any word or set of words with a group. Once I had people do this exercise three times with the following words – collaboration, resources, and sustainability.  You could do it with the word evaluation and quickly learn about the beliefs and assumptions that people have about evaluation.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

assumptions, collaboration, facilitation, presentations

Hi.  I am David Fetterman, President and CEO of Fetterman & Associates (with 25 years of experience at Stanford University).  I work in many areas of evaluation including  as you may know – empowerment evaluation.  One of my primary concerns is fostering collaboration in evaluation.  My Hot Tip and Cool Trick both spring from the same resource: Google.

Hot Tip: Google Docs and Spreadsheets. Google Docs and Spreadsheets enables people to work on a document collaboratively – all at the same time.  Instead of using track changes in Word and sending it from one person to another, this tool allows you to all creatively produce a document.  For example, we use it in empowerment evaluations to invite colleagues/clients to participate in the development of a critical documents for the group, e,g., 1) mission statement; 2) taking stock -  assessing what has been done to-date; and 3) plans or the future. 

It also makes the group’s work transparent to the larger community and possibly funders.  It provides a visual reassurance or validity to the group’s work.  It also reassures the funder that diverse groups can and do work together.  As long as there is critical information in the folder people will gravitate to the information.  For more information, go to the page: http://docs.google.com/

Cool trick: Collaborative Web Pages. The next step in this process is inviting people to create their own web pages on the same group site. This is precisely what Google’s collaborative web sites allows.  Clients can post pictures of their activities, descriptions, and even attach documents.  The group can control who is invited to see the web site and contribute to it.  It is a powerful tool to foster collaboration. http://sites.google.com

I hope you experiment with some of these tools with your family to stay in touch and build strong healthy relationships.  The more comfortable you are using these tools with your family the more prepared and comfortable you will be using these tools with colleagues and clients.  Enjoy!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

collaboration, google, technology

My name is Cassandra O’Neill and I’ve been a consultant for the past 10 years. I’m a member of a network of consultants and coaches called Wholonomy Consulting.  I’m also the President-Elect for the Arizona Evaluation Network and a member of the AEA Local Affiliate Council. A theme in my work is using effective engagement for high impact collaboration.  I have several resources to share with others interested in increasing the effectiveness and impact of collaborations.

Rad Resource: I wrote an article for the Charity Channel on High Impact Partnering. In this article, I talk about a set of practices called the Seven Norms of Collaboration described by the Center for Adaptive Schools. Research has shown that high performing teams utilize these norms.  The 7 norms are as follows:

• Promoting a Spirit of Inquiry .Building a culture of inquiry promotes the exploration of what people mean rather than making assumptions.
• Pausing .Pausing is one of the most powerful practices. Pausing allows time for thinking and reflecting. Learning comes from reflecting, not from doing.
• Paraphrasing .Starting with a paraphrase of what you heard another person say is one of the most helpful ways to clarify what is meant. When hearing a paraphrase people often realize that what they said isn’t actually what they meant.
• Probing. Asking for more details or for clarification is very effective at increasing understanding.
• Putting ideas on the Table .One of the principals of successful brainstorming is that no criticism is allowed. This is because idea generation is richer when people focus exclusively on generating ideas.
• Paying Attention to Self and Others .This is something that has a big payoff when working with groups. Watching how people are responding and reacting will allow for changes to be made in the conversation that will positively impact the group.
• Presuming Positive Intentions . Often people get caught up in attributing negative intentions for a behavior they are noticing, without doing any checking. When people report others negative intentions, it can lead to misunderstandings which are not based in reality. Further investigation often reveals that there was a positive intention.

Hot Tip: Using the Seven Norms to Build Skills for High-Impact Collaboration.  Skilled facilitators often use these norms. If you want to start building your collaboration skill set, you can assess your use of these Seven Norms individually and in the groups that you are part of. This offers a way to look at what your meetings and conversations look like now, and identify opportunities for growth. Like any new skill, they can’t be learned overnight. They are something you can aspire to in your organization and among your partnerships.

In addition to using these norms when collaborating with other organizations, boards and executive directors can use them within their own communication as a way to increase proficiency, and model the behavior they would like to see in their staff.

See the Center for Adaptive Schools Website to learn more about these norms and how to assess them at http://www.adaptiveschools.com/inventories.htm

The following links are to my article on High Impact Partnering.

• http://www.wholonomyconsulting.com/docs/high-impact-partnering-part1.pdf

• http://www.wholonomyconsulting.com/docs/high-impact-partnering-part2.pdf

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

collaboration

My name is Kim Norris and I am the Evaluation Coordinator for University of Maryland Extension’s Food Supplement Nutrition Education (FSNE) program. Included in my work is to assist educators in developing useful strategies for assessing the impact of their work on our target audience, limited-income, and often, low-literacy, populations.

Hot Tip: Utilize Audience Response Technology System in group class settings for immediate, anonymous assessment and feedback.  The “clicker” technology, as we sometimes refer to it, allows questions to be asked both orally and in writing, allows individuals to respond anonymously, and allows for immediate feedback after responses for both educator and audience.  We used these recently with our own educators to ask questions that, in a setting in which anonymity was not guaranteed, could lead to false answers due to high motivation to fall within socially acceptable norms.  Since results are calculated and visible to all on the spot, group responses can be reviewed, analyzed, interpreted, and addressed by the group, thereby increasing potential for empowering participants.

Other advantages of using this easy-to-teach technology include the ability to:

• collect data from larger numbers of people in a shorter amount of time
• eliminate data entry errors by direct transfer of electronic data to a data base
• engage technology-averse populations in computer technology to their benefit
• help low-literacy populations participate in surveys as respondents
• provide confidentiality for respondents

The technology can lead to missing data if not preceded by sample questions and as group sizes become larger or less engaged.  Studies are underway to better understand strengths and limitations of the technology as an educational and evaluation tool.

Rad Resource: A Bibliography of Selected Readings on Audience Response Systems: http://bit.ly/audienceresponsesystems.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment, participation, technology