Hello! We are Carolyn Sullins and Ladel Lewis of Meaningful Evidence, LLC and we have years of experience evaluating children’s mental health systems of care. These evaluations were challenging because attention to local communities’ cultures was crucial – but so was adherence to a national battery of questionnaires and to local regulations. The community vs. academic/professional cultures was a more salient divide than race, religion, or ethnicity. This divide was sometimes challenging but always crucial to bridge.
Hot Tips: Get advice from members of your population. One way is inviting representatives to your evaluation team meetings. It can be more helpful to attend their meetings! Members will feel more comfortable and candid on their own “turf,” and more of them will be there. Breaking the ice with humorous examples of cross-cultural misinterpretations in research/evaluation has helped.
Here are some questions that may be helpful:
- Which survey items may be unclear? How can they be clarified? What local terms can be used?
- What recruiting methods and incentives are likely to work?
- How can we help people feel more comfortable answering questions?
- What items are people likely to refuse to answer or not answer honestly?
- What questions should we add that will be helpful?
Lessons learned: Some surprising distinctions between the community culture and the academic/professional culture involved conceptions of confidentiality, even for interviews that included sensitive questions:
- For confidentiality and accuracy, academia often insists that interviewers cannot interview people they know. Achieving this goal can clash with the goal of having ample consumer representation on the evaluation team. Our consumer representatives found this restriction condescending and counterproductive.
- Conducting interviews in a fast food restaurant can be taboo in the academic world, but several of our participants specifically requested this.
- When interviews are conducted in an interviewee’s home (the request of the overwhelming majority of interviewees), what does an interviewer do when a family member enters the room during an interview? Respecting the interviewee’s preference may clash with the academic expectations of confidentiality. Still, when in doubt, the answer is obvious: Pause, and ask if they would like for you to continue.
- How should interviewers act if they encounter their interviewees in public? Traditional academia recommends a cool detachment to protect privacy, but interviewees may find this cold and an indication that they had been callously “used.”
Rad Resource: While Morse’s 2003 article “Bending the ivory tower: Communities, health departments, and academia” on Public Health, it is relevant to other areas of human service.
Paying attention to the preferences of our population while maintaining our academic integrity was a balancing act, not a contradiction. Attending to both helped us with participant retention and the accuracy of our data.
The American Evaluation Association is celebrating SW TIG Week with our colleagues in the Social Work Topical Interest Group. The contributions all this week to aea365 come from our SW TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.