I’m Jim Burdine and a few weeks ago I posted a “tip-a-day” (see post here) on presenting community health assessment data back to the community. This is part two of that discussion. Last time I focused on “what to present” this posting focuses on “how to present . . . with results.”

There are two goals for any assessment presentation: (1) reporting the data, and (2) identifying next steps for improving health status. I’ve used a format that is effective in a variety of settings. This assumes, as discussed in my last posting, that the assessment has been a “participatory process.”

Hot Tip: Hold a “regional health summit.” Folks invited include those who have participated in various aspects of the assessment as well as others representing a broad cross section of community sectors. My experience has largely been in medium to smaller communities (500K to 10K population) but participation is roughly the same – about 150-250 folks is a good turnout. The event is split into two parts and can be accomplished in one day or two. The two parts are a presentation of the data, followed by a workshop. Ideally the data presentation is limited to an hour and the actual presentation, or much of it, is by a community member. Avoid the temptation to have more than a couple of speakers – “inclusiveness” is a nice idea but counterproductive in this context. You can invite community members to do “welcomes” and other remarks.

Following the data presentation is another opportunity to develop for community buy-in. In the workshop, we ask folks to select among the “findings” of the assessment (e.g., “access to mental health,” “childhood obesity,” “rural transportation”) and sign up for those topics. Have rooms/tables setup for each group. A convener appointed for each group opens the discussion and is responsible for encouraging folks to commit to participating in an ad hoc task group on this topic that that would agree to meet 2-3 times over the next 2-3 months. At the end of that time (3 months) the larger group is reconvened to hear reports from the various ad hoc task groups and then to endorse a regional health improvement strategy developed at that session (based on the group reports). During the 2-3 month work period, each team reviews pertinent findings and identifies “best practices” of interventions related to that issue. The group then identifies one or two priority activities that incorporate those best practices as adapted to your community.

By following this model, what might have been just a data presentation yields a comprehensive regional strategy with multiple sub-parts that is endorsed by the entire community (at least as represented by the individuals and agencies participating in the overall process).

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

communication, participatory

I’m Cheri Hoffman, a program evaluator with Centerstone Research Institute in Nashville, Tennessee, evaluating two System of Care grants for children’s mental health. The values of a System of Care demand that the entire process, including the evaluation, be youth-guided. We recently trained selected members of the program’s youth council in research and program evaluation skills using the “Stepping Stones” curriculum by Youth in Focus. The youth then created a performance aimed at “Stomping Out the Stigma of Mental Illness” and measured the impact that performance had on people’s perceptions of youth with a mental health diagnosis using a pre- and post-survey and a focus group.

The week-long experience was eye opening on many levels. The quality of the work that was produced far exceeded our expectations in many ways. Here are some of the lessons we learned:

Hot Tip – Trust the Process: Young people are capable of much more than we tend to believe. As we taught the Stepping Stones curriculum, the youth went much deeper much faster than we expected. They were able to identify root causes (lack of early screening, discrimination, the need for education, not enough parent and youth involvement) of the issue they focused on (stigma of mental illness). As a result the youth left the week with not only new skills and a successful research project under their belts, but also with a clear direction for future youth-led action research and evaluation.

Hot Tip – Be Flexible: The young people we worked with have been diagnosed with a mental illness at some point in their lives. Many of them had ADD or ADHD. Putting nine youth with attention difficulties in a room for 8 hours a day might not have been the best way to approach this task (although it worked)! It demanded that we be flexible with the way we approached the subject matter. Make sure the techniques you use are relevant for your youth. If something isn’t working, scrap it and try a different approach.

Rad Resource: Youth in Focus has an excellent curriculum for training young people in youth-led action research and program evaluation. Find out more at www.youthinfocus.net

Rad Resource: The Institute for Participatory Action Research and Design did a project with youth called Echoes of Brown that was the inspiration for the project and performance that we created. For more information, you can email them at parinstitute@gmail.com or see more about Echoes of Brown at http://web.gc.cuny.edu/che/projectmf.htm

Rad Resource: Kim Sabo Flores has the best book out there on the topic, called Youth Participatory Evaluation:  Strategies for Engaging Young People.*

*Youth Participatory Evaluation is published by Jossey-Bass, an AEA publishing partner. AEA members receive 20% off on this title when ordered directly from the publisher – just sign in to the AEA website at http://eval.org/ and select Publications Discount Codes from the Members Only menu.

collaboration, mental health, youth

I’m Jim Burdine, Assistant Dean for Public Health Practice and Co-PI/Director of a Prevention Research Center at the School of Rural Public Health, Texas A&M. Over the last 30 years I’ve used community health status assessment as both a community organizing tool and intervention planning tool. In more than 200 different communities (including multiple iterations in the same community) I’ve presented/seen data presented from community assessments in a number of different formats with varying degrees of success and failure.

Lessons learned: What I’ve observed as most important, is presenting the data to them in a manner that matches the expectations of the audience – audiences, really. In other words, the groundwork you’ve already done in your assessment process (hopefully incorporating community-based participatory research principles) should dictate the format. If community members have been involved in planning, conducting and analyzing the data, they should play the major role in presenting the results. If they have been more passively involved, they may expect a “report” FROM you. Obviously the degree of “buy in” to the findings varies dramatically as a function of the degree of participation.

Given that starting place, the next challenges you face are: (1) the sheer volume of information you have to present, and (2) the variation in sophistication around understanding data within a community audience. So first, you have to accept that you can’t present EVERYTHING. You need to decide what are the key points you want to make and focus on those. We’ve all sat through a presentation where somebody reads us the demographics of a community or lists every chronic disease ever found in that population and an hour later you’ve learned nothing new. As a general rule I don’t both to report anything unless it is (1) statistically significantly different from some external reference point (e.g., Healthy People 2010, a state or national rate) and (2) unless there is something that could likely be done locally to impact that problem (it’s actionable).

If well-planned, you will have representatives from all community sectors in your audience (e.g., health care, business, elected officials, religion, education, the media, consumers and representatives of special interests/special needs groups). So you have to decide on what common denominators (e.g., educational attainment, exposure to health statistics) you are going to assume for your audience. You need to be comfortable with knowing that some aren’t going to understand everything you say and other are going to be bored with your “simplistic” presentation. Don’t make the mistake of trying to explain every point to each group n your audience. It just frustrates them and makes for a very ineffective presentation. Plan to do multiple presentations for different audiences rather than a “one size fits all” presentation.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

community, presentations, Public Health

My name is Susan Kistler. I am AEA’s Executive Director and I contribute each Saturday’s aea365 post. If you didn’t have the opportunity to join your colleagues at the AEA/CDC Summer Evaluation Institute this past month, you missed some great sessions  - the agenda may be found online.

Hot Tip: Over 80% of the presenters have shared the handouts in the AEA Public eLibrary! These materials, ranging from slides to bibliographies, to resource lists, are available to anyone to kickstart your own learning (and perhaps whet your appetite to join us next year in Atlanta – I’ll keep you informed about dates as they are finalized via aea365).

• Bringing Funders, Researchers/Evaluators, and Practitioners Together to Achieve Outcomes: The Interactive Systems Framework (ISF) for Dissemination and Implementation (Wandersman & House)
• Conducting and Using Success Stories for Capacity Building (Lavinghouze & Price)
• The Democratization of Data Insight: 20 Free Tools for Data Access, Analysis, and Reporting and Considerations for Their Use (Kistler)
• Dissemination and Implementation Research (Emshoff & Noonan)
• Enhanced Group Facilitation: Techniques and Process (Dewey)
• Evaluating Environmental Change Strategies… Beyond Counting the Number of Laws Passed (Pettibone)
• Every Picture Tells a Story: Flow Charts, Logic Models, LogFrames, Etc. What They Are and When to Use Them (Chapel)
• Handling Data: From Logic Model to Final Report (Barrington)
• How Policy Is Made and How Evaluators Can Affect It (Grob)
• Improving Survey Quality: Assessing and Increasing Survey Reliability and Validity (Germuth)
• An Introduction to Economic Evaluation (Corso)
• Introduction to Evaluation (Apt)
• Leading Concepts in Community Health Evaluation (Goodman)
• Lessons From the Field: Ways to Improve Your Evaluation Practice (Barrington)
• Logic Models as a Platform for Program Evaluation Planning, Implementation, and Use of Findings (Honeycutt)
• Methods for Analyzing Change Over Time (McKnight)
• Minding Your Mind (Grob)
• Performance Management in the Obama Administration (Newcomer)
• Program Evaluation and Program Improvement: Challenges, Opportunities, and New Directions at the Centers for Disease Control and Prevention (Collins)
• RealWorld Evaluation (Rugh)
• Qualitative Evaluation Approaches (Goodman)
• Quantitative Methods for Evaluation (McKnight)
• Rethinking Social Change Using Social Media (Fine)
• Strengthening Evaluation Through Cultural Relevance and Cultural Competence (Kirkhart & Hopson) also available here
• Systems Level Evaluation of Communities of Practice (Hegedus & Jernigan)
• Transformative Mixed Methods Evaluation (Mertens & Bledsoe) Plus list of references
• Translation of Evaluation Capacity Building Strategies to Community-Based Organizations Conducting Health Promotion Activities: Tools, Tips and Lessons Learned (Akintobi)
• Using GIS in Evaluation (Meyers)
• Using Program Evaluation to Improve Nonprofit Outcomes (Newcomer)
• Using the Guiding Principles to Improve Your Evaluation Practice (Goodyear)
• Writing Questions that Elicit What You’re Looking For (Pohl)

No tags

My name is Lija Greenseid. I am a Senior Evaluator, with Professional Data Analysts, Inc. in Minneapolis, MN. We conduct evaluations of stop-smoking programs. Smokers generally have lower education and literacy levels than the general population. Therefore, we want to make sure the materials we develop are understandable to smokers.

Rad Resource: Use a “readability calculator” to check the reading-level of your written materials. I have used this with program registration forms, survey instruments, consent statements, and other materials. Not surprisingly, the first drafts of my materials are often written at a level only grad students (and evaluators) can understand. With a critical eye and a few tweaks I can often rewrite my materials so that they are at an eighth-grade reading level, much more accessible to the people with whom I want to communicate.

A good Readability Calculator can be found here:

http://www.editcentral.com/gwt1/EditCentral.html

It provides you with both a reading ease score, and a number of different measures of the US school grade level of the text.

This blog posting is rated at a high-school reading level. Do you agree?

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

esl, Instruments, Literacy, Readability, reports, Smoking, surveys

Greetings colleagues!  I am Elizabeth Harris, Ph.D., Vice President of Evaluation, Management and Training Associates, Inc. (EMT).  The focus of this blog is a free resource for measuring youth resiliency that we developed out of necessity.  For over 25 years, we have focused our evaluation, technical assistance, and training work on the prevention of substance abuse and other behavioral health needs; on policies and programs promoting the positive social-emotional and behavioral development of children and youth; on family service needs; and on related fields of public health.

The evaluation of youth initiatives represents a critical aspect of our work.  Unfortunately, as we sought to evaluate one of the largest national federally-funded youth initiatives, available instruments reflected the prevailing thinking that the only aspects worth measuring were attitudes of despair and hopelessness and illegal and risky behaviors.   Available measures did not reflect the reality of programmatic objectives at the local level, many of whom provided activities grounded in youth development theory, seeking to strengthen existing assets.

Resource:  In order to honor the evaluation tradition of measuring what programs actually intend to impact, we developed an instrument of youth resiliency, the Individual Protective Factors Index (IPFI).  The IPFI is a 71-item questionnaire which provides a single measure that captures the various protective factors that have been identified as contributing to individual resiliency in youth between the ages of 10-16 years who may be at risk for developing substance use and other problems. For the national study, we combined the IPFI with federal GPRA measures in order to also measure the risky behaviors and attitudes that were required by the funding agency.

The instrument has been used extensively and is the product of extensive conceptual development and empirical testing, including norming and validation studies on 2,416 youth in 15 states nationwide.

The IPFI is available free of charge to our colleagues and can be downloaded at http://www.emt.org/ipfi.html.  A Spanish language version is also available.  Our only request is that you share the results of your evaluation with us.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

instrument, resilience

My name is David McDonald and I am the Director of a small Australia-based consultancy Social Research & Evaluation Pty Ltd, and a part-time academic at the National Centre for Epidemiology and Population Health, The Australian National University, in Canberra, ACT, Australia. My consultancy work is largely policy analysis and policy & program evaluation in the alcohol, tobacco and other drugs (ATOD) field, and my academic work is in Integration and Implementation Sciences.

Rad resource: People who use and/or experience problems with alcohol, tobacco and other drugs are typically the objects of stigma and discrimination in the community, and too often receive second-rate services in ATOD treatment and other human service agencies. For this reason, it is important that such services conduct regular review of service user satisfaction with the services provided. The eight-item Client Satisfaction Questionnaire (CSQ-8) is a simple and effective evaluation instrument for this purpose. It has sound psychometric properties and is easy to administer. It produces a single satisfaction score from the eight questions. I have used it in a system-wide ATOD service user satisfaction survey in which I used the numerical scores derived from the CSQ-8 as part of a larger client satisfaction survey instrument. I have also used it in an evaluation in which I asked drug treatment clients to complete the instrument, and then I used their responses as a basis for discussing their experiences and assessments of the treatment program. This produced a productive mix of qualitative and quantitative evaluation data.

[Update: The link to the instrument that originally appeared here has been removed by the blog administrators. Unfortunately, the link had been made available by a third party inappropriately. The CSQ Scales, including the CSQ-8, are copyright and cannot be used without the written permission and payment of use fees to Tamalpais Matrix Systems, LLC. Information about use of the CSQ-8 and the other CSQ Scales can be found at:  www.csqscales.com. Please accept our apologies to CSQScales and to our readers for the error.]

The instrument’s psychometric properties and information about its origin are provided in Fischer, J & Corcoran, K 2007, Measures for clinical practice and research: a sourcebook, 4th ed, vol. 2, Oxford University Press, Oxford, pp. 155-6. Its use is discussed and illustrated in ‘Workbook 6: client satisfaction evaluations’, part of the Evaluation of psychoactive substance use disorder treatment workbook series published in 2000 by the World Health Organization in conjunction with the United Nations International Drug Control Programme and the European Monitoring Centre on Drugs and Drug Addiction.

The CSQ-8 is suitable for clients of most human services, not only those specialising in treating people with substance use disorders.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

instrument, satisfaction

We are P. Antonio Olmos-Gallo, Kathryn K. DeRoche, and C.J. McKinney.  We work at the Mental Health Center of Denver, a non-profit community mental health center which has become the de-facto mental health authority for the City and County of Denver. On any given day we provide services to about 4000 adults and 1000 youth. In addition to our duties associated with the collection and analysis of outcomes for the center, we also provide evaluation services to the multitude of Federal and local grants our center receives every year, which include not only treatment, but also prevention services to a multitude of individuals across Colorado. We have made a very concerted effort to involve multiple stakeholders in every evaluation we conduct: that includes youth, parents, adult consumers, and clinical and manager-level individuals.

In the last 5-7 years, a big part of our work has concentrated in the development of instruments to measure recovery from mental illness. Although we have degrees in psychology (either MA or Ph.D.), our training is not in clinical psychology, therefore we rely heavily on the expertise of multiple people for clinical interpretation of the data. We also teach graduate and undergraduate statistics and experimental methods at different colleges and universities in the State of Colorado. We believe this unique combination provides us with an edge when it comes to doing evaluation.

Hot tip: Do not short-change your evaluation efforts by trying to use techniques/tools that may not fully answer your questions. In our private practice, we sometimes have to step in to evaluate programs that never managed to answer the key questions because the techniques were not the most appropriate. Evaluators are sometimes afraid to use anything more sophisticated than a t-test or a chi-square, because “stakeholders do not understand statistics”. This takes us to our next hot tip:

Hot tip: Despite what you and your stakeholders may think, they can understand very sophisticated evaluation concepts if you give them enough background and there is willingness to learn (and to teach). During the last 5 years, our stakeholders have learned about Logic models, instrument reliability and validity, Item Response Theory (Rasch models), Hierarchical Linear models, Cost-benefit analysis, and more recently, Quality control charts. They may not believe it, and may not even accept it in public, but they are able to understand when an instrument is not working (and the importance of that), understand the power of predictive models, and the importance of using the right tools for improving day to day operations. More importantly, they also understand the limitations of some of these tools.

In 2009, we shared at AEA several examples of how we have managed to explain our stakeholders sophisticated concepts in evaluation and statistics in very intuitive ways. Please visit our website (http://www.outcomesmhcd.com/pubs.htm) to see that and many other examples of our work in evaluation of mental health.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

involvement, statistics