AEA365 | A Tip-a-Day by and for Evaluators

CAT | Alcohol, Drug Abuse and Mental Health

My name is Ann Price and I am the President of Community Evaluation Solutions, Inc. (CES), a consulting firm based in Alpharetta, Georgia just outside of Atlanta. I am a community psychologist and work with many federally-funded, community-based alcohol and substance abuse prevention programs. Across the state of Georgia, whether in rural areas or suburban areas of the state, community coalitions are working together to prevent youth substance use using environmental strategies.

Several of the Georgia prevention collaboratives with whom I work are using Positive Social Norms (PSN) campaigns to prevent youth substance abuse. Many choose either the “Most of Us” approach taught by the Montana Institute or the methodology proposed by Michael Haines. Both involve developing substance abuse prevention messages to correct incorrect perceived norms of rates of use. The goal of the PSN approach is to bring perceived norms in line with actual norms of alcohol or other substance use. The hope is that by correcting the perceived norm, the rates of underage drinking and binge drinking will decrease. For example, many youth overestimate actual rates of alcohol use on college campuses, assuming that ALL students drink, when in fact, this is not true. A PSN campaign might include messages like “Most Central High School students don’t drink” that are shared on campus posters and through social media. Some of our clients are beginning to demonstrate a correction in the perceived norm and a reduction in youth alcohol use.

Lessons Learned: Community-based programs are most effective when they are grounded in the needs of the community and reflect the “But why here?” That is, the factors in the community that support teen initiation and use and misuse of alcohol or other drugs.

Hot Tip: Think of the “But Why?” as the overarching cause of a social problem such as substance abuse and the “But Why here?” as the local condition helps communities focus on what is really driving a particular issue in their community. Designing the logic model, community intervention, and the evaluation around the “But Why?” and “But Why Here?” helps to focus your work.

Lesson Learned: You can lead a horse to water but you can’t make your clients drink. Coalitions and community collaboratives get stuck along the prevention road in many different ways. For example, some get stuck in the beginning phases of coalition development. Others get stuck after they design their logic model but never move on to implementation. Some talk about, but never develop a sustainability plan. Evaluation is also about program development and implementation- be there for your client as a prevention partner every step of the way.

Rad Resource: Community Anti-Drug Coalitions of America (CADCA) is a great resource for prevention information.

We’re looking forward to October and the Evaluation 2016 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

My name is Gisele Tchamba and I work for a faith-based organization where I lead in evaluation capacity building. I am also one of the ADAMH TIG leaders.  Today I want to share my experience with the implementation of a new treatment model for behavioral health providers in my county.

The Recovery Oriented Systems of Care Model (ROSC) model is a network of formal and informal services developed and mobilized to sustain long-term recovery for individuals and families impacted by severe substance use and mental health disorders. This model was created to replace the Acute Care model that was not very successful at providing long-term recovery outcomes.

Lesson Learned: The ROSC model is the dream treatment for behavioral health field. However, its implementation has been in process for over 10 years without success. I compared the implementation of the recovery-oriented systems of care model to the Transtheoretical model of behavior change and evaluated my county’s efforts to create its own recovery oriented systems of care model.  This model posits that health behavior change involves progress through six stages of change: precontemplation, contemplation, preparation, action, maintenance, and termination. I used this model because of the complex process in the county’s attempt to implement the process. The evaluation revealed that the county was in the second stage of the TTM model (the contemplation stage). But today my county is in the Action stage of the implementation and ready for another evaluation. In reality the ROSC is not a myth, it’s just very difficult to implement.

Hot Tips

  • Each state can create its own ROSC model by forming a task force that can comprise four workgroups (funding, prevention, screening and assessment, and treatment and recovery). These workgroups represent segments of the recovery oriented systems of care model.
  • A successful way to implement the ROSC model in a given community requires commitment from each provider.
  • To facilitate dialogue between providers, there must be funding for liaison. That element is currently missing in my county. I think that must be the reason for the slow progress noted so far.

Lesson Learned: A successful implementation of the recovery oriented systems of care model in this county will lead to the following:

  • Best practice, increase knowledge and facilitate evaluation
  • The county will become a recovery community with sustained long-term recovery outcomes for individuals and families; and
  • There will be better allocation of resources, better community involvement, and honest dialogue among providers.

Rad Resources

The Role of Recovery Support Services in Recovery-Oriented Systems of Care White Paper.

Willam White, an experienced researcher, offers a definition of the ROSC Model.

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings AEA365 community.  My name is Paul St. Roseman, and I am the owner/principal investigator with DataUse Consulting Group.  The intersection of evaluation and professional development has been an interest of mine throughout my 20-year career as an educator and evaluation researcher.  Within my practice, I have observed how participation in evaluations impact how people: (a) view themselves professionally, (b) approach their work, (c) communicate with one another, and (d) discuss the operation of their organization and its impact within the community.

Recently, I have begun to consider what these influences suggest about an evaluation’s overall impact on both learning and leadership development within organizations with whom I work.  To this aim, I have developed an Evaluation Capacity Building Learning and Leadership Conceptual Framework that I use to monitor how learning and leadership development within an organization transition as a collaborative evaluation study is completed.

Connecting evaluation within learning and leadership development allows me to examine the impact of an evaluation study beyond documenting organizational processes and accounting for services.  With the addition of these lenses, it is possible to locate the professional and organizational development that accompanies collaborative evaluation efforts in ways that are personal, contextual, transparent and immediate.  While the indicators used in this framework are far from perfect, they do provide a glimpse into what is possible when multiple theoretical lenses are used to push understanding regarding the experiences and lessons that accompany an evaluation experience.

Lesson Learned: Evaluation’s influence on client development is strengthened when efforts are professionally relevant and provide increased opportunities for participation across an organization.

Lesson Learned: The benefits of evaluation efforts deepen when clients are motivated, emotionally connected, and trusting of the experience.

Lesson Learned: In instances where reflective dialogue occurs between clients the potential for mutual learning increases and leadership capacities shift.

Lesson Learned: When clients develop the capacity to leverage existing and new skills within an evaluation study, confidence and self-determination applying evaluation findings to decision making strengthen.

Lesson Learned: Newly developed evaluation processes and skills can be difficult to sustain and integrate within an organization.  This is the case even when such efforts are valued by stakeholders.  Given this factor, the benefits of evaluation capacity building may be most apparent in multi-year evaluations that incorporate capacity building workshops, coaching, and technical support alongside data collection and analysis efforts.

Roseman

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Julie Slay, Senior Director of Evaluation at Arabella Advisors, a consulting firm that helps philanthropists and investors achieve the greatest good with their resources. Often, we are asked to evaluate a funder’s portfolio or initiative using indicators based on secondary data sets that reflect community-level and individual-level outcomes. Administrative data have potential to help us do that for behavioral health – or does it?

In the world of health care, using administrative claims data is attractive for a number of reasons. First, what are claims data? They are large data sets, held by public and private payors, that contain information about what a health care provider submitted to get paid for their service. Recent claims data, thanks to HIPAA, are in digital form – downloadable data spreadsheets that provide information about each client health encounter. But how accurate are those claims data? Do they really tell you the story of behavioral health care? Probably not.

Lesson Learned: Claims data have a number of constraints that are important to remember.

  1. Claims data reflect what the provider and payor need, not always what happened. Studies continue to compare claims data with service encounter data and find significant discrepancies. Often, claims data underestimate the amount and type of services provided, because providers know their payors’ rules and will not submit claims for a non-reimbursable services.
  2. Data required for reimbursement may not provide you with an accurate picture of quality. Mental health conditions are chronic; over one’s lifetime, there may be shorter, intense periods of using services, and longer periods where fewer services are used. However, we cannot know just from patterns of service use whether the client is receiving high quality services. High quality services might be reflected in a greater frequency of seeing a provider, but it also may indicate poor quality that requires multiple visits to get the care that one needs.
  3. Claims data are often dated. Once a service is provided, the data must be cleaned before it is claimed. Once sent to a payor, it might go back and forth between the payor and provider due to an error, or a denial then an appeal, and after several months, the service is paid for, or not. So in a world where people want real-time data, administrative claims have NO potential for that.

Lesson Learned: Try using clinical data from an electronic health record (EHR) instead of claims data to assess outcomes. Rules related to reimbursement should not prevent providers from entering these service, and data are close to real-time. If you use claims data, do not rely on it as the only measure of an outcome; use other indicators from different data sources to augment your findings.

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Dr. Trena Anastasia, an independent evaluator and principle at QDG Consulting.  In our office spring means grant writing and grant writing means pulling together logic models quickly.  In my practice, communication is essential and having a quick way to synthesize a client’s ideas, thoughts, goals, etc. in a logic model so I can determine how to help them move forward is critical.

My area of expertise is in Suicide Prevention and I have been a member of the ADAMH TIG for almost a decade, serving in a chair/co-chair capacity for three of those years. During that time I have attended multiple workshops on developing logic models, ideas for thinking through them, presenting them, and making them usable. I have found many of those methods useful at informing my process but nothing has changed the way I do them more than assembling them in PowerPoint.

Rad Resource: PowerPoint is much more user friendly and intuitive than most programs designed strictly for modeling. It may not be the best when you have multi-page models to consider, but when synthesizing a project down to a single page required for grant proposals, or to give a picture of the process to a stakeholder it works great. It also allows for:

  • Reviewing on a variety of platforms, and makes for easy sharing and editing by multiple stakeholders.
  • Drawing of arrows, highlighting of the section you are presenting on, even making it in the color scheme of your client’s logo.
  • Setting up and printing on multiple sizes of paper to fit the margins required by a grant, and it can be saved quickly as a PDF and imported into a Word file as an image.

To top it off, it is essentially free because if we have the Office Suite, we have PowerPoint.

I hope this tip is helpful as you work on your next logic model or proposal.

Anastasia The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

I’m Dr. Kathleen Ferreira, Director of Research and Evaluation at the Center for Social Innovation in Needham Massachusetts and one of the chairs of the ADAMH TIG. I’ve spent a number of years providing technical assistance (TA) in theory of change logic model development in systems of care serving youth with behavioral health challenges. A number of TA recipients have one goal in mind: Complete and submit a logic model to the funder and check that required task off their list. However, carefully crafted logic models can serve as an important roadmap to meaningful change and goal attainment for organizations. I’d like to share some insights I’ve gleaned from my work.

Lesson Learned: Organizations must “own” the process. It is not uncommon for organizational leaders to assume that I will develop their logic model and submit to them a completed product. However, it is more important that I serve as a facilitator of its development and that organizations work through the process of developing a shared understanding of who they are (vision, mission, values), their intended service recipients (population), where they need to go (goals and outcomes), and how they will get there (strategies). Interestingly, many TA recipients assume that they are in agreement until they begin to articulate these components.

Lesson Learned: Inclusion is critical to success. Create a logic model team that includes participants at all levels of the organization (or system), including service recipients. A group that is too large can make it difficult to move forward. I recommend a group of 5-10 people on the core logic model team. I also chunk out the process instead of trying to complete it in one session. A one-hour meeting every 1-2 weeks gives team members time to work through components and gather feedback from more stakeholders. Also, do not exclude a “Negative Nellie” from the team. Although the work may be challenging at times, their direct participation creates buy-in, neutralizing their ability to create discord when the logic model is implemented.

Lesson Learned: “The best way out is always through.” This Robert Frost quote is a good mantra for the team. Developing a meaningful logic model takes work, and the process can be frustrating, especially when balancing many different personalities and priorities. Offer encouragement. Acknowledge that this is a difficult process, but remind the team that it will pay dividends.

Rad Resources: For helpful logic model resources and examples, see the work of my former colleagues at the University of South Florida and the Kellogg Foundation’s logic model guide:

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings everyone! – My name is Roger Boothroyd and I’m from the University of South Florida. I have been conducting evaluations on the delivery of mental health services and programs for nearly 30 years, primarily working with government entities. It is my pleasure to introduce the Alcohol Drug Abuse and Mental Health (ADAMH) TIG’s week of contributions to aea365.

I know it is no news to any of you that evaluators often face a range of challenges evaluating behavioral health programs. For example, limited funds to support the evaluations, unrealistic timelines, competing political agendas among stakeholders, understanding the program theory of change and/or data issues to name just a few. This week several ADAMH TIG members are pleased to share with you some of the challenges they have encountered during their evaluation efforts and, more importantly, some of the resources, suggestions, and lessons learned that they have to offer to help minimize those challenges.

Lesson Learned: I’d like to begin by sharing one of my thoughts related to politics and evaluation and how the manner in which we present our findings can impact our clients. Evaluators always have the option of portraying findings as signifying the glass is half full or the glass is half empty. Take the following two statement for example:

  • 80% of the respondents reported that the services they received were effective in meeting their needs.
  • 1 of every 5 respondents reported that the services they received were not effective in meeting their needs.

Despite the fact that both statements are functionally equivalent, I think everyone can guess which one would make a better newspaper headline or require some detailed explanation at a legislative hearing.

Hot Tip: Understand that how you phase your findings has a potentially negative impact on your clients.

In working with government entities, I inform my clients that during our work group meetings and on our conference calls we’ll focus on the glass being half empty, because it is on those individuals who are not benefiting or succeeding where we need to focus our greatest attention. However, recognizing that our evaluation findings are a potential newspaper headline, I also inform my clients that when the final report is written and the public briefing is held the results will be presented as if the glass is half full but that when they hear or read this they should recognize the real important message– that the glass is also half empty and more work needs to be done.

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Lija Greenseid. I am a Senior Evaluator, with Professional Data Analysts, Inc. in Minneapolis, MN. We conduct evaluations of stop-smoking programs. Smokers generally have lower education levels than the general population. Therefore, we want to make sure the materials we develop are understandable to smokers.

Rad Resource: Use a “readability calculator” to check the reading-level of your written materials. I have used this with program registration forms, survey instruments, consent statements, and other materials. Not surprisingly, the first drafts of my materials are often written at a level only grad students (and evaluators) can understand. With a critical eye and a few tweaks I can often rewrite my materials so that they are at an eighth-grade reading level, much more accessible to the people with whom I want to communicate.

A good Readability Calculator can be found here: http://www.editcentral.com/gwt1/EditCentral.html

It provides you with both a reading ease score, and a number of different measures of the US school grade level of the text.

This blog posting is rated at a high-school reading level. Do you agree?

The American Evaluation Association is celebrating Best of aea365 week. The contributions all this week are reposts of great aea365 blogs from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Greetings! I’m Robin Kipke and I work for the Center for Evaluation and Research at the University of California, Davis, which provides evaluation training and technical assistance to the 100+ projects which advocate for tobacco control policies in our state. To help these organizations discern how to effectively harness the power of social media, I’m partnering with the California Youth Advocacy Network to develop a handbook that explains what social media can do and ways to evaluate its use.

There is still a lot of discussion in the field about how to meaningfully evaluate these new media, but here are some ideas I’ve found to be informative.

Hot Tip: So many articles on evaluation focus on the various metrics or latest analytical applications that are available. However, evaluators know that the best measures in the world aren’t of much help without having a clearly-defined social media plan. In the Internet Advertising Bureau’s Social Media Measurement and Intent Guide Emily Dent says: “Launching social media activity, but not having any idea what you want to achieve is a little bit like having a map, but not knowing your destination.” The metrics have got to spring from measurable goals that lay out what you hope to achieve through social media.

Rad Resource: The Nonprofit Social Media Decision Guide has worksheets that can help define your purpose for using social media, identify the target audiences you want to reach, develop SMART objectives, and decide on the best new media channels to use.

Hot Tip: A great piece of advice from The Brandbuilder Blog is that in order to determine whether or not your forays into a particular SM channel are worth the cost and effort, evaluation should be activity-specific rather than medium-specific. For example, what was the return on investment (ROI) of shifting 20% of staff time from traditional educational outreach (developing fact sheets and meeting apartment managers) to generating buzz through Facebook and Twitter? Now you have a benchmark to compare your SM results against conventional ways of building public sentiment.

Rad Resources: I also like the following sites for collections of materials on social media: Mashable, Nonprofit Technology Network (NTEN), Occam’s Razor, Community Anti-Drug Coalitions of America (CADCA), National Center for Media Engagement.

Rad Resource:  This year at the 2011 evaluation conference, a number of sessions focused on social media. I gave a demonstration on Social Media’s Evaluation Power. Kurt Wilson and Stephanie Evergreen spoke about Evaluating Website Usage and Social Media engagement. To find these and other resources on this topic, you can also search the AEA Public eLibrary.

Rad Resources: To end with, I’d like to recommend a thought-provoking article by Matt Owen which makes the case that evaluating social media has to be more than metrics. After all, the very nature of social media is about relating to others….

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello, I’m Linda Cabral, a Senior Project Director from the Center for Health Policy and Research at the University of Massachusetts Medical School.

In an effort to more fully engage people from different cultural backgrounds and their communities in one of our recently-completed qualitative evaluation projects for our State’s Department of Mental Health, we employed the use of cultural brokers as members of our evaluation team. Cultural brokering has been defined as the act of bridging, linking or mediating between groups or persons of differing cultural backgrounds for the purpose of reducing conflict or producing change (National Center for Cultural Competence, 2004). In our case, we were seeking information from people with mental health conditions from specific population groups: Latinos and persons who are deaf or hard of hearing. We brought to our evaluation team cultural brokers who were people with mental health conditions and who were also members of the cultural groups we were interested in. They helped us develop our data collection instruments, led recruitment efforts, and participated in the data collection and data analysis phases.

Lesson Learned: The cultural brokers were able to establish a rapport and level of trust with study participants that would have been impossible to otherwise achieve. This rapport was important not only during the recruitment phase, but also during the data collection itself, thereby improving the quality of the data collected.

Lesson Learned: A barrier often cited with collecting data from non-English speakers is the need for interpreters. By using cultural brokers, participants were able to communicate as they felt most comfortable. Consider the use of cultural brokers when exploring sensitive topics with people from different cultural groups.

Lesson Learned: As the cultural brokers had little to no experience with evaluation work, it was necessary to build in time to educate the cultural brokers on evaluation basics. This helped to make our cultural brokers feel like a fully participating team member.

Rad Resource: The National Center for Cultural Competence (http://nccc.georgetown.edu/) has a host of resources to help programs design, implement, and evaluate culturally- and linguistically-competent service delivery systems.

Rad Resource: For those of you interested in using cultural brokers in the mental health field, the following article might be useful.

Singh NN, McKay JD, and Singh AN. (1999) The need for cultural brokers in mental health services. Journal of Child and Family Studies, 8(1):1-10.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Older posts >>

Archives

To top