Hi, I’m Lisa Melchior, President of The Measurement Group LLC, a consulting firm focused on evaluation of health, behavioral health, and social services for at-risk and vulnerable populations. I’m delighted to share some thoughts about the role of federal performance data and reporting requirements as they pertain to local evaluation. Since the inception of the Government Performance and Results Act (GPRA) in 1993, my colleagues and I have conducted dozens of evaluations that involved complying with these data collection and reporting requirements.

GPRA is U.S. federal law which created a mechanism for government agencies to engage in strategic planning, performance planning, and performance reporting. In 2010, the GPRA Modernization Act was updated, emphasizing priority- setting and cross-organizational collaboration. It also increased its focus on using goals and measures to improve outcomes of federally funded programs.

Many of the programs my colleagues and I evaluate are funded by the US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Substance Abuse Treatment (CSAT), which is required to report to Congress on progress on its designated GPRA measures. SAMHSA/CSAT requires organizations that provide direct client services funded by CSAT discretionary grants to implement the Client Outcome Tool. Although it can vary depending on the grant program, grantees are generally required to collect this information from individuals at program intake, at 6-months post-intake, and at program completion or discharge. The GPRA tool includes questions regarding the participant’s housing situation, physical and mental health, substance use, sexual activity and criminal justice system involvement. While this standardizes data collection for all grantees within a funded initiative (and to a broader extent across all the funding agency’s discretionary grantees), it also has a substantial impact on evaluation design, implementation, and analysis at the local level.

Hot Tip:

GPRA data can be used for program monitoring and evaluation at the local level. Grantees and their evaluators can access summary reports of selected GPRA data (SAMHSA grantees use a system called SPARS). Grantees also have the capability to download their raw data for local use.

Cool Tricks:

Try to integrate required performance measurement data collection protocols with additional measures used for project-specific evaluation. Aim to minimize repetition and redundancy. Pay close attention to the flow of when and how questions are asked of program participants.

Lessons Learned:

While GPRA data is useful for local evaluation, it is often not sufficient for this purpose. For example, self-reported substance use is only measured in the past 30 days. Mental health symptoms are also measured by responses to single-item self-report measures of past 30-day symptoms. If it is important to document lifetime substance use patterns, or to know about actual DSM-based mental health conditions, it is necessary to supplement the GPRA measures.

Rad Resources:

General information: https://en.wikipedia.org/wiki/Government_Performance_and_Results_Act

https://obamawhitehouse.archives.gov/omb/mgmt-gpra/index-gpra

Information about SAMHSA’s GPRA Measurement Tools: https://www.samhsa.gov/grants/gpra-measurement-tools

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Government Performance and Results Act

Greetings! We’re Kathleen Ferreira and Roger Boothroyd, Co-Chairs of AEA’s Behavioral Health Topical Interest Group (TIG). The Behavioral Health TIG is pleased to sponsor this week’s blogs for AEA365. We have chosen to draw upon our members’ experience, expertise, and thoughts as it applies to GPRA, the Government Performance and Results Act. As you will read more throughout the week, the Act was originally enacted in 1993 and updated in 2010 and is designed to improve government performance management. As a result of the Act, members conducting evaluations of grants funded by the Substance Abuse and Mental Health Service Administration are required to report data on the GPRA measures. At the Evaluation 2017 conference, we had a thought-provoking discussion during our Behavioral Health TIG’s Business Meeting on issues such as how federal performance monitoring reporting requirements can be used to enhance local program impact and how we can maximize the use of the large volume of data collected at the local level in making day-to-day decisions in improving local service delivery. We thought that this would be an interesting topic for this week’s blog. We hope you agree.

If we go to our dictionary (does anyone have a real dictionary anymore?) requirements is defined as “a thing that is compulsory; a necessary condition.” As many are aware, GPRA has mandated federal reporting requirements. That doesn’t sound appealing, does it? In particular, it sounds like more work! Yet, as evaluators, this creates a real opportunity—one in which we can work closely with our program staff to determine how data that we must report can best meet our local program staffs’ needs. It may not be readily apparent to us at first, it may take us some time to figure it out, but in the end, if we are thoughtful and persistent, we’ll find the answer. Why? Because we’re evaluators – it’s our job! It’s what we do.

This week, readers will learn:

• the definition of GPRA and how GPRA data can be used for program monitoring and evaluation at the local level.
• why it is critical for us to continue to collect and report GPRA data, even during governance challenges and uncertain times.
• how to create data reporting sheets in order to reduce burden on program staff.
• (taking this idea of data reporting sheets a step further) how to maximize the use of GPRA data by creating program fact sheets.
• lessons learned during a 5-year evaluation of a Screening, Brief Intervention, and Referral to Treatment (SBIRT) program in which behavioral health services were integrated into a medical setting.

We hope you enjoy the week!

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Government Performance and Results Act

Hi, my name is Jessica Byrnes and I’m a member of the AEA marketing team. With Evaluation 2018 in full swing, I’ve been keeping up with the conference on Twitter using #Eval18.

Twitter is a great tool for collaboration, expanding your professional network, and staying on top of the latest conversations in evaluation – whether you’re attending Evaluation 2018 in-person or virtually! Check out some of the ways #Eval18 attendees are utilizing Twitter during the conference.

Hot Tip: Tweet from the Presentations
Sharing quotes, insights, and tidbits you especially enjoyed from presentations is a popular way to utilize Twitter during a conference. If the presenter shows an interesting stat or slide, take a photo and share it with your followers! Pro tip: If you’re taking pictures of speakers, get their Twitter handle and tag them in the photo.

INCREDIBLE plenary session at #Eval18 today with Brooke Haycock. #Performance as #Evaluation. A powerful look at complex stories of #education in #Detroit#sketchnotes #Sketchnoting #qualitativedata #quantitativedata #mande pic.twitter.com/e4ZhqtAUWa

— Sarah Ellison (@sgellisonBenin) November 1, 2018

Good recommendations on how to incorporate CRE in the classroom. 'Don't silo CRE.' Thanks for an informative session (Teaching Culturally Responsive Evaluation: Theory, Practice, and Reflections) Drs. Ashford, Davis, Garza, Gonzalex, Lemmons, Lui, & Scott. #eval18 pic.twitter.com/x1iQWIdKkS

— Kelly (@KellyRobEval) November 1, 2018

Fave quote today: “She took us to task for dressing up data with nowhere to go.” It’s important to create the right tools based first on the right issues. Loving the humor in Ian Watt’s Designing Human-Centered Interactive #DataViz session this morning! #eval18 pic.twitter.com/pAWJGX5vDo

— Vantage Evaluation (@vantage_eval) November 1, 2018

Hot Tip: Ask Questions
Sharing questions is a great way to engage with presenters, connect with others, and find out information. Contribute to the conversation yourself by posing a question on Twitter using the hashtag #Eval18.

My (unasked) question for @InnoNet_Eval from their #Eval18 presentation: have you looked at critical mass (where the balance of power is tipped) in your research? pic.twitter.com/APDFnmcvPR

— Nick Petten (@petten) November 1, 2018

In this morning outcome's harvesting session, there was a question about credibility of the method. Best response: "Well, aren't the voices of the people credible?" #Eval18

— Anne Laesecke (@AnneLaesecke) November 1, 2018

As @danawanzer pointed out, these models can be applied to #evaluation too. What are the implications for #speakingtruthtopower of each model? Whose power? #eval18 https://t.co/eKQPxR0r6Z

— BetterEvaluation (@BetterEval) October 31, 2018

Hot Tip: Let Other Attendees (and AEA!) Know What You’re Enjoying
Events like Evaluation 2018 have a lot to offer. Why not reflect on your favorite parts of the conference?

You know it’s a great start to #Eval18 when there are six concurrent sessions I want to attend in the first time slot!!

— Lyssa Wilson Becho (@Ly_Wilson) October 31, 2018

What a great idea! Mindfulness, quiet, contemplation together = priming the learning brain #eval18 https://t.co/Cf76QywAly

— Cense (@CenseLtd) November 1, 2018

Food trucks (inside) and music!!!!!!! Best @aeaweb conference yet!!! You are killing it president @lkg4! #Eval18 pic.twitter.com/t90l1yJ4fu

— Ayesha Boyce (@AyeshaBoyce) November 1, 2018

I hope you enjoy the last day of the conference. Feel free to use these tips at other evaluation events you attend!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello dear readers! Sheila B Robinson, aea365 Lead Curator, sometimes Saturday contributor AND Conference Week blogger here (again!) with a few photos of the fabulous artwork here at Evaluation 2018.  No evaluation tips to share today…because sometimes we just have to step back and find beauty and joy in things outside of our work!

These are just a few of the vistas we are being treated to as we traverse the halls and walkways between the Hilton Cleveland Downtown, and the Huntington Convention Center of Cleveland.

 

I regret that I didn’t have the time to stop and find out the proper names of these installations or the artists names. If you happen to see this information  any where, would you please send it to me so that I may update this post? Thank you!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation 2018

Hello faithful readers! Sheila B Robinson, aea365 Lead Curator, sometimes Saturday contributor AND Conference Week blogger here with a quick update on the goings-on in Cleveland!

Our series of two-day workshops got off to a great start Monday and continued on Tuesday, while a number of one-day workshops also took place Tuesday. Wednesday will feature a slew of half day workshops in the morning and afternoon and those will lead us right into our opening plenary!

Many of my friends are here taking the pre-conference workshops and it’s wonderful to see the mix of veteran evaluators continuing the learning journey right alongside first time conference attendees.  I’m hearing great feedback about the workshops people are participating in and these conversations spill out into the break areas as people gather for mid-morning coffee and tea, and again in the elevators and sidewalks on the way back to hotel rooms after the day comes to a close. Even the late night conversations over adult beverage seem to gravitate back to evaluation and our collective work and learning. It’s incredibly heartening to see so many people engaged in talking about work that inspires them and that they clearly enjoy.

Have you made it to Cleveland yet? Perhaps you’re coming in for the start of the concurrent sessions on Wednesday afternoon. This year, you’ll be greeted by a pretty fancy registration station and information booth, along with the usual plethora of signs that help us find our way in unfamiliar territory. Many thanks to the AEA staff and volunteers who make this happen!

Lesson Learned: And did you know the while many of us were having fun think-pair-sharing, scratching out ideas on post-it notes, writing on chart paper, numbering off, getting in groups, moving around, drawing pictures, practicing new skills…and otherwise having a grand old time learning and adding to our toolboxes, our AEA Board of Directors, presidents (current, past, and elect), executive director, and operations coordinator were working intently and intensely on key association issues? Let’s give them some love and thanks too, for all they do to support us. Offer up a handshake, a high five or a hug to these hardworking folks!

 

 

 

 

 

 

 

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation 2018

Hey! I’m Sara Vaca, independent evaluator and part of the AEA365 team. I’m not in Cleveland (sad face) this week (as I just came back for the European Evaluation Conference (EES) in Greece), but I wish I was, as I love evaluation conferences.

I am not an expert compared to so many admired evaluators who have been going to AEA conferences since 1990’s, but from my experience with 3 AEAs, 2 EES and 1 SAMEA (South African Evaluation M&E Association), here are things I learned not to do anymore.

Cool Tips: Some things NOT to do! (;-P)

1. “Don’t talk to other people you don’t know”

We are all shy to start a conversation with someone you’ve never met: you think you are not that interesting, maybe the person is not that interesting and that you don’t have much in common. But I’ve learned that many, many times the person is interesting, she or he finds you interesting too, plus you have plenty of things in common that occasionally even bring other unexpected developments.

2. “Improvise your program: Don’t look at the program in advance, and 5 minutes before sessions, just choose one randomly”.

Well, chances are you end up missing things that really interested you!

3. “Study the program in detail and decide in advance what you will be doing at each moment”. Well, some improvisation is always welcome, as unexpected meetings, acquaintances and events happen at some point of the conference that are worth not missing too.

 

4. “Don’t go out in the evenings or by night”

To be honest, this one has never happened to me (maybe is my Spanish blood), but I know friends who want to keep it calm and they go back to the hotel as the sessions end. Well, each one has to do what is best for them, of course, but mingling after conference hours is a great way to know colleagues in a different circumstances, and of course, of having fun!

5. “Stay up all night”

Well, let’s not get crazy either, and if you want to enjoy the sessions and have the stamina to go through it all, you also need some reasonable resting time, no matter how fun the night might seem.

6. “Pretend to go to every single thing you think you should go”

Even if you study the program, allow certain flexibility and try to be reasonable and keep a balance, don’t feel frustrated if too many things happen or simply too many good sessions overlap and you miss plenty that you would have liked to attend.

Please take all these as personal ideas or suggestions, hoping they resonate with you and help you have a better experience if this is your first time. And please feel free to share more “Not to do things” in the comments, or if some of these Don’ts on the contrary work very well for you! :-)…

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

conference

Hi everyone! I’m Leigh M. Tolley, Assistant Professor, Secondary Education at the University of Louisiana at Lafayette, Past Chair of the PreK-12 Educational Evaluation Topical Interest Group (TIG), and member of AEA’s TIG Council Group on Conference Quality and Logistics. This year will be my ninth AEA conference, and I foresee many more in my future!

A big part of why I love coming to the AEA conference each year is catching up with old friends and colleagues and making new friends and connections. I have always found AEA members in all stages of their careers to be friendly, welcoming, and willing to chat with and learn from others. TIGs are an integral part of why I feel AEA is my “professional home.”

Lessons Learned: TIGs help you grow professionally. At my second AEA conference in 2011, I decided to check out a few TIG business meetings and find ways to possibly get involved as a doctoral student. While at the PreK-12 TIG’s business meeting, they asked for volunteers to be Members-at-Large, and I raised my hand. I have been a Member-at-Large or part of the Program Chair and Chair teams—we have elect, current, and past positions for each in our TIG—ever since. Through these roles, I have been able to develop an excellent network, gain experience in leadership, and better understand the proposal submission and review process, which all help me tremendously as a tenure-track academic and evaluation practitioner.

Hot Tip: Business meetings are open to all AEA members, not just current TIG leaders. Don’t let the name fool you: these are non-stuffy, collegial sessions where like-minded folks can come together to share ideas, learn more about what works in their areas of expertise, and discover ways to engage with AEA.

Hot Tip: Take advantage of myriad opportunities to interact with TIG members while at Evaluation 2018! There will be TIG Tables set up during the Poster Exhibit and Meet the Authors session on Wednesday, and TIG Receptions for groups of TIGs and TIG Business Meetings will be held on Thursday evening. Information will be posted by the Registration Desk during the conference, where you can also pick up badge ribbons to represent your TIGs.

Hot Tip: Members of AEA may join up to five TIGs as part of membership. To join, view, or change your TIGs, log on to the AEA main site, then select Preferences and Update TIG Selections from the far right of the top menu.

Rad Resource: Check out the many different TIGs AEA has to offer on the Topical Interest Groups page of the AEA website. There are almost 60 TIGs, including those focused on research methods, special groups, and the theory, practice, and teaching of evaluation itself. Two new TIGs to check out this year are the Graduate Education Diversity Internship (GEDI) TIG, open to GEDI alumni as well as all AEA members interested in the program; and the Health Professions Education Evaluation and Research TIG.

See you soon in Cleveland!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation 2018

Greetings AEA members, conference-goers, virtual conference goers and AEA365 fans! I’m Sheila B Robinson, your very excited AEA365 Lead Curator and sometimes Saturday contributor. I’m getting ready to spend nearly a week in Cleveland, OH at Evaluation 2018. The annual AEA conference is one of my favorite weeks of the year – time for me to learn more about my field, attend exciting presentations and see fabulous presenters, meet new people, and reconnect with old friends.

To help you get ready for the conference experience, I’ve bundled some handy resources right here:

Rad Resources: 

1. AEA365 LAWG posts: We had two weeks of LAWG posts this year – one in July and one in September – with the Local Area Working Group (LAWG) working hard to let us know what to do and where to go in Cleveland.
2. Tips for first timers (and veteran conference goers!): I contributed an AEA365 blog post on this,  Donna Podems wrote a great article on LinkedIn, Navigating AEA 2018, and Dana Wanzer started this conversation during #EvalTwitter chat a few nights back.
3. Don’t for get to explore and complete your profile in the Evaluation 2018 Conference App and all the great features it has to offer.
4. Look for special events, such as the Plenary Presentations,  Design Loft, Topical Interest Group (TIG) Business Meetings, AEA 2018 Awards Luncheon, and other Social Events.
5. And of course, use the Evaluation 2018 Conference Website for all your conference information needs.

Don’t forget to take a break from silly cat videos and share your conference learning and experiences on social media with the hashtag #Eval18.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings! I am Sarah Fleming, MSW, Grants & Evaluation Coordinator at Casa de Esperanza. I have been fortunate to work with amazing evaluators such as Dr. Josie Serrata and Dr. Rebecca Rodriguez at our National Latin@ Research Center on Family and Social Change and experienced Latin@ practitioners like Ivette Izea-Martinez in our community-based domestic violence prevention initiatives. As a white student evaluator at a culturally-specific organization, it has been important for me to learn from these mentors about listening to Latin@ community members and shaping evaluation around community voices.

For my Master’s program, I worked with Casa de Esperanza’s Fuerza Unida Amig@s Community Engagement Initiative to conduct an evaluation of the Mother Daughter Retreat, a component of a culturally-specific, family-based teen dating violence (TDV) prevention program. The Mother Daughter Retreat grew out of community listening sessions which found that Latin@ youth and parents desired for a space to talk about healthy relationships. In addition, emerging research suggests that family connectedness, cohesion, and communication are important protective factors against TDV for Latin@ youth. The retreat provides an opportunity for Latina girls and their mothers to spend a day relaxing at a camp facility with the goal of strengthening mother-daughter relationships and learning about TDV and healthy relationships.

After working closely with staff to learn about retreat objectives and participant experiences, we conducted a bilingual retrospective outcome evaluation. Two months after the retreat, both mothers and daughters perceived statistically significant improvement in their mother-daughter relationship, mother-daughter communication, and knowledge of healthy relationships and TDV. These preliminary findings point to the potential effectiveness of family-based strategies to prevent teen dating violence among Latin@ youth.

Lessons Learned:

• Conducting an evaluation as a white woman in a culturally-specific organization has challenged me to become aware of my own privileged identity and the power that it holds when working with communities of color. As Beverly Peters recently wrote in AEA365, evaluators, especially those from privileged groups, hold immense power. It was important for me to listen to Latin@ colleagues and community members in order to gain a nuanced perspective without letting my biases get in the way.
• Building relationships is essential to successful evaluation, especially in culturally-specific communities. Developing relationships with Casa de Esperanza staff and community members helped me to better understand the programs and be a more effective evaluator. It is also essential to nurture organizational relationships with community. Because of the strong relationships that Casa de Esperanza has built over time with the community members who attended the retreat, we had a fantastic response rate in our evaluation.

Rad Resource:

• Casa de Esperanza’s Research Center has developed the Building Evidence Toolkit, a bilingual evaluation guide for culturally-specific, community-based organizations. The toolkit contains resources for organizations with different levels of evaluation capacity: those getting started, those looking to enhance their evaluation practices, and those looking for a deeper dive in to culturally-specific evaluation.

 

 

 

 

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi, I’m Barbara Klugman. I offer strategy support and conduct evaluations with social justice funders, NGOs, networks and leadership training institutions in South Africa and internationally.  This blog is about the value of establishing an Evaluation Advisory Group when your task requires more skills that you have!

I’m currently the ‘learning and assessment partner’ of Tekano an organisation which runs the Atlantic Fellows Programme for Health Equity South Africa.  I have found this experience challenging, not least because it’s my first experience in long term developmental evaluation, because I have multiple accountabilities – to funder, Tekano board, and staff, as well as to the public good given the country’s desperate need for a strong fellowship programme on social determinants of health; and because my primary expertise is in strategizing and evaluating social justice advocacy initiatives rather than leadership development. Finally, I am a white older woman playing the role of ‘critical friend’ to an initiative whose staff and fellows are mostly black and young and whose advocacy experience is taking place in a vastly different historical moment from when I built such experience.

With these constraints in mind, when I developed my terms of reference, I motivated a budget to allow me to establish an Advisory Group. I sought members to fill three gaps in my expertise (that I was aware of) – an organizational psychologist whose expertise traverses organisational behaviour and monitoring and evaluation; a fellowship evaluation expert; and an expert in community-building and evaluation in the South African context.

I engaged the group to help me review the vast quantity of data and analysis I had produced in order to hone in on priorities for my final 18 months in this role, and to help me clarify how to manage my layers of accountability. It was fantastic!! The group brought totally independent eyes to bear on my questions. They helped me distinguish the nature and hierarchy of my various accountabilities. They confirmed my own conclusions on achievements and challenges thus far while hauling me out of the data, insights and relationships I was buried in.  This allowed me to hone in on powerful questions and ways of asking them, for the second half of my tenure in this role.

I loved every minute of the experience even while it was exhausting and challenging. I would encourage you to build in space for this kind of independent support if you’re conducting developmental evaluations.

Hot tips: You have to be a fairly confident person to open yourself up to criticism. Choose people who are ethical and don’t have oversized egos!

Lesson Learned:

• Building a formal advisory group into evaluation budget allows you to identify exactly
• what skills might complement your own, and to compensate people appropriately  for their time
• My huge thanks to Drs. Jane Reisman, Mark Abrahams and Suki Goodman for their efforts on my advisory group.

advisory group · developmental evaluation