AEA365 | A Tip-a-Day by and for Evaluators

Search

Hello, we are Sheila Matano and Dani O’Neill from Carson Research Consulting (CRC). We at CRC are currently in the process of conducting a survey on relocation and housing stability. Although surveys are an incredibly useful method for gathering quantitative information from large populations, response rates vary widely. In order to promote our project, interact with the community and hopefully increase our response overall rate, CRC conducted a Community Survey Day for residents in addition to providing the survey online and mailing out paper surveys.

Lesson Learned: Planning and preparation.
Planning is essential! Staffers started planning for community survey day about two months in advance.  We had meetings to discuss logistics and created a master task list to make sure everything we needed would be ready for the day of the event. Google Drive provided a way to easily edit and update important documents that could be accessed by all team members.

Lesson Learned: Know your target audience!
Using background information on the target audience, we attempted to tailor the event to the resident’s needs. We found a central location that was easily accessible by public transportation and also offered free parking. In addition, we gave residents the option of taking the survey online, completing a paper survey or taking the survey over the phone. We wanted to make sure that respondents could choose an option that was most comfortable for them.

Lesson Learned: Advertise!
We advertised the event in three ways:

  • We sent a letter to our targeted residents informing them of the event
  • We created event flyers that were distributed to residents
  • We also made phone calls to residents a few days before the event

Hot Tip: The Day of the event
Community Survey Day was hosted on a Saturday to give residents who work during the week the opportunity to attend, and we also offered breakfast and lunch to all attendees.  Staff interviewers were available to do survey interviews with residents from 9am to 5pm. The interviewers read questions aloud to the residents and inputted their responses directly into SurveyMonkey via laptops; residents did not use computers themselves.

Hot Tip: Having the survey online made it readily accessible to interviewers and made the process of completing surveys faster and easier than using paper surveys (although we did have paper surveys on hand in case technology failed us!).

Outcome: Overall, Community Survey Day was a success. We had the opportunity to directly interact with residents and get valuable feedback about our survey. Residents appreciated that they could come in and talk with us directly, as well as offer their suggestions about our project.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

word bubble with lego manHi folks! I’m Jill Scheibler, a community psychologist and Senior Research Analyst at Carson Research Consulting, a women-led firm whose mission is to help clients thrive by using data to measure impact, communicate, and fundraise. We’re passionate about storytelling with data to make a difference.

At CRC I’m the “word nerd”, implementing our qualitative projects. Like many evaluators, I’ve had to translate academically-honed skills to the often faster-paced world of evaluation. A recent project for a county health department’s substance abuse initiative provides an example of how I tailor qualitative methods to meet clients’ needs.

Hot Tips

Allot ample time for clarifying goals. As with all good research, methods choices flow from the question at hand. In this case, our client wanted to understand the impact of substance abuse on their county, and new resources to be tapped. Like many clients, they lacked research savvy, and thought they required services exceeding their budget and available time. We gradually learned they had access to lots of quantitative data and support from the state to help interpret it. They were missing community stakeholder feedback. So, we provided a qualitative needs assessment component.

Build in more meetings than you think you’ll need, and bring checklists. Be prepared to leave meetings thinking you have all needed answers and learning afterwards that you’ve been (well-meaningly) misinformed! (Quantitative sidebar example: after building a data dashboard for another client in Excel2013, based on their word, we learned they had Excel2007. A costly reminder to always ask more questions!)

Choose tool(s) carefully to maximize usefulness. I generally opt for interviews where probes can offset “one-shot” data collection situations. Here, I instead designed a qualitative survey, using mostly open-ended questions, for efficient gathering of perspectives. The client collected surveys themselves, disseminating hard copies and a SurveyMonkey.com link, and accessed a targeted sample from within a community coalition.

Familiar guidelines for interview and survey design apply to qualitative surveys, but I advise keeping questions very focused and surveys as short as possible to mitigate higher skip rates with qualitative surveys.

Cool Trick

You may think your reporting options are limited compared to quantitative results. Not so! Instead of writing text-heavy reports that eat up valuable time, and folks are disinclined to read (#TLDR), consider telling “data stories” using bullet points and visualizations. This client received a two-pager for internal, local stakeholder, and state use. I’ll also provide an in-depth explanation of results and action steps in a webinar.

Rad resources

Jansen’s “The Logic of Qualitative Survey Research and its Position in the Field of Social Research Methods.”

Great tips on qualitative surveys from Nielsen Norman.

Awesome tips from CRC colleagues for larger community surveys.

Achievable qual visualization ideas from Ann Emery.

Some tools for qual analysis and visualization from Tech for Change.

I genuinely enjoy working creatively with clients, because it makes evident how suited qualitative methods for linking research to action. I’d love to hear how others do this work, please get in touch!

image of Jill Scheibler

The American Evaluation Association is celebrating Community Psychology TIG Week with our colleagues in the CP AEA Topical Interest Group. The contributions all this week to aea365 come from our CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Taj Carson and I’m the CEO of CRC, where we help organizations to tell their stories using data and data visualization. Last year, the Data Visualization & Reporting TIG hosted an Ignite session at the annual conference. I knew our work in mapping, including the development of the Baltimore DataMind, would be a great way to introduce people to the beauty and wonder of maps. The Ignite format involves a five minute presentation, using only 20 slides. That means I had better get to the point quickly.

Lessons Learned: Trying to be funny (haha) makes you feel funny (strange). I rehearsed my presentation and co-wrote it with two colleagues, Sheila Matano and Jill Scheibler. We laughed so hard making this presentation. But the whole time I kept saying “What if WE think it’s funny, but no one else does? What if other people don’t get it?” Trying to do a dynamic presentation in a new format is scary. Lucky for me the other evaluators in the room liked it as much as we did.

Hot Tips: Practice like your life depends on it. I practiced my Ignite presentation so many times that I had it unintentionally (but not robotically) memorized.

  • Figure out what you want to say, what’s your message? For me it was to convey our enthusiasm and excitement about maps and mapping data and to show others how beautiful that can be, while making them laugh at the same time. We wanted to be funny and irreverent because that’s how we roll here at CRC, so it was a good fit for us, and entertaining for the audience.
  • No more than a handful of words per slide! Words are over-rated. Seriously, don’t read your slides. Doing this in a regular presentation is bad enough, you will be shunned if you do this (or have giant tables full of tiny numbers) in an Ignite presentation.
  • Use the opportunity to show, not tell, what you want to say. We showed good maps and bad maps. We mocked the bad maps and pointed out why the good maps were so easy to understand.

Carson 1

 

Carson 2

  • Don’t be afraid to hack the format. Layer multiple photos onto one slide. For me it meant that the timing had to be down to within 2-3 seconds as we went from image to image, but it was still only 20 slides!

Rad Resources:

  • Then read Jon Udell’s thoughts on how to actually practice the presentation.

The American Evaluation Association is celebrating p2i Week with AEA members who have used our Potent Presentations Initiative. The contributions all this week to aea365 come from members who have used p2i strategies in their presentations. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Archives

To top