Helping your staff ask smarter questions about their data with Paul Collier

Hello AEA365!  I’m Paul Collier. Over the last two years I worked as the Data and Evaluation Manager at the San Francisco Child Abuse Prevention Center (SFCAPC), a mid-size nonprofit focused on ending child abuse in San Francisco. In my time there and as a freelancer, I can’t count the number of times I’ve fielded questions from staff about data their organization has collected. They often go something like this…

How frustrating! But as someone serving as an internal evaluator or data analyst at an organization, I have to remind my staff questions like these are my friend. When my staff asked me questions about their data, I knew they’re engaged and interested in using it. But I often found the first questions they asked weren’t the questions that would really help them make decisions or improve their programs. This post is about helping your staff think critically and ask smarter questions about their data.

Hot Tip: Focus on highly strategic questions

Questions that can be answered with existing data come in all shapes and sizes. I like to consider first whether the results may help the organization improve or refine our programs. For example, questions testing the cause-and-effect relationships in our logic model or assumptions in our theory of change can and should inform programming. A second aspect of a strategic question is whether our team has expectations for the result. I often realized that our staff didn’t have expectations around average improvement or effect size, so I would find a few studies using comparable assessments and interventions to identify some benchmarks. Perhaps the most useful aspect of a strategic question is whether our staff can take action based on the results. I found that if my staff can’t envision how the results might actually be used, its wiser to help them think through this before spending my time (and theirs) analyzing the data.

Cool Trick: Plan for Analysis.

To be more strategic about the analysis questions I focused on, I built time between the request for analysis and doing the work. An initial conversation with the program manager or staff to learn more about the context of a question usually helped me refine it to be more specific and actionable. I found that batching analysis for a certain time in the year was also a useful planning approach that protected my time. I preferred to have this ‘analysis period’ in the winter, because my organization set its budget in the spring. This way, any changes to programming that resulted from the process could be planned for in the following year’s budget.

Rad Resources:

As you can tell, I think helping staff ask smarter questions is one of the most valuable things I do as an internal evaluator. For more reading on this topic, check out:

• Michael Hyatt’s Blog on Asking More Powerful Questions: Michael Hyatt is a business author who provides some clear and easy to understand advice to aspiring leaders on asking questions.
• Peter Block’s book, Flawless Consulting: Block’s Flawless Consulting provides many helpful suggestions for structuring analysis processes so they influence action. There are also several great chapters about overcoming resistance in clients, which I’ve found highly relevant for dealing with inevitable resistance in results within my team.
• Rodger Peng, Ph.D.’s E-Book, The Art of Data Science: Peng illustrates what a practical data analysis approach looks like, framing each step as a process of setting expectations and understanding why results did or did not meet those expectations.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.