Hi there, fellow evaluators! We are Carolyn Fisher, Martina Todaro, and Leah Zallman, of the Institute for Community Health. ICH is a nonprofit consulting organization that specializes in participatory evaluation, applied research, and strategic planning. We help health care systems, governmental agencies, and community-based organizations improve services and create meaningful impact. In a two-part blog entry, we’re going to tell you about a survey design problem we encountered that was unexpectedly tricky.
Once upon a time, we were creating a survey to help a client understand if their program was reaching vulnerable members of low income communities.
At first, we assumed we would define “low-income” as “below the federal poverty level (FPL)”. We thought we’d ask:
What is your household income? ______________
To determine whether a household falls above or below the FPL, which varies based on household size, we’d also need to ask:
How many people in your household? ______
However, we found a number of problems with this set of questions.
- Household income may be difficult to calculate for some households. Do you know the income of everybody you live with? Do you share all expenses? Do you count the income of your adult child? Do you count the income of someone who only lives there part of the year?
- Household size is also a tricky thing to ask for some households! In particular, people may not know how to calculate their household size if they have members who do not live there permanently, or with people who contribute to the household’s income but do not live there. This is most common among the economically vulnerable households we expected to be identifying.
- Poverty and vulnerability are relative to the cost of living. A household at the FPL is better off in rural Alabama than in New York City, for example.
- In our experience, many respondents skip survey questions about income. This could be for the reasons above, but also because of taboos about money and financial vulnerability.
- Finally, we didn’t need this much information!
Hot Tip: In survey research, limit the questions you’re asking to what you really need to know. Here, we only needed to know whether a respondent was vulnerable due to their low income. This is fundamentally a Yes/No question.
Our next idea was to use a proxy measure for low income, such as:
Do you get ANY of the following benefits? SNAP/Food stamps, WIC, SSI, SSDI, TANF, Housing Assistance, Medicaid
- Yes
- No
- Unsure
- I don’t want to answer
This was a less-problematic question than our first attempt, because it was easier for the respondents to calculate. However, it was only our second stop on a journey that wasn’t over yet…
To be continued!
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.