AEA365 | A Tip-a-Day by and for Evaluators

TAG | data collection tool

Hello! We are Laura Beals (program evaluator) and Noah Schectman (database administrator) from Jewish Family and Children’s Service (JF&CS), located in Boston, MA. At JF&CS we use a cloud-based case management system to facilitate data collection about our clients and services. We are part of the internal evaluation department at JF&CS (i.e., he is not part of IT) and so we work closely together to ensure that we are collecting data in a way that allows us to complete our analyses in the most efficient manner possible.

Lessons Learned: Though our system has built-in reporting tools, we often download data for more complex analysis in another tool, such as Access or SPSS. In addition, though the data collection tools are designed as easy-to-complete forms in the system, we do have to bulk upload data regularly.

Many case management/performance management systems allow for back-end customization of the data collection tools—you may have the ability to do so in-house (as we do) or you may have to work with a third-party developer. Regardless, as an evaluator, if you are working with an online performance management system, you should ask yourself: “What does the data need to look like when it is downloaded? When it is uploaded?” In general, we first think about how the data will be used, then design the data architecture to match.

Hot Tips: When designing new data collection tools in our database, we ask several key questions about how the data should be formatted on the back-end, including:

–       What are the unique identifiers for each case that will need to be downloaded with or uploaded to the database?

–       Should the data be arranged so that a case is on a row or each assessment is on a row?

–       For each variable, are the variable labels or the numerical values used?

–       How are multiple response variables formatted? As dummy variables?

–       If names are used, how will they be formatted? What about addresses? What about dates?

Even when we think we have it figured out, we always enter fake assessments for fake clients in the system, through the online form and through a bulk upload, and then download the data. We then review the resulting import/download and triple-check that the data is formatted in the manner we expect. We find doing the work to prepare the system ahead of time saves us a lot of data formatting and manipulation down the road!

2014-02-03 AEA365 Multiple Response_2

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Hi, my name is Pei-Pei Lei. I’m a survey research analyst in the Center for Health Policy and Research’s Office of Survey Research at the University of Massachusetts Medical School.

As colleagues Michelle Landry and Judy Savageau pointed out in their earlier AEA365 blog on Project Management, Microsoft Office provides some options for no/low cost project management tools. I’d like to share our experience using InfoPath, an MS Office solution for data collection purposes, and how it can also fulfill many project management needs.

InfoPath helps users to collect data electronically from end to end. You can design a form template with the needed data elements, and have the collected data automatically submitted into a designated database. In our survey research work, we also use InfoPath for project management; i.e., we collect employee time sheet data (see below). After employees fill out their time sheets and click “Submit” in InfoPath, the data is automatically submitted to a linked Access database.

Lei 1

InfoPath can be customized to monitor project status, track budget/expenses, manage contracts and related information, assist data collection, and submit an application (see below for a sample template).

Lei 2Lei 3Lei 4

The advantages of InfoPath include:

  • Easy to create – You can create an InfoPath form template from scratch, or convert an existing Excel or Word form into an InfoPath form template.
  • Flexible in form functions – You can embed many useful functions in the template (e.g., data validations, range limits, conditional formats for certain work flow, automatic time stamps).
  • Convenient to distribute form templates and collect data – You can distribute form templates to and receive data from multiple recipients using MS Outlook. With access to a shared drive, users can save form templates to their computers and submit data from there.
  • Allows different forms for the same project to be collected simultaneously – Multiple InfoPath templates can be linked to one database, allowing users to view/complete different forms specific to their roles, and all submit into the same database.
  • It’s free – if you have these versions of MS Office.

Lessons learned: InfoPath can be very useful in project management processes; it is easy to build and distribute for collecting and submitting data into a database (such as SQL, Access, or SharePoint). You will need InfoPath Designer to create the form template. Users will need InfoPath Filler to enter data unless the database is deployed on SharePoint, which allows users to fill and submit data in a webpage from a computer or mobile device.

Rad Resources: Introduction to Microsoft InfoPath 2010 and Project plan template

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, I am Kerry Bruce, Director of Results and Measurement at Pact.  I am part of Pact’s central technical team that provides monitoring and evaluation support to more than 20 country offices and more than 70 projects around the world.  In 2012 we started to roll out the use of mobile technology in our programs. We have begun integrating mobile technology into our programs by using mobile phones for baseline and endline data collection.

Hot tips:

  • Bruce Mobile phones 1Mobile technology has advanced significantly since the last time you likely considered using it and now is the time to invest in learning about this technology.  Many of the early bugs have been worked out and the commercially available platforms make collection of data via mobile phone or tablet quite easy.
  • New platforms are easy to use, there are many to choose from and most include built in dashboards that help you to review and visualize your data.
  • A careful assessment of network coverage, power and power back-up should be done before you decide on a type of phone and platform.  While you don’t necessarily need a signal to use mobile phones to collect data (you can collect data offline) you will need a phone with long battery life! Many phones are now GPS enabled—and you should consider these if you would like to collect GPS waypoints and conduct geospatial analyses.
  • Understand the skills and competencies of your data collectors.  Will they be people who are familiar with mobile phones or will they need significant training and mentoring?  What type of phone will be easiest for them to use?
  • If you are using mobile phones for data collection of a baselines survey, for example, will you have a follow on use for the phones? You’ll want to consider what type of phone will be most useful for future activities so that you can yield a higher return on investment of your initial purchase.

Lessons Learned:  Bruce Mobile phones 2

  • A careful assessment of your data collection needs, logistical issues, and possible future projects is necessary before you start utilizing mobile technology.
  • Because not everyone sees the benefits of mobile technology, a basic overview of the advantages of this innovation is helpful to get your co-workers on board.

Rad Resources:

  • Online mobile technology training for a variety of uses is available for a fee from TechChange.
  • There is a free online mobile data collection selection assistant at NOMAD.

*Thank you to Mobenzi Researcher and DataWinners (DataWinners free data collection App for Android devices built using Open Data Kit tools) for the use of their images in this post.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

We are Hilary Loeb and Kelly Bay of the Research and Evaluation Department at the College Success Foundation.  Many of our scholarship and support programs host events in which we collect data from students and educators.  As internal evaluators, we often rely on colleagues to collect and enter survey data from these groups. The results are used for staff learning internally and external reporting.  To help evaluators increase survey relevance, decrease demands on respondents’ time, and ultimately boost data quality and response rates, below are tips on instrument design and data collection.

Lessons Learned:

Look for ways to make surveys easier for staff to administer up front and more useful to stakeholders at the back end. The key is keeping the main focus on your programs while building support for data collection and analysis efforts.

Hot Tips

Survey Design:

  • Ensure that survey content is relevant: Meet with the entire program team and start with the question, “What do we want to learn about our program?” before discussing what’s needed for grant-reporting requirements.
  • Draft a survey using previously tested questions:  You don’t have to reinvent the wheel. By using previously tested survey questions from existing “banks” of items, you can save time and often improve the quality of the data collected (see Rad Resources).
  • Pilot test surveys with your program team and other stakeholders. This exercise never fails to elicit important feedback and takes only a modest amount of time. It’s amazing what fresh eyes can find! Where possible, use trainings and even Board meetings as opportunities to pilot and discuss surveys.

Survey Data Collection:

  • Be strategic about paper versus online surveys: When event participants can’t readily access computers, paper surveys may help increase response rates.  Online surveys are more appropriate when participants are able and willing to access technology.
  • Designate sufficient time and staff to collect survey data: Ensure that there is a specific time slot dedicated for survey completion. It should be near to but not at the very end of the event.  We suggest providing a script to help staff describe the survey’s purpose and value.
  • Consider using scanning software for paper surveys: Scanning software automates data entry by reading the optical marks on paper survey forms, which can reduce errors and save time.  Before purchasing, it’s best to test. We piloted a Free Demo of Remark Office OMR, to confirm that this was the right software for our organization.

Rad Resources:

A Bing search of survey item banks yields over 60 million results.  Our favorites in the education and youth development field include:  Ansell-Casey Life Skills Assessments,  the Youth Behavioral Risk Surveillance System and  National Center for Educational Statistics resources .

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· ·

Hello. My name is Nancy Lee Leland and I am Senior Research Associate and Evaluation Team Leader for the University of Minnesota’s Prevention Research Center for Healthy Youth Development. I also do independent evaluation-related consulting with private, governmental, and nonprofit organizations. I strive to engage program and other organizational staff in the evaluation process and have found an approach that helps engage them in creating outcome-related data collection tools. This is helpful when an already existing data collection tool that fits the evaluation project is not available.

Hot Tip: Create a “Q by Q” table (Question by Question table). The step of creating a Q by Q table happens after staff have completed creating their logic model, have articulated their key evaluation questions, and have identified several indicators or measures that will help answer each question. The evaluation questions and indicators identified help define the “domains” of the Q by Q table. These domains are articulated and reviewed by staff for fit.

Once the domains are identified and agreed upon, a search is conducted to identify existing data collection tools that have questions related to the domain. Questions and other key information are transferred to the Q by Q table (See example below).

An attempt to list a good selection of questions for each domain helps with the next step (caution: these tables can be quite lengthy—but it is worth it!). This step involves bringing the completed Q by Q table to staff so they can select the questions that best suit their population. After questions are selected, the data collection tool is developed and formatted for pilot testing.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Archives

To top