Hello, I am Kerry Bruce, the Director of Results and Measurement at Pact.  I’m currently based in Madagascar and support Pact programs in Africa, Asia and Eastern Europe. I am part of Pact’s central technical team that provides monitoring and evaluation support to more than 20 country offices and more than 70 projects around the world.  In 2012 we started to roll out the use of mobile technology in our programs including evaluation.  This is the second post on mobile technology and focuses on choosing your platform. Check out my first post on this topic: Getting Started with Mobile Phones.

Hot Tips:

• Look at a wide range of available platforms and ask yourself:
  • What is my budget for phones?  Some platforms work better than others with entry level (vs. Android) smartphones.
  • What is my budget for the data collection?  Will my data collection reoccur frequently (on-going evaluation) or is this a one time event?  Each platform has a different pricing structure and each lends itself to different types of data collection.
  • Platform operators will promise you the moon – but will their platform deliver?  Test basic issues such as skip logic, ease of set-up and use, how data download and dashboards work before you buy.  Most platforms have a trial version that you can use and some allow small data collection projects for free.
  • Will I need help to set up my survey, or do I have the skill set to set it up in house?  Some platforms offer survey set up and technical support (useful for complicated data collection exercises) and some are all do-it-yourself.
  • What language will the survey be in and can the platform support it?  This is especially important for non-Latin alphabets.
  • Get a reference.  All these platforms should be able to provide you with a reference from someone who has used them before and can tell you what is good and what needs work.

Lesson Learned: Evaluate two or more platforms before you decide which one to use. 

• Some have recurrent or annual costs and others only charge for the data that you collect.  Others are free up to a certain level of data collection.
• Each platform has its strengths (and weaknesses) – you’ll need to understand what you need it to do and shop around until you find it.
• Just because a platform can not do something today does not mean they won’t be able to do it tomorrow, check back and give feedback.  This technology is rapidly adapting.

Rad Resources: Here is a list of some of the mobile technology platforms that are commercially available today.

*These are platforms I have used.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

technology

Hello!  We’re Michelle Landry and Judy Savageau from the University of Massachusetts Medical School’s Center for Health Policy and Research.

As Sean Allen Levin suggested in a recent aea365 post, organizing an evaluation project or multiple projects can be daunting. The details of timelines, responsibilities, and deliverables can easily be lost if they are only in the project manager’s head. As with any project, maintaining a quality and structured work environment benefits the entire team and clients. To expand on Sean’s blog, we’d like to share hot tips and lessons learned during our recent review of project management tools and how it’s brought to light not only the vast sea of options, but also what our needs/wants are from the tool.

Lesson Learned: Work backwards to determine your specific needs. What outputs are most useful? What information is needed to report regularly? How large is the project or how many projects need to be captured in the database?

Lesson Learned: Determine if you have a budget or spending limit to support a new tool. There are affordable options that aren’t readily obvious. However, if you have the budget, there are options to satisfy your every whim.

Hot Tip: For no/low cost options, look to Microsoft Office. This is affordable because most users already have the software; e.g., MS Excel, MS Access, and MS Project, which have onsite relational databases with ranges in user-friendliness/abilities. A number of software vendors sell robust project management tools, but they come with a price tag. We reviewed tools used by our university colleagues; e.g., Quickbase (now piloting) and Journyx. Many others are available with websites offering comparisons among the applications.

Lesson Learned: Review your needs against the software’s options. Many websites allow testing project management tools through a virtual tour. Take advantage of this; it’s best to see how user-friendly the software is before purchasing.

Lesson Learned:  Each application has budgetary implications, so if, of necessity, you’re budget conscious, check into the vender’s software offerings. Is it a one-time cost or license needing annual renewal? Does it require a monthly user fee?

Hot Tip: Customize, customize, customize… Most software packages are customizable. Do not take it “out of the box” and assume that’s all you get. Many vendors offer customization options to meet your needs. Customizing can take a few rounds as test-driving one change often uncovers additional changes. Customizing the software may reduce frustration and better meet your needs. See if there’s someone “in-house” who can customize your software before paying the vendor – a great budget saver!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

project management · tech tools · technology

Hello, we are Linda Cabral and Laura Sefton from the Center for Health Policy and Research at UMass Medical School. We often collect qualitative data from interviews and focus groups. One challenge we frequently face is how to quickly and efficiently transcribe audio data. We have experimented using voice recognition software (VRS), and we’d like to share our approach.

You will need headphones, a microphone (stand-alone or attached to a headset), and a computer with audio playback and VRS installed on it. We use Dragon Naturally Speaking Premium Version 11.5 voice recognition software, however other VRS is available. Use of audio playback software will allow you to control the playback speed, so you can slow it down, pause, fast forward, and rewind as needed.

Open the audio file in the playback software and open a new document in the VRS. While listening to the audio via the headphones, repeat what you hear into the microphone. During this step, you can format the document to indicate who is speaking and to add punctuation. Because VRS works best when trained to understand a single voice, a designated team member should repeat all spoken content, regardless of how many voices are in the audio file.

This process will generate a document in the VRS that can be saved to your computer as a Word file. As a final review, read through the Word file while listening to the audio file and make needed corrections. This could be done by another member of the project team as a double check of the document’s accuracy.

Hot Tips:

• Spend time training the VRS to recognize your voice. A few practice sessions with the software may be needed where you can read dummy data into the software in order for it to learn your voice. This will improve the transcription quality, minimizing the time spent editing.
• Train the VRS to recognize project-specific acronyms or terminology prior to starting transcription.

Lessons Learned:

• Often, financial resources for evaluation projects are limited. In an effort to keep the transcription process in-house, our administrative staff transcribed the audio files. By using the VRS and someone from our project team familiar with the data as the designated recorder, we have found savings in time and efficiencies.
• No transcription yet has captured 100% content accurately the first time. Therefore, build in time to listen to the recording and to make manual edits.

Rad Resources:

These resources may be helpful as you explore whether VRS is right for you.

• “The Voice Transcription Technique: Use of Voice Recognition Software to Transcribe Digital Interview Data in Qualitative Research” by Jennifer Matheson

• VRS products Review by consumersearch: “In reviews, it’s generally Dragon vs. Dragon”
• 

  (Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

qualitative · technology · transcription

Hi, I am Kerry Bruce, Director of Results and Measurement at Pact.  I am part of Pact’s central technical team that provides monitoring and evaluation support to more than 20 country offices and more than 70 projects around the world.  In 2012 we started to roll out the use of mobile technology in our programs. We have begun integrating mobile technology into our programs by using mobile phones for baseline and endline data collection.

Hot tips:

• Mobile technology has advanced significantly since the last time you likely considered using it and now is the time to invest in learning about this technology.  Many of the early bugs have been worked out and the commercially available platforms make collection of data via mobile phone or tablet quite easy.
• New platforms are easy to use, there are many to choose from and most include built in dashboards that help you to review and visualize your data.
• A careful assessment of network coverage, power and power back-up should be done before you decide on a type of phone and platform.  While you don’t necessarily need a signal to use mobile phones to collect data (you can collect data offline) you will need a phone with long battery life! Many phones are now GPS enabled—and you should consider these if you would like to collect GPS waypoints and conduct geospatial analyses.
• Understand the skills and competencies of your data collectors.  Will they be people who are familiar with mobile phones or will they need significant training and mentoring?  What type of phone will be easiest for them to use?
• If you are using mobile phones for data collection of a baselines survey, for example, will you have a follow on use for the phones? You’ll want to consider what type of phone will be most useful for future activities so that you can yield a higher return on investment of your initial purchase.

Lessons Learned:  

• A careful assessment of your data collection needs, logistical issues, and possible future projects is necessary before you start utilizing mobile technology.
• Because not everyone sees the benefits of mobile technology, a basic overview of the advantages of this innovation is helpful to get your co-workers on board.

Rad Resources:

• Online mobile technology training for a variety of uses is available for a fee from TechChange.
• There is a free online mobile data collection selection assistant at NOMAD.

*Thank you to Mobenzi Researcher and DataWinners (DataWinners free data collection App for Android devices built using Open Data Kit tools) for the use of their images in this post.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Data Collection · data collection tool · technology

I am Tarek Azzam, assistant professor at Claremont Graduate University and associate director of the Claremont Evaluation Center.

Today I want to talk about crowdsourcing and how it can potentially be used in evaluation practice. Generally speaking, crowdsourcing is the process of using the power of the many individuals (i.e. the crowd) to accomplish specific tasks. This idea has been around for a long time (e.g. the creation of the oxford dictionary), but due to recent developments in technology, the ability to access the power of the crowd has become much easier.

I will focus on just one crowdsourcing website because Amazon’s Mechanical Turk is the most widely known, used, and studied crowdsourcing site. This site helps to facilitate the interactions between “requesters” and “workers” (see figures below). A requester can describe a task (e.g. please complete a survey), set the payment and allotted time for completing a task, and determine the qualifications needed to finish the task. This information is then posted on MTurk website, and interested individuals who qualify can complete the task for the promised payment.

(Share Clip)

This facilitated marketplace has some really interesting implications for evaluation practice. For example, evaluators can use MTurk to establish the validity and reliability of survey instruments before giving them to intended participants. By posting a survey on MTurk and collecting responses from individuals with similar background characteristics as your intended participants, an evaluator can establish the reliability of a measure, get feedback on the items, and if needed translate the items into another language. All this can be accomplished in a matter of days. For me personally I’ve been able to collect 500 responses for a 15 minute survey, at a cost of 55 cents per survey in less than three days.

Hot Tip: when selecting the eligibility criteria for MTurk participants choose those with 95% or higher approval ratings.

There are other uses that I am currently experimenting with. For example:

• Can MTurk respondents be used to create a matched comparison group in evaluation studies?
• Is it possible to use MTurk respondents in a matched group pre-post design?
• Is it possible to use MTurk to help with the analysis and coding of qualitative data?

These are things that are yet to be known but I will keep you updated as we progress in exploring the limits of crowdsourcing in evaluation practice.

(Share Clip)

Hot Tip: I will be presenting a Coffee Break Demonstration (free for American Evaluation Association (AEA) members) on Crowdsourcing on Thursday April 18, 2013 from 2:00-2:20pm EDT. Hope to see you there.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Data · Evaluation · survey · technology

My name is David Fetterman.  I’m President & CEO of Fetterman & Associates, an international evaluation consulting firm (with 25 years experience at Stanford University) and past-president of the American Evaluation Association (AEA).  I am probably best known for empowerment evaluation work (helping people learn how to evaluate their own programs).  For examples see our blog and an article about empowerment evaluation in the School of Medicine at Stanford University in Academic Medicine, and the book Empowerment Evaluation in the Digital Villages:  Hewlett-Packard’s $15 Million Race Toward Social Justice, Stanford University Press

LGBT-Related Survey

One of my recent evaluations, conducted with my  Stanford School of Medicine students, focused on LGBT curricular training in medical schools throughout the U.S. and Canada.  The results – a median reported time of 5 hours of LGBT-related content in the entire curriculum – were published in this article in the Journal of the American Medical Association (JAMA). It received considerable attention in the press, in part because it is as much a human rights issue as a medical education issue. I’ll share a few tips and tricks that emerged from conducting and publishing this study.

Hot Tip:  We used an online survey program to ask Deans of Schools of Medicine to evaluate their institutions’ level of coverage of 16 LGBT related topics.  Online survey tools, such as SurveyMonkey, save time and money and sort data almost instantaneously.  Surveying Deans automatically enhances the credibility of findings (especially if findings suggest minimal coverage of the material, as in our case).

Reporting survey findings was as much a social responsibility as a scholarly one.  See Anne Dohrenwend’s challenge to speak out about gay rights in Academic Medicine.

Cool Trick: Videoconferencing programs, including Skype, ooVoo, and Google Hangouts are invaluable tools to facilitate communication with team members at remote sites.  Most team members were located across the country, completing residency requirements.  Videoconferencing allowed us to function remotely and inexpensively.

Rad Resource: The Association of American Medical Colleges maintains a curriculum management and Information (CurrMIT) database that helps you determine the coverage of specific topics in medical schools. This database was particularly useful as a form of triangulation when our reporting format – “reported hours of instruction” – was questioned in a draft of our article.

Recommended LGBT cultural competence resources:

Fenway Health

Institute of Medicine of the National Academies

Lesson Learned:  Be prepared for significant opposition to unpopular or controversial findings.  Be prepared to speak with the press.  Highlight key findings and recommendations simply and concisely and be prepared to see how journalists use the information (see example of highlighted findings in New York Times.)  Appreciate your team and enjoy the media blitz for as long as it lasts.

aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. We’re celebrating LGBT Evaluation week with our colleagues in AEA’s LGBT Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. 

blog · cultural competency · lgbt · survey · technology

Hi. My name is A. Rae Clementz and in addition to being the co-chair of the Graduate Student and New Evaluator TIG, I am also a techie.  I believe technology is of value when it helps us accomplish our goals in ways that are better, easier, and/or cheaper. I have evaluated several educational technology integration programs. Consistently one of the biggest barriers to successful implementation is teachers’ perceptions of the tool’s cost-benefit ratio. If the cost is too high, it’s a non-starter; the program or cool new toy will never fit in their school’s stretched budget. Even if the tool is free, if it’s too hard to use or doesn’t add some new or improved dimension to student learning, it’s not worth the effort.

I often feel similar time and budget constraints in my evaluations. Below are some cheap, efficient, and effective tools for two common evaluation tasks.

Rad Resource for conducting & recording interviews:

• Google Voice | I’m one of those people who only has a cell phone. To avoid burning minutes during the day, I make my calls with Google Voice. Google voice uses the internet connection on either your computer or cell phone to make calls. Bonus feature: incoming calls can be recorded, and Google Voice automatically creates a transcript and .mp3 recording of the call in your Google Voice Inbox!
• Skype + Evaer or Pretty May | Skype is one of the most common video and voice conferencing tools and its basic levels are free. Evaer and Pretty May are programs that record the voice and video feeds of Skype and save them out as either .mp3 or .wav files. Pretty May is free, as is the basic version of Evaer. Full version of Evaer is $20 with lifetime support and upgrades.

Lesson Learned:

It is critical when recording anything that you inform everyone that you’re recording the call, for what purposes, and ask them if they agree to be recorded. Many states have laws prohibiting unauthorized recording of phone conversations.

Rad Resource for disseminating evaluation findings:

• Weebly + Scribd | Weebly is a simple, free, drag-and-drop, web-based, website design program. If you can use e-mail and PowerPoint, you can create a website using Weebly. Scribd is a free online publishing site. You can upload documents and either direct people to them on Scribd or embed them in websites or other social media sites.

Lesson Learned 1: Sadly, just because you built it doesn’t mean they’ll come.  But having a website for the evaluation is still a good way to provide transparency, encourage comment from stakeholders, and disseminate findings to broader audiences. The process of building the site also promotes more organized communication about the evaluation.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Rae? She’ll be presenting as part of the Evaluation 2012 Conference Program, October 24-27 in Minneapolis, MN.

findings · recording interviews · technology

My name is Melissa Cater, and I am an assistant professor and evaluation specialist at Louisiana State University AgCenter.

“At its most fundamental, visual research draws on our basic

capacity to interpret the world through our sense of sight.”

                                                Christopher J. Pole

Visual evaluation methods leverage the power of art, photography and video as a lens for exploring program outcomes. The images provide a bridge between thought and verbal expression.

Hot Tips:

Pose a simple question to which participants can respond with artistic expressions, photographic representations, or video productions. In two recent evaluations, we asked school-aged children to draw a garden. It was a very simplistic request, but the results were quite insightful.

Provide training on equipment and/or software. While this seems to be very obvious, we sometimes overlook the most basic needs. A brief refresher on equipment operation is beneficial for everyone, particularly if you are providing equipment with which everyone is not familiar. Simple tricks for framing shots, using ambient lighting, and getting sounds bites are always useful. Finally, don’t forget to offer guidance in using photo and video editing software.

With the evolution of social media, the opportunities to use mobile devices, like cell phones and tablets, bring a whole new dimension to this approach. Collaborative planning between the evaluator and participants is essential for success. While the mobile platform (e.g. cell phone, tablet, social media, apps) may be familiar, the evaluation process will probably be very new.

Allow time for both individual and group reflection. Participants must first delve into their own thoughts. Providing a structured process for guiding this reflection is especially useful with youth. We’ve had success using graphic organizers for this step. The advantage is that the graphic organizer may then be used to scaffold group discussion.

As group discussion evolves, use group facilitation skills to help participants compare and contrast their individual interpretations. Ultimately, the individual contributions become part of the larger group story.

Rad Resource:

Pole, C. (2004). Visual research: Potential and overview. In C. Pole (Ed.), Seeing is believing? Approaches to visual research. San Diego, CA: Elsevier.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Melissa? She’ll be presenting as part of the Evaluation 2012 Conference Program, October 24-27 in Minneapolis, MN.

photography · technology · visual

Hello, I am Maxine Gilling, Research Associate for Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP). I recently completed my dissertation entitled How Politics, Economics, and Technology Influence Evaluation Requirements for Federally Funded Projects: A Historical Study of the Elementary and Secondary Education Act from 1965 to 2005. In this study, I examined the interaction of national political, economic, and technological factors as they influenced the concurrent evolution of federally mandated evaluation requirements.

Lessons Learned:

• Program evaluation does not take place in a vacuum. The field and profession of program evaluation has grown and expanded over the last four decades and eight administrations due to political, economic, and technological factors.
• Legislation drives evaluation policy. The Elementary and Secondary Education Act (ESEA) of 1965 established policies to provide “financial assistance to local educational agencies serving areas with concentrations of children from low-income families to expand and improve their educational program” (Public Law 89-10—Apr. 11, 1965). This legislation also had another consequence: it helped drive the establishment of educational program evaluation and the field of evaluation as a profession.
• Economics influences evaluation policy and practice. For instance in the 1980’s evaluation took a downturn due to the stringent economic policies. Program evaluators resorted to lessons learned through writing journals and books.
• Technology influences evaluation policy and practice. The rapid emergence of new technologies all contributed to changing goals, standards, and methods and values underlying program evaluation.

Resources:

•  HeinOnline: The Modern Link to Legal History. HeinOnline consists of major full text legal collections including: U.S. Federal Legislative History Library where you can study the changes in evaluation policy over time.
• Elementary and Secondary Education Act (also known as the No Child Left Behind Act).
• David Cohen’s historical article:Politics and Research: Evaluation of Social Action Programs in Education.
• Caroly Weiss’ historical book:Evaluating action programs: Readings in social action and education.
• Ralph W. Tyler’s many works. For example see the chapter, Ralph W. Tyler’s contribution to program evaluation in the book Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

economics · Elementary and Secondary Education Act · ESEA · history · NCLB · No Child Left Behind Act · policy · politics · practice · technology

Hi, I’m Craig Wiles, Senior Consultant at Public Sector Consultants in Lansing, Michigan. I provide research and evaluation services for clients in health and human services, education and the environment. I am sharing a tip on how to use Tableau as part of a data exploration process with a group of stakeholders.

To begin, I did the heavy statistical lifting outside of Tableau, so this would not lapse into a data-mining exercise. In this case, I worked with a state-level stakeholder group to identify data sources, research priorities, and statistically significant correlations in the data. Once we had our short list of correlated variables to explore in more detail, we convened a series of two hour, interactive data exploration sessions. At these sessions, we used Tableau to visually display the data (in this case, educational data), identify high and low performing school districts, and look for other obvious patterns or outliers in the data.

We tended to use stacked bar charts and scatter plots to help with this visual part of the data exploration. One tool in Tableau that was especially helpful in this context was the filter bar. Using the filter tool, we were able to adjust the range of scores we were looking for in our combination of variables according to tolerances set by the stakeholders. For example, we looked for school districts that had a high graduation rate, low dropout rate, and a higher ratio of students with disabilities in general education classrooms.

I recommend using Tableau for data exploration because it is:

• Visual,
• Interactive, and
• Builds capacity and ownership.

I could have presented this data in charts and graphs and led a typical ‘sit-and-get’ meeting and landed at the same place (conceptually) at the end of the day. This kind of visual and interactive process, however, really helped to engage my stakeholders, especially those that are usually averse to numbers and data. Ultimately, this was as much about the process as it was about the data.

After our interactive sessions, this group began a series of local focus group conversations with voluntary school districts to further explore the relationships we identified. This qualitative data has provided depth of detail and rich context to the quantitative relationships we explored together.

Using the filter tool:

Using a scatter plot:

Click to increase size

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Capacity Building · data exploration · software · technology