AEA365 | A Tip-a-Day by and for Evaluators

TAG | Information Technology

Greetings from Catholic Relief Services (CRS)! We, Suzanne Andrews and Shaun Ferris, from the Baltimore-based Agriculture and Livelihoods Program, presented at the American Evaluation Association’s Annual Conference in Washington DC on a Farmbook suite of on/off line tools to help us to better build capacity, gather data and develop business plans with smallholder farmers.

Andrews

Photo by Suzanne Andrews (Catholic Relief Services)

Lesson Learned: A Challenge: One of the key problems we face in working with smallholder farmers is understanding who our clients are, where they live, their cropping systems, their costs of production and the market opportunities near their communities. There are very few tools to help field agents gather these types of monitoring and evaluation data in a systematic way and few means of aggregating and sharing this information.

A Product: CRS has been working to develop tools that help field agents to develop farmer group business plans, to gather data on production and profitability levels, sharing this information with farmers, local project managers and globally through a digital data platform.

Rad Resources: We manage, analyze and share our data through cloud-based data management systems that allow global users from CRS and other organizations to view our data and create customized reports.  We are also working with Nethope’s cloud services, creating webinars to share ideas and get feedback, and also link with potential users. We have held several webinars about our e-learning platforms and the business planner /profitability tool . We also share the information through our  ICT4D conferences that we hold every year, in Africa.

Lessons Learned: Field agents who tested the Farmbook business planner and profitability calculator, performed much better when they first enrolled in the e-learning course in marketing and gross margin analysis. We have developed comprehensive training curricula for smallholder capacity building, to support the farm business plan development and data gathering process.

Developing the Farmbook suite required a team of people with diverse expertise, ranging from agriculture advisors, software architects, programmers, instructional designers, subject matter specialists, editors, and artists, to innovative field managers and field agents to design, develop and test the beta versions of Farmbook.  Holding that team together in the build, test and deploy phases has been critical to getting to the starting point. We are still working on the business models!

Get Involved: If you would like to test drive our learning tools, the Farmbook business planner, or the Map and Track service delivery audit, let us know!  Contact Suzanne.Andrews@crs.org to request a training version of the software, allowing you to assess the profitability of your farm and your farmers!

The American Evaluation Association is celebrating Information and Communication Technology for Development (ICT4D) for Monitoring, Evaluation, Accountability and Learning (MEAL) week. The contributions all this week to aea365 come from members who work in ICT4D for MEAL. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi, my name is Marianna Hensley, Program Quality Manger for Health with Catholic Relief Services (CRS) in India. I currently support the Reducing Maternal and Newborn Deaths (ReMiND) project that CRS implements in partnership with Dimagi, Inc. and Vatsalya.

The ReMiND project works with government community health workers (CHW) to improve the frequency and quality of their home visits to women and children. CHWs use basic mobile phones operating Dimagi’s open-source CommCare software, which equips them with job aids to support client assessment, counseling, and early identification, treatment and/or rapid referral of complications. With the project’s use of CommCare as a case management tool and job aid for CHWs, leveraging information and communication technologies (ICT) for project monitoring and evaluation (M&E) with the same software platform was an obvious choice for ReMiND. All routine project monitoring is done through CommCare operated on basic mobile phones while data collection for the project’s baseline household survey was done using CommCare on tablets.

Lessons Learned: For all data nerds out there, imagine the excitement of realizing that ICT-enabled M&E means you get all those numbers now! Beware the lure of real-time data with ICT for M&E.

Hensley

Photo by Marianna Hensley (Catholic Relief Services)

With the use of ICT for data collection in either routine monitoring or evaluation comes the strong temptation to ask every question you can think of—just because it’s so easy to capture responses with fewer worries about the delays or errors typically associated with manual data entry following paper-based collection. The risks are multiple: 1) you find yourself left with more data than you can or feasibly will analyze and use; and 2) you hazard user (data collector) and respondent fatigue from a questionnaire that delves too deeply into non-essential information.

Faced with the lure of real-time data from ICT, M&E practitioners must remember more than ever to focus on the need to know information that supports project or evaluation decision-making and objectives.collector) and respondent fatigue from a questionnaire that delves too deeply into non-essential information.

 

Hot Tips:

  • Make sure to choose an ICT device that fits your needs in terms of screen size and resolution. Long questions or lists of select options are easier to deal with on a larger-screen rather than on a smaller-screened device that requires scrolling.
  • Don’t forget to assess the battery life of your device as part of field testing an ICT tool. And have a plan that includes resources such as solar or car chargers to ensure devices are adequately charged throughout data collection or monitoring.

Rad Resources: The ReMiND project’s monitoring tool application and baseline survey application are available for free download on CommCare Exchange.

ReMiND is featured as a case study and the example of M&E in mobile health programming in the Global Health e-Learning Center’s  new mHealth Basics course.

The American Evaluation Association is celebrating Information and Communication Technology for Development (ICT4D) for Monitoring, Evaluation, Accountability and Learning (MEAL) week. The contributions all this week to aea365 come from members who work in ICT4D for MEAL. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi, my name is Shenkut Ayele, Early Warning Assessment and Response Manager with Catholic Relief Services (CRS) for the Joint Emergency Operation (JEOP) in Ethiopia. JEOP is a USAID-funded emergency food assistance program that over two years is providing food aid to almost 1 million people. The program operates across Ethiopia and is a partnership between many agencies and government.

Until last August, I faced serious challenges: data were slow to arrive and often of poor quality. As a result, reports were delayed and decision-making hampered with serious consequences for JEOP’s ability to respond effectively. However, since August 2012, JEOP has been using an innovative solution that is strengthening our ‘Participatory Early Warning and Response System’. We are using DataWinners, an SMS-based solution implemented in partnership with Human Network International. Registered individuals across 79 districts collect and upload data via SMS each week onto a web-based database. I am able to use these data in real time to inform decision makers. Here are two graphics how the system works and the data collection and information flow.

Ayele 1

Ayele

Lessons Learned: After implementing our system for one year, we have learned that:

  • Vulnerable communities should be viewed as both sources and recipients of early warning information.
  • Adoption of our new SMS-based system has empowered local officials who are now using the reports to undertake better estimates of the number of individuals who might be affected by a disaster.
  • Local officials are better able to represent the needs of vulnerable communities in discussions at higher levels of government.
  • Local officials and others in JEOP have found the better quality data has improved their ability to target the most vulnerable communities.
  • The system has the potential to accommodate other innovative uses, and government officials have expressed their interest to adopt the SMS system more widely.

Hot Tip: An effective SMS-based system provides a strong basis for a participatory early warning and response system because it enhances the likelihood that any data generated will be used to support better decision-making among different users.

The American Evaluation Association is celebrating Information and Communication Technology for Development (ICT4D) for Monitoring, Evaluation, Accountability and Learning (MEAL) week. The contributions all this week to aea365 come from members who work in ICT4D for MEAL. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Hi, my name is Or Dashevsky, Chief Solution Architect for Catholic Relief Services based in Baltimore. I’m responsible for providing technical leadership to a team who develop CRS’ enterprise architecture.

Malaria is endemic in Sierra Leone, with stable and perennial transmission in all parts of the country. As such, the entire population is at risk of developing the disease. Malaria accounts for about 50% of outpatient morbidity and is presently the leading cause of morbidity and mortality among children under five years of age, with a mortality attributed to malaria estimated to be 38% among this age group and 25% for all ages (Outpatient morbidity statistics, MoHS, 2009, MIS 2010).Dashevsky

Catholic Relief Services (CRS) and the Ministry of Health and Sanitation (MoHS) of Sierra Leone are co-implementing a Global Fund project to fight AIDS, Tuberculosis, and Malaria (Global Fund Round 10). The overall goal of the Global Fund Round 10 Malaria project is to achieve the malaria-related Millennium Development Goals (MDGs) by 2015, not only nationally, but also among the poorest groups across Sierra Leone.

In order to track progress and impact, CRS led the implementation of a Malaria Indicator Survey (MIS) from 31 January – 8 March 2013 covering 6,720 households throughout the country.  Despite the great surge of mobile technologies to accelerate data collection, all surveys prior to this were using paper-based systems in Sierra Leone. The 2013 MIS used Apple 3GS iPhones to collect data via the iFormBuilder platform, a Software as Services application allowing for timely data collection, monitoring, and analysis.

Lessons Learned:

  • Allow enough time to digitize paper questionnaires: It took approximately 9 weeks of intense programming and testing over a 10-month period to program the MIS questionnaire into iFormBuilder.
  • Allow enough time to pre-test: The tool was pre-tested in 100 households in both rural and urban areas three months prior to the start of MIS data collection.
  • Spend enough time on training enumerators prior to data collection:  Data collection training for 28 teams lasted three weeks, which was necessary to ensure that all individuals collecting MIS data fully understood the questions, the functioning of iPhones, and the sequencing and logic of the questionnaires
  • Provide central technical support throughout data collection effort:  Throughout data collection, a CRS Freetown-based team was available 16 hours a day to respond to phone calls from the field teams, especially during the first 10 days of fieldwork. This allowed for real-time review of data and timely corrections.

Hot Tip: Digital data collection will improve timelines and accuracy of the data. It may look more expensive than traditional paper-base systems but in reality cost of digital data collection can be less in the long run.

The American Evaluation Association is celebrating Information and Communication Technology for Development (ICT4D) for Monitoring, Evaluation, Accountability and Learning (MEAL) week. The contributions all this week to aea365 come from members who work in ICT4D for MEAL. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi! My name is Mike Matarasso and I’m responsible for leading the design, testing and global roll out of a Monitoring and Evaluation / Information Communication Technology (M&E/ICT) platform for Catholic Relief Services (CRS). The platform will help us gather timely and high quality data to track performance across the agency, to inform change at project level and to report to donors and other stakeholders. The platform includes:

  1. Recommended mobile devices for simple and detailed data collection with options for solar charging and collection/syncing offline,
  2. A standard form library and form building interface where projects can select and use existing forms, adapt existing forms or create customized forms,
  3. A database with an interface for data management, cleansing and advanced analysis,
  4. A Geographical Information System (GIS) interface for mapping service delivery,
  5. A real time web reporting and dashboard interface with a standard library
  6. A complete training curriculum for users and support staff
  7. A help desk with tiered service support

We’ve built and piloted the platform in one food security project in Ethiopia with outcomes for Water, Sanitation and Hygiene (WASH), food security, mother and child health, public works and microfinance. Following the initial design, validation was done with five additional projects across four countries.

Experiences from this pilot will be combined with a cost-benefit analysis and field assessment carried out with Accenture and a global CRS project level assessment of requirements to determine the architecture and next steps for scaling globally.

Lessons Learned: If you were to design a similar system, here are some suggestions:

  • A mandate and support from leadership are essential
  • Sufficient budget should be planned and in place
  • The right number of qualified staff should be available to work on building the system and for piloting in the field. Everyone should believe in the system and be excited about it!
  • Requirements should be documented and confirmed by all stakeholders before starting work
  • Testing and adaptation are imperative and should be done in one project before all kinks are worked out. And did I say testing?
  • A training curriculum should only be developed after the initial system design is complete.  Otherwise the training materials will constantly change and be outdated as will the knowledge and skills of the trainees. Intensive mentoring is required.
  • A cost benefit analysis is integral to make a business case for the platform and to improve adoption
  • An Information Technology (IT) help desk and skilled support network need to be in place.
  • Focus on small, realistic releases and timelines and get something done initially to demonstrate success to others. Work in phases rather than expecting to deliver everything at once.

The American Evaluation Association is celebrating Information and Communication Technology for Development (ICT4D) for Monitoring, Evaluation, Accountability and Learning (MEAL) week. The contributions all this week to aea365 come from members who work in ICT4D for MEAL. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Hi, my name is Guy Sharrock. I work in a small team of advisors working for Catholic Relief Services (CRS) responsible for improving the quality of the agency’s Monitoring, Evaluation, Accountability and Learning (MEAL) activities. CRS is working to improve the lives of individuals in over 90 countries. CRS views the rapid expansion of cellular technology as hugely transformative for even the most vulnerable communities: in 2013, the rate of mobile-cellular penetration in developing countries stands at 89 per cent. For the last five years a cross-departmental Community of Practice (CoP) has sought to understand and promote the use of Information and Communication Technology for Development (ICT4D), inter alia, to improve the efficiency and effectiveness of the agency’s MEAL operations.

The theme of this week is ICT4D for better MEAL. Our journey to date has been both exciting and challenging: over the last few years, the agency’s use of ICT4D has become widespread with over 100 support requests logged in the last year. We have benefited greatly from the work undertaken by many other agencies that have embarked on a similar course; in return, we would like here to share some of our learning!

Hot Tips:

  • ICT4D supports better communication – data captured electronically is available in real-time, leading to prompt corrections and more accurate data, enabling timely feedback to stakeholders, and improved decision making.
  • ICT4D generates greater impact – technology provides greater access to information to help empower local communities, and provides a medium for voices to be heard.
  • ICT4D enables programs to reach scale more quickly – permitting greater coverage and reduced intervention costs per person.
  • ICT4D improves cost efficiencies – the technology pays for itself by eliminating expenses associated with conventional data collection and reporting approaches, and by allowing re-usability of its components.

Lessons Learned:

  • Understand context – how the solution will be used; the impact on day-to-day practices of different stakeholders; environmental (e.g., connectivity and power), political (e.g., government regulations), and cultural (e.g., language, gender, and education levels).
  • Identify solutions that do not require heavy investment in infrastructure; software services that are easily configured and maintained; solutions that are appropriate to users’ data needs and that work in occasionally connected environments with intermittent access to power and work across a range of user devices.
  • Articulate the true cost of the solution including procurement, deployment, maintenance and support over the life of the solution, not merely over the project cycle.
  • Compare the costs of working with and without the support of an ICT4D solution.
  • Implement a change management program to facilitate the behavior changes required to enjoy the benefits of ICT4D for enhanced MEAL.

Rad Resources: Check out the learning taking place around CRS’s annual conferences.

Clipped from http://www.crsprogramquality.org/ICT4D/

The American Evaluation Association is celebrating Information and Communication Technology for Development (ICT4D) for Monitoring, Evaluation, Accountability and Learning (MEAL) week. The contributions all this week to aea365 come from members who work in ICT4D for MEAL. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

I’m Cameron Norman and I’m the Director of Evaluation at the Peter A. Silverman Global eHealth Program at the University of Toronto. Since the first time I saw the World Wide Web back in the early 1990’s, I was interested in how social tools and technologies like it could be used to help people solve problems, learn together and share what they know. Some think my evaluation niche is technology, but that’s rather misleading. The tools always change, but people — real humans — remain the constant no matter whether we’re using one of those ‘brick’ mobile phones or an iPad. We’re natural learners and love to share and work together so anything that lets us do more of that and do it at a distance is going to be an attractor. Evaluation can help de-mystify technologies and make them accessible to more people, in more places, more often.

Lessons Learned: One of the biggest lessons I’ve learned is to never make assumptions about how any person or group will use a tool or technology. I’ve seen seniors who are more savvy with the Internet than 15-year olds and single working moms who are more prolific on social networks than full-time media professionals. One of my early surprises was learning how well street youth navigated through life using information technology in ways that often dwarfed that of their wealthier, more technology-enabled peers. The surprises never end.

Rad Resource: Twitter. Twitter answers questions you never thought to ask and gets answers from people you never knew existed. People are shocked when I tell them that I, a university professor, get more high quality, useful and relevant evaluation information from Twitter and blogs than I do scholarly databases.

Tips & Recommendations: Dive in with both feet. Learn by doing. Create a blog, start a Twitter feed and never worry about getting it “right”, because there isn’t such a thing. There are strategies that are more appropriate than others depending on what you want to accomplish, but the more you share thoughtfully, openly and with your own voice, the more the online world will give back to you. It’s quite remarkable and a lot of fun. I find it helpful to protect a small amount of time regularly to contribute to my blog and Twitter feed and use ‘dead time’ — like riding the bus or waiting in line — to catch up on my reading using a mobile device to make it all more productive.

This contribution is from aea365, the AEA Tip-a-Day alerts. Just starting out on Twitter? Consider following @aeaweb for headlines and resources from AEA or follow @cdnorman to learn more from Cameron.

DMNYQHDWSNXX

· ·

Archives

To top