Sustainability is the theme for the 2014 American Evaluation Association conference to be held in Denver on October 15-18th. Donors, and evaluators themselves, are demanding more and more from the evaluator profession as accountability and transparency have become hot topics. Adding to the complexity of the profession, this year’s AEA conference has called for a “visionary” evaluation for the 21st century: Using systems thinking, how can evaluators tie the complex, interconnected worlds in which they work into long-term global sustainability?
Evaluation will need to rely ever more on technology at both the micro level (tablets and cell phones for data collection) as well as the macro (ICTs for visualization, mapping) in order to envision an answer. To show you how ICT needs have evolved in the evaluation field, this year’s conference highlights panels such as “How to Use Analytics in a Visionary Evaluation” while the professional development workshops have doubled their “Data Visualization and Reporting” offerings, including classes in “Evaluation Dashboards” and the “Role of Data Visualization in Reporting.”
The step from using technology for data collection to mapping and visualizing has been building over the last several AEA conferences. Mobile data collection panels were the most popular sessions last year in Washington, DC. Crowds, sitting on floors and squeezed against the back wall, packed into sessions to hear the best cellphone and tablet options, software tips, and lessons learned from the field. Evaluators from the health, banking, and agriculture sectors drilled presenters from the National Opinion Research Center (NORC), Pact, and Population Services International. Representatives from UNICEF wanted to know the best IT strategy for field support. Mobile Accord, a Denver-based socially responsible mobile platform, inquired about the advantages of buying devices and software in-country versus pre-loading and shipping devices in pre-tested.
Here is a sample of last year’s takeaways to give you a sense of what to expect at this year’s AEA conference:
1. Buy devices, software, and support in-country
For large scale surveys, Daisy Kisyombe of Pact recommended buying devices, software, and support in-country, as import taxes were normally cost-prohibitive. Android phones can easily be purchased for under forty U.S. dollars in her native Tanzania, and survey software and support were available all over Africa from Mobenzi, based out of South Africa. In Tanzania, the cost of doing survey evaluations among farmers had dropped to one cent per survey, allowing for a typical 100 person sample to be done for as little as six hundred U.S. dollars.
2. Select a field surveyor as a point person for IT support
For IT support, several NGOs recommended selecting a field surveyor as point person on cell phone and software support. In exchange, the surveyor will receive the phone at the end of the year. “Theft is never an issue for the phones,” Daisy said. “It is the wear and tear in the field that is the issue. This is an incentive to take care of the phones. And the more surveyors who volunteer, the more we make them teach us IT support back at headquarters.”
3. Use mobile phones to reduce data collection costs
Aleck Dhliwayo of Population Services International (PSI) in Zimbabwe shared that their evaluation survey costs had also dropped dramatically since moving to mobile phone collection. It also allows headquarters to review data in real-time. If there are any questions, headquarters can call surveyors to clarify while they are still in the field. Results used to take up to three weeks on the old paper-based system. Now they have them instantly. Yet, Aleck, too, said the management of new technology required a different mindset: It is tempting to collect more information than you need. But that information needs to be analyzed, stored, and linked to other systems – Data collection used to be the driver of cost. “Now it is the analysis that is more expensive,” Aleck said.
4. Capture more quality data in different forms with tablets
The non-profit NORC, based out of the University of Chicago, has a long track record of working with tablets overseas, a mode of survey evaluation everyone agrees will greatly open up the response possibilities. Cell phone surveys usually limit qualitative responses to one word. Tablets can show a farmer an image of his crops, record his thoughts for transcription, and still capture the traditional quantitative data associated with cell phone surveys. Yet, echoing Aleck from PSI, Samuel Haddaway from NORC says analysis and transcription can be expensive, especially in languages other than English. Yet the metadata produced, such as time stamps from when the evaluation was conducted, is invaluable for data quality. NORC normally buys tablets in-country, but doesn’t give them to surveyors as they are too expensive right now. They leave them for the home office because, again, taxes make it prohibitive to ship them out of the country.
5. Pre-test, pre-test, pre-test
The lesson all panelist stressed to international development evaluators: Pre-test, pre-test, pre-test. “The first time we did not sync the phones until we arrived in the field, where connectivity was low.” Shared Daisy Kisyombe of Pact, “It took us half a day. The next time we did it in our office, it took fifteen minutes.”
Expect this year’s AEA conference to provide more takeaways on data analysis, visualization, and mapping as the needs of the field evolve beyond the needs of data collection to ensure project success.
Christian is a recent graduate of George Washington University’s Elliott School of International Affairs mid-career program focusing on Monitoring and Evaluation in International Development. Christian has eight years of experience of program design, implementation, training, and evaluation including a USAID Rule of Law initiative in Russia; USAID private sector develop project in Ghana; and conflict analysis and evaluation with the U.S. State Department. A former international private sector consultant who conducted on-the-ground economic research in 75 countries, he has also been a business journalist and researcher for the Harvard Business School. He is currently an independent contractor working with private sector international development firms in the D.C. area.
If you interested in these topics, please enroll now in the Technology for Monitoring and Evaluation online course that runs Sep 22 – Oct 17, 2014.