Experiences with mobile data collection in UNHCR standardised expanded nutrition surveys
By Ellen Cecilie Andresen, Mélody Tondeur, and Caroline Wilkinson
Since 2011, Ellen Cecilie Andresen has been a nutritionist with the Public Health Section at UNHCR Headquarters. She led the initiative to introduce and scale-up the use of mobile data collection in UNHCR SENS surveys in many African and Asian countries. She has experience in emergency nutrition and in designing and analysing nutrition and food security surveys.
Mélody Tondeur is currently a UNHCR consultant. She is a researcher and public health nutritionist specialising in micronutrient malnutrition and emergency nutrition assessments. She has extensive experience in the design and analysis of nutrition surveys. Her field work experience includes many countries in Africa.
Caroline Wilkinson is the Senior Nutrition Officer with UNHCR Headquarters. She has been fully involved in the development of the SENS and the introduction of mobile data collection in UNHCR SENS surveys. She worked previously for 14 years with Action Contre la Faim (ACF) in several countries and the HQ in Paris.
The authors would like to thank the team at CartONG for providing ongoing support on mobile technology to UNHCR colleagues in the field and for sharing their experiences; the SENS supervisors for sharing their experiences; Ismail Kassim, former UNHCR Nutritionist in the Regional Support Hub Nairobi, for his contributions to implementation of SENS with MDC in the region; Paul Spiegel, Deputy Director of Division of Programme Support and Management, UNHCR and Marian Schilperoord, Chief of Public Health Section, UNHCR for supporting the mobile technology initiative and review of this article.
Location: Global
What we know: Mobile phone data collection (MDC) is increasingly used by UNHCR in nutrition surveys.
What this article adds: UNHR conducted a review of MDC survey experiences over 2 years. It found the approach improved data quality, saved time in data collection and analysis, the software and hardware are user friendly, and there are cost savings. Training and technical support to survey coordinators and enumerators was effective and critical to MDC success.
In November 2011, UNHCR and partners conducted the first nutrition survey using mobile data collection (MDC) in Kakuma refugee camp in Kenya. Based on the initial positive experience, UNHCR continued and expanded MDC use in nutrition surveys; it is now an integral part of the revised (version 2) UNHCR Standardised Expanded Nutrition Survey (SENS) guidelines1. Box 1 outlines how MDC was developed and works in practice. After two years of using MDC in SENS, a review of experiences has been conducted to guide further use and development of the technology and is summarised here.
Box 1: How MDC works in surveys
To use mobile technology for data collection in surveys, UNHCR has chosen a system using standard smartphones with an android platform compatible with Open Data Kit (ODK) applications. ODK is a set of free, open-source applications for creating questionnaires and storing data. The equipment and software needed to set up MDC are android smartphones (one to two per team), ODK applications, wireless router, computer and steady power supply.
The standardised SENS questionnaires are readily programmed and available for MDC and can be reused in any setting. During data collection, the survey teams record all responses directly on the phones. Every day, survey coordinators and supervisors review questionnaires for inconsistencies and provide immediate feedback to the enumerators. At the end of the day and once data quality is ensured, the phones are connected to a local network set up specifically for the survey and data from the phones are transferred to an offline server, where multiple data can be stored. The offline server is accessed through a computer connected to the same local network, and downloaded to Excel-readable format ready for analysis. There is no need for an active internet or mobile network connection to collect and save data.
A beneficial feature of the ODK is that one can switch between languages during an interview, or the enumerator can fill in the questionnaire using one language and the supervisor and survey manager can check in another. This was used in Sudan, for example, where the survey was conducted in Arabic and checked in English.
From the onset of the MDC initiative, UNHCR Public Health Section has worked in close collaboration with the technology non-governmental organisation (NGO) CartONG in the development of all technical elements, training and capacity building, as well as on-going support and development. A strong support system exists, where all field operations have access to remote or in-country support from UNHCR HQ, UNHCR Regional Support Hubs, or CartONG. For more details, see under lessons learned (training and support).
Experiences were collected through a standardised open-ended questionnaire provided to all eleven SENS survey coordinators who had used MDC by early 2014 (100% response rate), through mission reports from UNHCR´s technical partner CartONG2, outcomes of a teleconference held in 2013 with experienced MDC in SENS users and technical advisers, and data extracted from SENS reports using MDC, including SMART plausibility reports3. Open-ended questionnaires were sent via email to survey coordinators and covered a wide variety of topics, such as duration of various activities, challenges, and access to technical support.
Advantages to using MDC
Improved data quality
The survey coordinators highlighted that using mobile data collection has enhanced data quality. This was confirmed by 90% of SENS surveys with MDC showing excellent or good scores on the SMART plausibility checks4 and is attributed to reduced data collection errors and the electronic transmission of data avoiding possible errors during data entry5. The variables in the plausibility check that provided most penalty points were mainly age distribution and height digit preference, of which neither could have been influenced by mobile data collection.
High data quality is linked to features of the MDC technology. The questionnaires in ODK allow for setting pre-coded skip-patterns (this reduces the chance of missing questions or incorrect skipping of questions, e.g. after entering a child’s age, all relevant questions for that age group will appear), and required questions and pre-set ranges for continuous responses, such as height, weight and MUAC (e.g. if the child weighs 8.4 kg it is not possible to accidently write 84 kg as the weight range for children 6-59 months will be set to 3-31 kg).
In addition, MDC allows for cross-checking for missing data and other mistakes very soon after data collection. Daily upload and transfer of data allows for same day checking and immediate feedback by the survey coordinator. The survey coordinators reported spending less time than normal on data cleaning, on average one day per survey, and during data cleaning, have noticed less mistakes and errors than with paper-based questionnaires.
Reduced time from data collection to presentation of results
Survey coordinators stated that the use of MDC has made the data collection more efficient. Paper-based questionnaires can take up to one hour to complete in each household; when using MDC, the enumeration teams were reportedly quicker in each household and hence could survey more households per day. Reduced data collection time is linked to the pre-coded skip patterns which make the data collection flow easier, filters to automatically verify age and other parameters, and autofill of certain details such as date and household information. Use of MDC also allows for quicker production of the preliminary results, within four days of the last data collection. With paper-based surveys, double data entry can take up to one week, and cleaning of data can take several days.
Ease of using equipment and software
MDC has been well received by both the coordinators and the enumerators. Aided by proper training, guidance sheets and standardised operating procedures (SOPs) the ODK applications, server and download process are easy to manage for the survey coordinators. Survey managers also stated that it is an advantage that the data can easily be transferred to Excel files which is compatible with both ENA and EpiInfo (used in SENS data analysis).
Most enumerators quickly learned how to navigate the ODK format questionnaires and apply them in the field. However, all coordinators reported that some team members struggled with the phones when less frequent procedures had to be used; in these cases the teams were supported by rotating supervisors. During the course of the survey, significant improvements in data entry speed were noted and even those who were less comfortable at the beginning later became positive about using MDC in surveys. The survey coordinators mentioned that MDC makes logistics in the field easier as enumeration teams can more easily carry and handle one or two phones instead of a large amount of paper.
In some surveys, the coordinator had taken advantage of additional software available on the phone such as GPS and camera. For example, recording of GPS coordinates was found useful to help find the way back to households when teams had to re-visit due to absence at first visit. The camera was used in some surveys for verifying children with oedema.
No need for internet or mobile connection
The offline server system has distinct advantages. Many refugee sites are in remote areas with poor internet and mobile network connection where instant connection to a server would be impossible. Survey teams typically return to a central location where phones can be synchronised with the server. Transferring data offline is more secure6. As of Spring 2014, the offline server requires a slightly more complex synchronisation system than simple online server systems. If and when an online server is deemed more useful and data security can be assured, this solution may be used in settings where this is feasible, including sites with stable access to wifi at least at a central location or where teams are working remotely from a central location and mobile 3G is available.
Reduced cost
Using MDC reduces costs linked to the survey. Money is saved on printing and hiring data entry clerks and time spent on data collection, entry and cleaning. Some survey costs are similar to paper based surveys, such as equipment for anthropometric and haemoglobin measurements and external survey coordinators needed. Training of teams is estimated to last one day longer when the teams are to be trained on using mobile phones, however around 20% fewer days for data collection are needed. Table 1 indicates the budgetary difference between paper-based data collection and mobile phone data collection, based on costs from a paper-based survey in Dollo Ado, Ethiopia 2013. Estimates suggest approximately 25% of variable expenses are saved when using mobile data collection.
Table 1: Approximate budget costs for MDC versus paper-based SENS7 |
||
Paperbased |
MDC |
|
Fixed expenses in USD |
||
|
Survey coordinator8 |
20,000 |
20,000 |
|
Equipment for anthropometry and haemoglobin measurements |
5,300 |
5,300 |
|
Total |
25,300 |
25,300 |
Variable expenses in USD |
||
|
Training9 |
6,000 |
7,000 |
|
Mobile equipment10 |
0 |
1,000 |
|
Printing of paper-questionnaires |
400 | 0 |
|
Supervisors11 |
6,300 |
5,040 |
|
Enumerators |
14,000 |
11,200 |
|
Data collection costs |
14,700 |
11,760 |
|
Data entry clerk |
2,700 |
0 |
|
Total |
44,100 |
36,000 |
Challenges in using MDC
Phones and network
For many survey coordinators, the technical language is new and sometimes difficult to understand at first, although most coordinators get used to the technology during the first survey. Some survey coordinators have encountered problems while transferring data from smartphones to server or from server to Excel. Most problems have occurred after settings have been changed on the computer during the survey, and have shown to be related to lack of knowledge and experience with the technology. All problems have been solved with support from IT specialists in-country or with support from CartONG. In all cases, the teams were able to retrieve temporarily lost data from other back up sources including SD card on phone and temporary server files. As the technology is constantly improving, fewer and fewer problems are encountered, and in later versions of ODK, the responses are automatically saved after each question so that recorded data will never be lost.
Several survey coordinators raised concerns that they have not themselves been able to adapt the questionnaires. In order to adapt the electronic questionnaire, specific training on the software is needed. The technology partner is responsible for making necessary local adjustments to the electronic forms after consulting with the survey coordinator. This involves the use of more advanced software for building the questionnaires needed to fully take advantage of SENS, and to minimise errors during this step. Furthermore, SENS includes standardised questionnaires which should not be changed from country to country, so practically speaking this should not be an issue. However, survey coordinators would want to be able to do this part in-country.
Capacity of survey coordinator and teams
Survey coordinators with some level of computer literacy and management experience with SMART or SENS surveys generally handled the mobile data collection well. Having good knowledge of nutrition surveys in general and SENS specifically has allowed the coordinator to pay more attention to the extra learning aspect of introducing MDC. Inexperienced survey coordinators have generally taken longer to learn the technology, and have also handled ´stress situations´ with unexpected errors or obstacles related to MDC less well. Some survey coordinators mentioned that MDC requires higher technical skills by the enumerators; some struggle and make mistakes although increasingly many are already familiar with using mobile phones and even smartphones.
Battery life and electricity supply
The battery charge of older phones does not last the whole day in the field (resolved by carrying extra phones, additional batteries or portable power packs). However, with newer mobile phones this has not been a problem. Stable supply of electricity is needed for charging the phones overnight. In remote locations the electricity supply might be unstable or planned power cuts might occur in the office during the night and alternative charging options are needed.
Other challenges
There is increased workload in the evenings during data collection. Checking one phone could take 15 minutes, so six teams with two phones each would take three hours. Connecting all the phones to the server and uploading the questionnaires might take another 30-60 minutes. If the survey coordinators also wish to check the excel files and produce a SMART plausibility check, which is recommended, the survey coordinator ends up working for several hours in the evening after data collection. Whether this workload is ´additional´ or not depends on the procedures the survey coordinator is used to in the past. Many survey coordinators are doing daily data check of all questionnaires even when they use paper-based questionnaires and daily data entry of the anthropometric data in order to check the data and produce SMART plausibility checks. For survey coordinators who are used to following this procedure the MDC will actually decrease their daily workload with automatic transfer of data. So, although the daily workload was a disadvantage for some survey coordinators others found that checking of the questionnaires was easier.
Lessons learned
Training and support
Thorough training of survey coordinators and enumerators is crucial for successful implementation with MDC. Survey coordinators especially need in-depth training when conducting a SENS with MDC for the first time. All SENS survey coordinators have either participated in a one-week training prior to survey or received in-country on-the-job training for up to two weeks the first time they were exposed to MDC. After this, remote support will be provided as needed. There is close communication between field and UNHCR HQ, Regional Support Hub Nairobi (RSH) and CartONG. Together with CartONG, UNHCR has produced a number of guidance tools, including training videos, SOPs, practical step-by-step guidance notes, and training presentations. In addition to survey coordinators, IT staff from relevant countries have been trained by CartONG to be able to give in-country support when needed. Survey coordinators reported that this level of training provides a solid base of knowledge, and the current support procedures work well in case of problems.
Timely and efficient support is especially essential during survey preparations and the first few days of data collection, which is when problems might occur before the survey procedures have ‘settled in’. Sufficient time needs to be planned for the preparation phase to ensure support with local adjustments to standard questionnaires. Survey coordinators were happy with the current system in UNHCR with close communication with HQ, RSH or CartONG on any day of the week. However, options for more in-country autonomy should be explored.
Technical awareness among survey coordinator and team members
In order to implement a survey with MDC, the survey coordinator benefits from having some level of computer literacy and nutrition survey management experience. In the survey teams, at least one or two team members need to have some level of computer or smartphone literacy.
Equipment and data handling
Smartphones are better than tablets in the field as they weigh less and can be carried in the enumerator’s pocket if needed. The smartphones should preferably have a large screen and a slide-out keyboard in addition to the touch screen for ease of data entering, especially for enumerators who are not used to smartphones with a touch screen. To decrease the risk of losing valuable data, UNHCR recommends daily saving of back-ups of the data. Data are automatically saved on the smartphone’s SD card during data collection, and should be uploaded from smartphone to server every evening. In addition, it is recommended to extract copies of the server database on a daily basis to the computer and a memory stick as well.
Limitations to the review
A direct comparison between mobile and paper based methodologies would have been interesting and would have strengthened the review. Furthermore, experiences presented in the review have been gathered over a two-year period while the technology has been constantly evolving. Hence some feedback was already out-dated at the time of the review, and this feedback has not been included.
Conclusion
Mobile data collection in SENS surveys has resulted in high quality data collected, analysed and presented in a timely manner with advantageous user-friendliness in the field and cost savings for the operations in the long term. UNHCR Public Health Section will continue to expand the use of MDC in SENS where feasible, and encourages partners to work together on mobile data collection in nutrition surveys.
For more information, contact: hqphn@unhcr.org
1 The UNHCR SENS guidelines (www.sens.unhcr.org) are based on the internationally recognised SMART methods for survey design and anthropometric assessments, and adapted to the specific requirements of refugee settings.
2 Only mission reports from 2013 have been included as the CartONG mission reports mainly address technical aspects and the 2011 and 2012 mission reports are already outdated due to improvements and evolution in software.
3 Plausibility reports have been produced for 44 surveys in 9 countries, and where the surveys had been done prior to the November 2013 upgrade to the SMART plausibility checks, original data were used to create updated plausibility reports.
4 The plausibility check for anthropometry is a tool in SMART methodology/ ENA for SMART software to allow for evaluation of anthropometric data to provide an overall score for the survey quality.
5 The list of surveys included in review and their plausibility check results are available on request from the ENN (marie@ennonline.net) or the author (see contacts at the end of the article).
6 Given that everyone follows the standard procedure of creating daily data back-up on external hard drive or memory stick.
7 Budget is based on costs in Dollo Ado 2013.
8 To be included if this function is needed in addition to an incountry nutritionist from UNHCR or partner. Cost is based on medium level international consultancy.
9 One extra day should be expected when using MDC.
10 Assuming phones will be used for 3 years and based on following estimates: 15 phones x USD 200 / 3 years. If phones are used for other surveys or activities in addition to nutrition surveys, the cost will be even lower per survey.
11 Cost for supervisors, enumerators and data collection when using MDC is based on 20% more efficient data collection.