Evaluation of mobile application to support the treatment of acutely malnourished children in Wajir county, Kenya
Summary of presentation1
By Emily Keane, Natalie Roschnik, Joanne Chui, Ibrahim Ahmed Osman and Hassan Mohamed Osman
Emily Keane is a Nutrition Advisor with Save the Children, with particular focus on supporting programmes delivering the community-based management of acute malnutrition. Emily has worked for ten years supporting nutrition and health programmes in Africa and Asia.
Natalie Roschnik is Senior Nutrition Advisor with Save the Children UK’s Programme Policy and Quality Department, leading a team of advisors to support UK-funded development nutrition programmes in Africa and Asia. Natalie has worked with Save the Children for nearly 20 years, supporting school health and nutrition research and programmes in Africa and Asia.
Joanne Chui has worked on nutrition programming in Kenya and is a registered dietitian. She holds an MSc in Nutrition for Global Health and a BSc in Dietetics.
Ibrahim Ahmed Osman is a Clinical Nutritionist with a BSc in nutrition and dietetics with information technology from Maseno University, Kenya. He currently works with Save the Children International as Reproductive Maternal and Newborn Health Officer in Wajir field office, Kenya.
Hassan Mohamed Osman is Area Programme Manager with Save the Children, managing both humanitarian and development programmes in Kenya. Hassan is a community health expert and management consultant, having worked in the humanitarian and development field for 11 years. Hassan is a registered public health nurse with a BSc in community health and development.
The mobile health application development and piloting was made possible thanks to the financial support of the Office of U.S. Foreign Disaster Assistance (OFDA). The project would not have been successful without the support and contribution of the Kenyan Ministry of Health and County Health Department in Wajir, World Vision (WV), Save the Children (SC) country office staff and participating health workers, community volunteers and caregivers. The project would like to thank the UK Department for International Development (DFID) and the Transform Nutrition Research Consortium for funding the mobile health application evaluation and research.
Location: Kenya
What we know: Integrated management of acute malnutrition (IMAM) is a proven approach to identify and treat acute malnutrition; however its effectiveness is limited if treatment protocols are not followed and data is unreliable.
What this article adds: A mobile health application (app) developed to help health workers deliver IMAM services was evaluated in 40 health facilities in Wajir, Kenya, over one year. The study found that the app reduced the number of reporting errors by 25 per cent; provided caseload and treatment data to decision-makers within 1.3 days of collection; increased the accuracy and reliability of treatment outcome data; and improved health workers’ adherence to the IMAM treatment protocol. The study found that effectiveness is dependent on health workers being well trained and having adequate time to manage cases and ongoing software support. Next steps to address challenges include simplification of protocols, working closer with Ministry of Health (MoH) on data management and exploring scale-up linked to existing health services.
Background
World Vision and Dimagi (an international technology development organisation) developed a mobile health app in 2013 to support the integrated management of acute malnutrition (IMAM). The application aimed to guide health workers through IMAM protocols and provide accurate and timely data for district health managers to respond to changes in caseloads and treatment outcomes, manage supplies and inform national statistics. The app was piloted in Chad, Kenya, Mali, Niger and Afghanistan between 2014 and 2016 by World Vision, International Medical Corps (IMC) and Save the Children (SC) (Frank et al, 2017). In January 2015 SC, through Transform Nutrition, launched a study to evaluate the impact of the IMAM app on the quality of IMAM treatment and data in 40 health facilities in Wajir, Kenya, including remote locations.
How the IMAM app works
The IMAM app is used on tablets or mobile phones and provides health workers with simple, step-by-step guidance on the assessment, treatment or referral of children visiting the IMAM programme. The app is built on the open source CommCare platform, which uses a touch swiping function to take health workers through IMAM steps. It reminds them of the treatment protocol, counselling messages and return dates and calculates z-scores and numbers of ready-to-use therapeutic food (RUTF) sachets needed. It also records each child’s information, making child follow-up and defaulter tracing easier. Data are regularly uploaded to the ‘cloud’, which enables the provision of live and accurate data for county-level management.
Methods
Forty health facilities from three sub-counties in Wajir were selected and randomly allocated to the intervention and control groups (See Figure 1). The 20 intervention facilities received a tablet with the IMAM app and 31 health workers from these facilities were trained over three days to use the tablet. Routine child-level IMAM data during the one year prior to the study were collected retrospectively from paper registers in all 40 facilities (N=1,200) to estimate accuracy of reporting and similarity between the intervention and control groups. After the app was introduced, the same paper register data was collected from the 20 control health facilities (N=903) and compared with the equivalent data generated by the app in the 20 intervention facilities (N=668) over the same one-year period. Direct observations of health workers providing IMAM services were carried out in both groups over three months to assess adherence to IMAM protocols.
Figure 1: Description of the mobile app intervention
Preliminary results
The app reduced data errors
Only 73 per cent of data from the 20 control health facility registers was usable (from the intervention period). The rest was either missing, unreadable or implausible. Exit data in particular was lacking; only 46 per cent of exit data and 37 per cent of data on exit outcome was usable. By contrast, 100 per cent of patient data from the 20 intervention health facilities via the IMAM app was usable and available. There were no missing or unreadable data because the app identified gaps and errors and prompted the health workers to correct and complete them while they were assessing the child.
The app provided ‘live’ data to decision-makers
In the control facilties using a paper-based system, it took several weeks for data on child treatment to be available to decision-makers. At the end of each month, health workers compiled summary reports of aggregated data from individual treatment data from the registers and physically sent the report to the sub-county office, where data officers entered the data into the national online Health Management Information System (HMIS). It typically takes 40 days from the compilation of monthly reports to data becoming available. It was also common for reports and data to be lost in the transfer process. When the data was entered into the HMIS it could be accessed and viewed at sub-county, county and national-level management teams.
App data took on average 1.3 days (0.4 – 9 days) to become available in the HMIS. When health workers saw a patient, data were collected and stored on the local memory of the tablet in real time. When the tablet had access to the internet these data were then automatically synced and uploaded onto the cloud server, where the data could immediately be viewed by management teams at all levels. The variation in time for data to be uploaded between health facilities reflects variations in internet signal strength between locations.
The app improves adherence to the IMAM protocol
Ninety-nine per cent of MUAC measurements observed in the intervention group were conducted correctly, compared with 84 per cent in the control group. MUAC measurement errors in both groups included misplacing the MUAC tape along the arm, lifting the arm while measuring and pulling the tape too tight or too loose, which can lead to misdiagnosis. The app has visual prompts and instructions to remind the health worker how to take measurements accurately, which helped to reduce errors.
Across the 18 recommended medical checks that should be performed to identify other health problems, 40 per cent of tests were carried out in the intervention group compared to 11 per cent in the control group. Only 13 per cent of children observed in the control facilities underwent an appetite test compared to 39 per cent in the intervention facilities. Although the app increased the likelihood that IMAM steps are followed, many protocols were still not carried out in the intervention group, despite the app prompting health workers to do so. Focus group discussions with health workers showed that time constraints were the main reason for skipping medical checks, as well having no dedicated space within health facilities to conduct the appetite test in privacy (desired by many caregivers).
Cure rates may be overestimated in the paper-based system
As the individual child data from the registers had many issues with missing and unusable data, data that were entered into the district HMIS from aggregated summary reports were analysed and compared to data obtained from the app. Reports from the 40 study facilities prior to tablets being introduced stated 95 per cent cure rates, suggesting that the majority of children who were admitted to the IMAM programme were cured. The app data, however, tells a very different story. In the 20 intervention facilities, only 56 per cent of children were cured and 42 per cent had defaulted2 compared to 84 per cent and 5 per cent respectively in the control facilities over the same period (See Figure 2).
Defaulters are often linked to operational issues such as RUTF stockouts at the health facility or long distances between home and health facility, which reduce caregivers’ willingness and ability to attend the IMAM service. Baseline assessment showed no significant difference between intervention and control facilities in these factors. The vast difference in cure rates suggests therefore that cure rates may be over-reported by health workers using the paper-based system. Assessment of individual treatment data from the registers showed that health workers rarely classified children as defaulters, even if it appeared that they had missed three consecutive visits. This may be because the quality of individual treatment data was very difficult to interpret, given that so much data were missing and the lack of consistency in how data were recorded. It is also very difficult to summarise data from individual treatment records into monthly summary reports, which may also contribute to errors in the reporting of defaulters.
Figure 2: Proportion of children cured, died, defaulted, not cured or referred on exit from the IMAM programme
Many children were misdiagnosed in the paper-based system
A reanalysis of height, weight and MUAC measurements found that weight-for-height Z-score (WHZ) is the most complex method for health workers in assessing a child, as it relies on the health worker first correctly measuring the child’s height and weight and then identifying the WHZ from a three-way table. A reanalysis of all WHZ scores from the raw weight and height data in the paper registers found that 28 per cent of WHZ scores were incorrectly calculated in control facilities. A reanalysis of children’s admission weight, height and MUAC data found that 17 per cent of children were wrongly admitted to the IMAM treatment programme (they were not severely malnourished).
The mobile app automatically calculates the WHZ score based on the entered height and weight of the child, therefore reducing errors in the classification of WHZ and eliminating incorrect admissions. Data using the app were found to be 99.3 per cent accurate (with the small number of errors being due to issues with app functionality early on, which were later corrected).
Operational challenges
Even when using the app, health workers continue to miss key IMAM steps, particularly relating to the medical checks and appetite test. This is because the treatment protocol takes time and health workers working in remote, low-resource health facilities with high caseloads are forced to take shortcuts.
The effectiveness of the app relies on health workers being properly trained and supported and the app functioning well. This requires ongoing support from software developers for software improvements and bug fixing. In instances where issues with bugs were discovered but not quickly resolved, it impacted on health workers’ motivation and willingness to use the tablet and app. Due to the rapid turnover and movement of health workers between health facilities, ongoing refresher training and supportive supervision during field visits were essential.
Conclusions
The paper-based systems did not provide reliable data in good time to allow decision-makers to respond to surges in caseloads and address operational challenges. The IMAM mobile app improved adherence to the IMAM protocol in intervention facilities and provided real-time, accurate data to decision-makers.
The IMAM app has the potential to dramatically improve the speed of response to surges in caseloads, identify and resolve operational bottlenecks, and improve coverage, quality of care and treatment effectiveness. However, its effectiveness is dependent on health workers being well trained and having adequate time to manage cases, and ongoing software support being available to ensure that the app functions properly.
Next steps
A range of next steps are currently being explored for use of the mobile app, pending further funding, including:
- Simplification of protocol: the experience of implementing the research showed operational challenges for health workers working in busy clinics with high caseloads to deliver the treatment protocol in its entirety and to a high quality, pointing towards the need for simplifications in the treatment protocol.
- Expansion of the app to other services, or linkages to other mHealth solutions: as the management of acute malnutrition is a service that is increasingly integrated into health systems, there are opportunities to add or link IMAM app components to other mHealth solutions for other services that health workers deliver; such as integrated management of childhood illness, antenatal care and postnatal care.
- Enhanced use of data by decision-makers: there are opportunities to work more closely with the ministry of health staff to analyse and respond to the data that is available in ‘real time’, particularly responding with resources (staff and supplies) to surges in caseloads, especially in drought/emergency contexts.
- Scale up within the health system: there are a number of outstanding operational and research questions relating to scaling up mHealth interventions within a health system that need to be answered, including: costing and cost effectiveness of the intervention; how to link data directly into the national reporting system (without the need for additional data entry); linkages with an expanded set of services; and data protection and cyber security issues.
Endnotes
1Presentation at the ACF Research for Nutrition Conference, Pavillon de L’Eau, 13th November, 2017.
2Defaulters are defined here as children who have not attended IMAM services for three consecutive weekly visits.
References
Frank T, Keane E, Roschnik N, Emary C, O’Leary M and Snyder L. (2017). Developing a mobile health app to manage acute malnutrition: a five-country experience. Field Exchange 54, February 2017. p7. www.ennonline.net/fex/54/mobilehealthapp