How to overcome data management challenges in research in crisis contexts

This session was moderated by Eva Leidman, of Centers for Disease Control and Prevention (CDC). Presentations were given by Mary Hodges of Helen Keller International (HKI), Karine Le Roch of Action Against Hunger (AAH) and Oleg Bilukha of CDC. Each presenter shared research they had undertaken, challenges experienced and results achieved.

Karine Le Roch (AAH) spoke about the follow-up of severely acutely malnourished children project (FUSAM), a psychosocial intervention in children aged 6-24 months in Saptari, Nepal.

Action Against Hunger conducted a randomised controlled trial (RCT) to assess the effectiveness of the addition of psychosocial support in an outpatient therapeutic programme (OTP). OTP sites were randomly assigned as either intervention sites (n=6) or control sites (n=6). The study was conducted in partnership with the Government of Nepal Ministry of Health (MoH), Child Health Division, Saptari District Public Health Office and the International Centre for Diarrhoeal Disease Research, Bangladesh (icddr’b) and supported by UK Aid, PATH, R2HC, Elrha and Wellcome Trust.

The study experienced significant challenges due to the 2015 earthquake, insecurity in 2015 and 2016 and floods in 2016. Although the original plan had been to monitor the study closely with real-time data capture and analysis, staff evacuations led to reliance on remote project management and loss of good visibility and quality control. Significant difficulties ensued in supervising record-keeping and physically extracting the data booklets from each site for analysis, as well as accessing homes for data collection due to floods and curfews.

In addition to these unexpected events, there was slow integration of psychosocial services into the healthcare system due to unfamiliarity with the approach and lack of recognition of the support required. Study sites were spread geographically, which created challenges in reaching beneficiaries and exacerbated pre-existing low treatment compliance, largely attributed to conflicting beliefs within the community.

Significant differences were found in child development scores between the group that received both psychosocial and nutritional support and the control group that had received only nutritional support. However, these differences were no longer significant at 11 months post-intervention and they never reached the level of children in the non-SAM group.

Mary Hodges (HKI) reported on a project to implement Essential Nutrition Actions (ENA) at the six-month point of contact for children at clinics before and during the Ebola crisis in Sierra Leone. The project was initiated prior to the Ebola outbreak in response to a government request to adapt the mass vitamin A supplementation (VAS) campaign conducted twice a year to a routine service at the six-month contact point (6MCP).

HKI established three groups to test three different approaches:

1. VAS integrated within expanded programme on immunisation (EPI) (at six months).
2. VAS integrated within EPI plus preparation of complementary food with the mother’s participation and then feeding of her infant by spoon.
3. As group two plus routine ‘quality’ confidential one-to-one counselling on family planning (FP) (provided in a private room with a dedicated health worker) and provision of short-term FP commodities as appropriate or referral for long-term provision.

The results showed that routine VAS at 6-8 months was 60 per cent, 72 per cent and 75 per cent in the three groups respectively; 96 per cent of children achieved full vaccination status at six months in all three groups. In Group 3, 62 per cent of mothers received routine counselling on family planning at the 6MCP. Of these, 70 per cent accepted family planning commodities, helping to fill the ‘contraceptive gap’. It was evident that the 6MCP for routine VAS enabled mothers to access routine family planning and boosted VAS coverage.

In response, HKI scaled up this integrated approach and trained health workers from 11 out of 14 districts before Ebola broke out.  Of 74 people trained in Kailahun district at the beginning of the Ebola outbreak, 16 died of the disease within six weeks.

HKI continued to support and monitor services throughout the emergency. It found that, of the clinics that had fully integrated the 6MCP, attendance fell significantly less during the Ebola crisis in integrated clinics (39 per cent) than in non-integrated clinics (59 per cent). The main reasons underlying this were the benefits perceived by mothers in preparing a complementary food to feed their infants, and, in terms of family planning, preventing pregnancy at a time when maternity services were scarce and pregnancy was regarded as risky.  

Since then HKI has been scaling up the 6MCP with the Ministry of Health to 340 of 1,281 health facilities nationwide and has plans and funding to complete the scale-up and expand to a 6MCP within four years.

Oleg Bilukha from CDC talked about conducting nutrition surveys in difficult places. UNICEF often coordinates surveys but has a focus on children and women. He posed several questions, including what happens when the elderly are the most vulnerable or when wasting prevalence is low? He also observed how infant and young child feeding (IYCF) activities are often the ‘knee-jerk’ reaction in response.

Recently in the Ukraine, the Nutrition Cluster carried out three investigations that required modification of cluster survey methods:

1. An IYCF survey among displaced women living outside the conflict zone

This survey focused on women with children under two years of age. Participants were difficult to find. Lists of women were sought that included addresses or telephone numbers, and methods to update the lists explored, to ensure that they were representative for selection for the study. Some women were registered with the government, but the government was not willing to share data; others were registered with non-governmental organisations (NGOs)/United Nations (UN) agency programmes. There was an anticipated challenge of potential high non-response. During the survey, 20 per cent non-response/telephone number not working was recorded; some women had left the area, while others were afraid to meet. The surveyors adopted a protocol of calling three times before moving on to the next name on the list and used a quota sample, continuing down the list until the sample size was achieved.

2. A representative survey of older people (over 60 years of age) in the conflict zone

The team recorded anthropometry (weight, MUAC, arm demi-span), diet diversity and food frequency, as well as chronic disease, access to medicines, disability and mental health. The team members followed cluster survey methodology as the available population data were reliable and there was little displacement at that time. Electoral precinct data enabled specification of boundaries and households. Three random start points were identified in each cluster and interviewers went from these points to select houses randomly. Approximately 40 to 50 per cent of households included older people; however around 50 per cent of households did not answer the door, while others refused to participate.

3. A survey of pregnant women living in areas around the conflict frontline, to include anaemia assessment

The main challenge with a survey of pregnant women is that they represent a small percentage of the population. It would require 15 to 20 household visits to find one pregnant woman, so a household visit methodology is not conducive for this type of study. The study was therefore designed around antenatal clinics. In Ukraine a woman is obliged to register as early as possible in pregnancy and attend for regular, compulsory check-ups, IYCF interventions, nutritional status assessment and access to humanitarian assistance. The survey team randomly selected clinics and days and surveyed all women who attended to achieve a semi-representative sample. Working through clinics also enabled them to use the last haemoglobin test result from the clinic data to assess anaemia, rather than collecting samples first-hand with a haemocue.

What would you do differently?

The panel was asked to reflect further on their research, what they might have done differently and what advice they would give to others when approaching research in crisis environments.

Karine acknowledged that they probably wouldn’t have conducted an RCT if they had known all the challenges in advance. This is a difficult study design in any context, but extremely so in insecure situations. Anticipating some of the potential obstacles and preparing a ‘plan B’ for the study design might have enabled them to adapt the design according to the contextual factors and constraints.

Mary noted that if they had increased the sample size of their study, they could have compared the three different groups in terms of growth of children and other outcomes. Their study was narrowly focused on one outcome, but with additional funds it could have furnished further interesting data on nutrition.

Oleg noted that when conducting surveys, the concern is often non-response and achieving sample size. It is important to try to anticipate these things from the start. For example, for the challenge of absenteeism, it is useful to consider which days of the week are market days, which hours people are working in the field and when are the peak agricultural working seasons to establish when people will be at home. Refusals are high in Ukraine, so this factor could be anticipated in that context and the survey design adapted accordingly. One way to assess the situation prior to embarking on a survey is to conduct a rapid test in advance; for example, call 50 to 100 people on the list and work out the non-response rate, or pre-test the survey instruments in the field to find out how many people refuse to participate. There are two types of bias of concern in surveys: measurement and selection. Measurement bias is often a manifestation of team quality and can be addressed early through supervision if one team is identifying more malnutrition than others. Where there is selection bias, it should be recognised and qualified in the text of the report. However, care should be taken not to exaggerate the relevance of bias. For example, wealth quintiles may not display huge differences in wasting; the data of the ‘middle-ground’ (e.g. clinic visitors) are useful and often reasonably representative.

Conclusion

All three presenters spoke about challenges faced when conducting research in unpredictable environments. A commonality between these reflections is the necessity of thorough preparedness, including scenario-building in advance to try to anticipate as many challenges and risks as possible. While some of these can be predicted (such as the high non-response rate/refusal to participate characteristic of populations in Ukraine and the absenteeism related to seasonal events or labour schedules), not all challenges can be foreseen, as was the case in   the Ebola outbreak in Sierra Leone and staff evacuation due to insecurity in Nepal. Study designs therefore require flexibility to adapt where necessary and feasible, a ‘plan B’ if possible and researchers need to maintain regular communication with donors/funders and stakeholders of the study, including the communities in which they are operating. This will facilitate anticipation and adaptation to ensure that useful and quality results can be obtained.