Methodological challenges for operational research in the humanitarian context

Challenges related to RCTs

RCT methodology is often used for nutrition research in humanitarian contexts and is acknowledged as the ‘gold standard’ in terms of quality of evidence. Nevertheless, implementing an RCT in a humanitarian crisis context poses many challenges.

Humanitarian programmes operate within strict and often circumscribed and short-term, donor-driven timeframes, whereas scientific research often requires a lengthier period for implementation.

In some contexts, RCTs are not feasible: a well-designed observational study may be better than a poorly implemented RCT. Nevertheless, RCTs are feasible in many humanitarian contexts, even in volatile situations. Critically, researchers must remain mindful of the context and ‘expect the unexpected’, which typically impacts the length and cost of the research. For example, there may be security issues during randomisation, a pipeline breakdown in nutritional products, or sudden difficulty in population access. An acute conflict or natural disaster may occur on top of a chronic emergency situation. From a research perspective, with the right attitude and preparedness, these are manageable situations and sometimes research consequences may be limited, e.g. an event may impact on the control and intervention groups equally. However, researchers must often engage in lengthy discussions with operations teams concerned with the challenges and risks of implementing research in volatile contexts. In a multi-sector research project, these conversations are multiplied by the number of sectors involved. Good communication with all parties regarding the project’s purpose and objectives can help minimise adverse impacts.

RCTs are important but can be a ‘risky’ investment for non-governmental organisations (NGOs). From the outset, funding can limit the scope of investigation, such as limiting study arms to a control and one intervention. Where no impact is found in a trial, programme teams may be disheartened by the lack of a clear ‘positive’ or immediately actionable result. This is particularly true when extraordinary efforts by the programme team have been required to carry out an RCT in challenging contexts. To help address this, a mixed-methods approach, including other elements such as qualitative research, monthly surveillance and process evaluations alongside RCTs, can help ensure practical learning takes place that can be applied even if the headline results are less ‘exciting’.

Conducting experimental studies on nutrition requires investment in robust context analysis to document impact pathways or process evaluations, given that nutrition is context-specific, multi-sectorial and related to seasonality. It is important to implement qualitative methodologies, capture seasonal features and take the necessary time for full analysis of all monitored indicators, involving experts from the field, operations and academia.

Funding challenges

Funding timing and flexibility is an ongoing challenge in humanitarian contexts. Programme and research funding sources are typically distinct, with different timeframes and donor requirements. For example, the REFANI project research element was funded by DFID, while programme activities were funded by ECHO. Programme funding was not fully committed until several months after the research funding was secured and study preparations needed to start, including hiring staff. This required both the research and programme partners to plan flexibly and to spend research funds before the trial was 100% guaranteed to happen. Single-source funding that covers both research activities and the intervention(s) would allow better coordination of activities. It would also facilitate study designs that could answer the most relevant questions, rather than just those permitted by the current programme design.

Managing the unexpected in research requires donor flexibility. Delays mean extended deadlines are often required. Having a financial envelope for unplanned events is key in the implementation phase, empowering researchers with the flexibility and reactivity needed to respond quickly to change.

Challenges with human resources

Establishing international and national staff teams is not straightforward. Cultural sensitivity is important but is seldom taken into account or given the priority it needs. Capacity development of national staff is often a ‘tick-box’ exercise without proper investment; agencies and donors need to take it seriously to have an impact. Also, we often fail to appreciate that capacity development is two-way; national staff have a wealth of contextual knowledge that is critical to the implementation of field research.

Experiences from the REFANI Project in Niger and ACF research have found that recruiting national staff with the necessary research skills and experience is challenging. This has led to operational/support staff not being hired as planned and overburdening some existing staff, who became responsible for both research and programme activities.

A number of lessons were learned. First, it is important to be realistic when recruiting research staff where capacity is known to be limited, i.e. appoint international staff or include capacity-building for national staff if time and budget permit. Secondly, recruit adequate numbers of extra personnel to support the implementation of research activities. Thirdly, having a part-time database manager is invaluable, given the need for remote data management in many contexts. Finally, paying casual-hire enumerators bonuses for undertaking all data collection rounds (e.g. baseline, mid-term and endline studies) saves time and money, helps build capacity (e.g. through refresher training) and supports data quality. Staff turnover at field level is common; for REFANI, the oversight and continuity of an HQ-based research coordinator was a successful element in the research implementation.

Monitoring and indicators

Data collection

During the Concern research studies, many personnel hired as study staff had different expectations about the quality and types of data collection required, given their experiences of working for NGOs rather than academic institutions (which have more rigorous standards for data quality). External parameters such as deteriorating security can constrain access for international research staff, which limits opportunities for researchers to provide support and oversight. For example, during the UCL REFANI study, data collection was undertaken with increasingly limited access for UCL’s international research staff due to deteriorating security. To compensate, the team conducted relatively lengthy trainings and hired consultants to provide support as trainers and supervisors. Tablets were used for data collection which posed some practical challenges but at the same time facilitated real-time access to recently collected data at office level in Tahoua, Niamey and London. This proved invaluable for remote data management.

Indicators

Having a comprehensive theory of change regarding nutrition-related issues is key to understanding research findings. To be truly effective, research needs to go beyond simple quantitative frequencies to learn which programme components contribute to improved nutrition outcomes; how and why; and whether the results are generalisable. Rigorous quantitative methods, complemented by qualitative research, are necessary to answer these questions. SPRING has successfully used mixed-methods research in several countries, but finding time and resources to fully analyse data, especially qualitative data, remains challenging.

Project indicators, while important for accountability, have limitations: they may under-emphasise or fail to capture key factors which can have direct impact and could benefit decision-making. Outcome indicators are unique to each country, making cross-country comparisons difficult. At country-level, however, they do allow for tailoring research and evaluation to local needs. This can help build ownership and investment, since the indicators measure what is directly relevant to countries.

Monitoring

Research challenges usually relate to project management, particularly regarding data collection and monitoring. Quality data collection and management can be lacking in the nutrition and health sectors. It is therefore essential to invest in good monitoring and evaluation systems and link this with observational research, resulting in strong data being embedded in programmes. This requires working with operation teams, better use of the field data documented routinely or through audit, and opening discussions around this. Investing in a good MEAL (Monitoring, Evaluation, Accountability, and Learning) system is key. Action Against Hunger, for example, is working on a tool (NEAP: Nutritional Evaluation Assessment; http://bit.ly/2ktDtuW) to improve the assessment of nutritional outcomes in programmes at field level.

Ethical dilemmas

Control groups present an ethical dilemma, particularly in resource-constrained settings. The idea of targeting an intervention based on random selection rather than on need presents operational agencies with a real quandary. In Concern’s experience, ways were found to leverage a control or comparison group while maintaining what was considered an acceptable level of accountability to the communities with which they work.

In Chad, the same intervention was provided to the control group, albeit three years later. In Niger, a comparison group – not a full control group – was used, whereby both study arms received the same total amount of cash but over six months versus four months. In Somalia, the control group (an internally displaced persons (IDP) camp close to the IDP camp receiving the cash intervention) was not randomly assigned. It was identified after it had not been prioritised to receive cash following an independent targeting process. The study team had also devised an alternative study design and analysis plan in the event that the control camp did become a target for cash; as it turned out, this did not happen during the three-month study period.

Research data sharing is also a challenge; a fundamental question is whether data should be accessible to all. Open data is transforming research methods and data treatment.

Multi-sector approaches and partnerships and international partnerships

The implementation of longitudinal studies increasingly requires multidisciplinary approaches and the creation of international (and national) spaces to enable the necessary connections and partnerships. Formal structures where NGOs and academics convene are rare. Creating a formal forum for partnerships to develop can be critically important. Even where financial and time constraints limit this type of collaboration, we must try to capitalise more on sharing past experiences and greater investment in multi-sector approaches.

Conclusion

Detailing the operational challenges of conducting research in humanitarian contexts is important. It is also important to invest in longitudinal and multidisciplinary studies to help understand the causes of undernutrition and the means to manage it. These studies should be complemented by in-depth observational studies. Such ambitious projects should be managed by consortiums of NGOs and academics, supported by donors willing to invest and innovate.