The REFANI Project in Pakistan: adapting research to a multi-sectoral programme for impact measurement

Location: Pakistan

What we know: Undernutrition is prevalent in Pakistan, associated with systemic poverty and shortcomings in feeding practices and maternal care that are compounded by food crisis and climatic challenges which impact on food security, such as flooding.

What this article adds: The European Union-funded Women and children/infants Improved Nutrition in Sindh (EU-WINS) project is a multi-sector approach to address undernutrition in the province, involving access to maternal nutrition and health services, SAM treatment, use of nutritious foods by women and children, and evidence-based learning. A randomised controlled trial, as part of the DFID-funded REFANI Research Consortium, is investigating the nutritional impact and cost-effectiveness of cash and voucher-based food assistance in the context of the WINS programme. Four research arms are being explored, led by ENN in collaboration with ACF. Results will be available in early 2017 that will include cost-effectiveness. Valuable learning regarding operational research in challenging contexts is emerging.

Introduction

Action Against Hunger (AAH) has been working in Pakistan since 1979 during recurrent humanitarian situations. Programming is now governed by a full nutrition security-oriented strategy, strengthening the links between Nutrition, Food Security and Livelihoods (FSL), Water, Sanitation and Hygiene (WASH) and Disaster Risk Management, to achieve a higher level of impact for the population at risk of malnutrition. Since 2011, Action Against Hunger has been successfully implementing intensive, community-based management of acute malnutrition (CMAM) and nutrition-sensitive programming in Sindh (Thatta, Tando Muhammad Khan, Badin, Dadu and Jacobabad), addressing the nutrition emergency affecting the entire province. In 2013, AAH expanded its presence in Dadu through a four-year, nutrition security-focused development project funded by the EU – the WINS project – partnering with Merlin in Thatta and Sujawal districts and Save the Children in Shikarpur district.

Project context

Pakistan has some of the most serious undernutrition problems in South Asia, reflected in Global Acute Malnutrition (GAM) (17.5%), Severe Acute Malnutrition (SAM) (6.6%), underweight (40.5%) and stunting (49.8%) prevalence rates in children under five years old in Sindh province (NNS, 2011). Key attributable factors are reflected in data from Sindh province that include poor infant and young child feeding (IYCF) practices (40.5% breastfeeding initiation, 64.7% exclusive breastfeeding); low dietary diversity for children 6-23 months (4.6% minimum acceptable diet for 6-23 month olds); women’s micronutrient deficiency (60% anaemic, 49.5% zinc deficient); low birthweight (11.2% and 10.1% in north and south Pakistan respectively); poor access to safe drinking water, sanitation and health services (50% prevalence of diarrhoea in under fives, 21% of boreholes not protected (AAH, 2011)); and high food insecurity (72% of Sindh population in state of severe and moderate hunger) (NNS, 2011 unless otherwise stated).

Systemic poverty coupled with the lack of alternative income sources affects the capacity of households to cope with shocks and climate trends in a sustainable manner (Nutrition Causal Analysis (NCA), ACF, 2012). This is compounded by high incidence of illnesses due to poor access to water and sanitation infrastructures, and poor knowledge and practices regarding child feeding, hygiene and care. The already limited access to nutritious food among chronically poor households in Pakistan has further deteriorated in recent years due to the 2007-2008 global food crisis and continued high volatility in food prices. This impacts intra-household food security, determining dietary patterns in pregnancy and early childhood. As reflected earlier, the scarcity of iron, zinc, iodine and vitamin A in the diet and poor IYCF practices contribute significantly to child undernutrition in the country.

Project approach

An integrated, multi-sector approach was recommended by the 2012 ACF NCA to tackle the identified basic causes of the high prevalence of SAM in Sindh. The NCA was conducted in Thatta and Dadu districts in Sindh Province. In response and in line with the Pakistan Integrated Nutrition Strategy and the Poverty Reduction Strategy Paper, in-country partners developed the EU-WINS project. The programme’s objective is to strengthen the capacity of district and provincial governments to address high rates of undernutrition in Sindh Province. The four key results and specific activities/modalities of the multi-sector approach are summarised in Table 1. EU-WINS targets, projected and reached by Year 3, are included in Table 2.

Nutrition-sensitive FSL and WASH activities were designed to integrate the nutritional treatment and prevention components of the programme. For example, complementary feeding food vouchers, livestock vaccination interventions, construction and rehabilitation of water points often happen in common geographical areas and will always include the same BCC/community mobilisation component. Since the start of the programme in 2011, strong synergies have developed at community level, as all components also involve community-based organisations.

The long-term vision is that local communities, healthcare workers and nutrition and food security stakeholders at the district and provincial levels will benefit from the project as key activities become accessible through government mechanisms. Within the scope of EU-WINS, all health and nutrition-related activities are undertaken in close collaboration with the Department of Health and the Provincial Nutrition Cell (Sindh), as well as the People’s Primary Healthcare Initiative. Cash-based interventions are linked with the BISP in order to support improvement in policy and delivery of social safety net services. The close synergy of all aspects of the project with government policies and processes is the foundation for achieving the longer-term vision.

The focus of this article is on the operational research, which has been designed to look more closely at the nutritional impact of this type of intervention.

Challenges of measuring nutritional impact of interventions

ACF’s International Nutrition Security Policy, published in 2014, specifically calls for a multi-sector approach in order to achieve a long-term and sustainable impact on nutrition (through 12 guiding principles specifically; see separate article in this edition of Field Exchange). The Policy also promotes mixing curative and preventative measures for undernutrition and targeting its underlying causes, as well as other structural factors. However, there are challenges in circumventing the barriers which exist between nutrition-related sectors.

Overall monitoring and evaluation mechanisms were put in place for the various WINS activities (including baseline/endline and external evaluations), yet there was still need for robust impact measurement. The DFID-funded REFANI project (see Box 1), further supported in its intervention component with one additional research arm by ECHO, proved a unique opportunity to scientifically measure impact of the nutrition-sensitive aspect of EU-WINS in relation to the social safety net cash-based interventions.

The REFANI Project in Pakistan

The REFANI-Pakistan study is a collaboration between ENN, as the lead research partner, and Action Against Hunger, as the lead implementing partner. The study specifically looks at the effectiveness of different cash transfer programmes (CTPs) in reducing the risk of undernutrition in children under five years of age from households who also have access to the same elements of the programme covered by the EU-WINS activities, which in this case are nutrition and BCC programming. The specific research arms are detailed in Box 2. 

Box 2: The four REFANI-Pakistan research arms

The study specifically compares four different arms of intervention:

1) Households receiving the standard EU-WINS intervention (control group);

2) Households receiving the EU-WINS care and 1,500 rupees per month (funded by the EU);

3) Households receiving the EU-WINS care and 3,000 rupees per month (separately funded by ECHO); and

4) Households receiving the EU-WINS care and a monthly fresh food voucher worth 1,500 rupees per month to be exchanged for fresh foods at specified traders (funded by the EU).

The standard EU-WINS intervention includes SAM treatment at an outpatient therapeutic programme (OTP) site for all research sites, micronutrient supplementation, complementary food vouchers for OTP discharged children, IYCF services and integrated sensitisation sessions on six key behaviours for prevention and treatment of malnutrition.

The additional cash-based support is provided for six months; thus totalling 9,000 rupees for arm 2 and 4 and 18,000 rupees for arm 3 beneficiaries.

Primary and secondary outcome data being collected to compare levels of risk of undernutrition are*:

1. Primary outcomes – prevalence of wasting (as measured by weight-for-height Z-score (WHZ) < −2) and mean WHZ score in children;
2. Secondary outcome in children – cumulative incidence of wasting, prevalence of SAM (<−3 WHZ), low mid-upper arm circumference (MUAC) (<125 mm & < 115 mm), stunting (<−2 height-for-age Z-score) and morbidity (mainly diarrhoea, difficulty breathing/cough, malaria);
3. Secondary outcome in mothers and children – mean haemoglobin concentration (Hb g/dL) and prevalence of anaemia (children: Hb <11.0 g/dL; severe anaemia as Hb < 7.0 g/dL; non-pregnant women Hb <12.0 g/dL; pregnant women Hb <13.0 g/dL); and
4. Secondary outcome in mothers – prevalence of low body mass index (<16, <17, <18.5 kg/m2), low height (<145 cm) and low MUAC (<230 mm & <210 mm).

* Measurements are collected on all study households (household questionnaire) and participants (mother and child questionnaire) at month 0, 1, 2, 3, 4, 5, and 12 and are collected during the week following the receipt of each transfer.

The process of developing the REFANI project in Pakistan followed a number of steps and will provide useful ways of working and lessons learned for partnerships engaging in developing research as part of an operational programme already in place. It was agreed by partners that a cluster randomised controlled trial (cRCT) of CTPs, focusing on nutritional outcomes in Dadu, was possible. The CTP is designed to address a seasonal hunger gap, where ideally the transfer transitions from an emergency setting to a national/regional safety net approach, which would offer the research teams an opportunity to address common research questions related to CTPs, their methodology and sustainability.

The overall research structure includes six rounds of quantitative data collection, including baseline and endline surveys, one more data collection round after one year, a mixed-methods process evaluation, a qualitative research component, and a cost-effectiveness analysis study. The total planned sample size is 2,496 households from poor and very poor households located in 17 union councils from different livelihood zones (barrage and riverine). The particular study design allows for probability analysis of the impact of different cash transfer interventions on undernutrition, comparing three different interventions within the same population and region to a control group.

A cRCT study, especially in humanitarian settings, brings a number of challenges. The difficult context of the study (including conflict, security issues, religious holidays, weather conditions, and risk of emergency) can easily impact on implementation. The potential for flooding in the area was a major concern, so a contingency plan covering three levels of flooding (mild to severe) was built into the study. The REFANI project has successfully recruited a qualified study manager, deputy study manager and qualitative researcher under Action against Hunger protocols. The national team has been successful in ensuring high standards of data collection, with remote supervision from the ENN team and good collaboration between ENN and ACF.

The REFANI-Pakistan study has been approved both by the National Bioethics Committee in Pakistan and the Western International Review Board. The study has been registered as an International Standard Randomised Controlled Trial (ISRCTN10761532).

The study is mid-way completed; baseline and five consecutive months of data have been collected (see Table 2). Qualitative data collection in ongoing, adapted as results from the quantitative research comes in. First results will be available later in 2016. The aim is to publish at least three peer-reviewed publications, as well as disseminate findings at relevant conferences, stakeholder and other meetings. 

Challenges

The EU-WINS programme encountered a number of challenges in the operationalisation of a multi-sector approach, mainly as a result of vertical sector-based government systems and dramatically variable climate in the intervention area. The REFANI-Pakistan study has been able to effectively and quickly address some challenges that arose within the first year of implementation. These included:

Recruitment: There was some difficulty around recruiting and retaining female data collectors, which is essential for accessing female household members. However, the Action Against Hunger office in Pakistan managed to locate the necessary female enumerators. Action Against Hunger also focused on capacity-building of staff on research during the implementation in order to counter the difficulty in hiring female staff.

Timing: The study overlapped with Ramadan (June-July 2015) as well as extremely high temperatures (>50 degrees Celsius), which resulted in more time needed to cover all eligible households, since office hours are reduced during this period and the capacity of staff was lower. This resulted in the baseline data collection taking longer than expected; as a result of this delay, the ENN principal investigator decided to forgo the first month of the monthly data collection as this would have impacted on the timing of subsequent programme implementation. This is not expected to impact negatively on the study.

Access: Floods and tribal conflicts occurred in August 2015, which led to the replacement of some study areas and participants (in case of security risk).

Loss of study participants: This was noted during the second month of data collection, mainly within the control group, and reflected lack of incentive to participate. Efforts to reduce this were made through regular sensitisation carried out with the help of community mobilisers.

Opportunities for operation research development

There is a growing trend among organisations to use cash transfers in their food assistance programming to combat growing rates of undernutrition, especially during emergencies.

The process and results of the EU-WINS and REFANI interventions will be of use to many stakeholders, both in Pakistan and globally. There are lessons to be learned on the optimal design of operational research and on the nutritional impact of multi-sector interventions. These can be used to inform and provide evidence to support scale-up and change.

For more information, contact: Maureen Gallagher

References

ACF (2015). EU-WINS Midterm Report, August 2015.

ACF (2015 [2]). Research on Food Assistance for Nutritional Impact (REFANI): Literature Review. March 2015. 

ACF (2012). Nutrition Causal Analysis Thatta and Dadu district, Sindh Province. www.linknca.org

ACF (2011) WASH baseline survey; Dadu district; Sindh; Pakistan.

DFID (2011). Cash Transfers Literature Review. 

EU WINS (2011). Project proposal.

Manley, J., Gitter, S. & Slavchevska, V. (2012). How Effective are Cash Transfers at Improving Nutritional Status? A Rapid Evidence Assessment of Programmes’ Effects on Anthropometric Outcomes. World Development. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

National Nutrition Survey (2011). Pakistan Aga Khan University.