Impact evaluation of child caring practices project on stunting in Ethiopia

Summary of research¹

Location: Ethiopia

What we know already: Health, nutrition and water, sanitation and hygiene programmes impact on child nutrition. Evaluation of impact of different interventions is important for programme design and adaptation and is increasingly demanded by donors.

What this article adds: In this evaluation, the hygiene component of the WASH intervention appears to have a strong impact on stunting but only cautious conclusions are possible due to many limitations in research design. Quality operations evidencebased research needs integration at project design stage, adequate funding and academic partnerships.

The Legambo Child Caring Practices (CCP) project was set up in 2004 in Legambo, South Wollo Zone, Amhara Region, Ethiopia, by Save the Children UK (SC UK). The CCP had a clear research objective to measure the effectiveness of different interventions, singularly and in combination, in reducing linear growth retardation (stunting) through improved childcare practices in children 6-36 months.

Legambo is a large rural area with fairly low population density. Within the district there are 33 villages (communities), which are further divided into smaller rural gotts (sub-villages). The CCP was run in a small area of Legambo selected because the area had homogenous demographic, socioeconomic and livelihood profiles. The main livelihood in the project area is agricultural, with some agro-pastoralism. Agriculture is predominantly belg rain dependent (a short rainfall that occurs between February and May). However, since 1996, Legambo has endured a number of droughts, resulting in both loss of crops and livestock and increasing reliance on relief food. Emergency programmes have been implemented during crisis periods in Legambo, including the period of the CCP study, and have included general ration distributions to protect livelihoods and supplementary and therapeutic feeding programmes that aimed to prevent and treat acute malnutrition in children. Stunting rates, especially in the more mountainous areas of the district, remain very high with prevalence of stunting over 60 per cent in children less than 5 years.

Design

The CCP project was a controlled quasiexperimental (non-randomised) impact evaluation with a comparison group. Eleven contiguous villages, out of the total of 33, were purposefully assigned to receive one of four interventions: health, nutrition education, water, sanitation and hygiene (WASH), or integrated comprising all three interventions. A fifth group was used for comparison purposes and did not receive any of the SC UK interventions. All villages, however, were covered by the Government’s Productive Safety Net Programme (PSNP) initiated in 2004, a cash or grain transfer in return for public works for selected poorest household members. In the first year of the CCP intervention, to cover the gaps in the first year of the PSNP roll-out, cash was given by the Department for International Development (DFID) (UK).

Each intervention group consisted of two villages and the comparison group consisted of three villages. The health and integrated interventions were selected because they were located close to a functioning health centre. Villages assigned the WASH intervention were those specifically identified by local authorities (with community participation) as having poor access to water sources. The remaining five villages were assigned the nutrition and comparison interventions.

The interventions were mainly educational, although free drugs and primary healthcare services were provided for the health and integrated intervention areas, and materials for the construction of pit latrines and clean water sources were provided in WASH intervention areas. Community demonstration gardens were set up in the nutrition education intervention areas. Educational messages were delivered either door to door or through community centres by trained community animators for all interventions. The health and integrated groups received home-based messages 10 days every month. The nutrition and WASH groups received home-based messages five days every month, as well as further five days centrebased education sessions.

The targeting strategy for the CCP was to some extent ad hoc due to the nature of the different interventions as well as for ethical reasons, but the interventions were generally prioritised for PSNP beneficiary households with a child or children less than 2 years or with a pregnant or lactating woman. However, in the WASH intervention areas, the whole community was targeted and in the health intervention areas, free drugs were made available for all children less than 5 years. In addition, in all of the intervention areas, no one was excluded from the education sessions or door-to-door visits. Furthermore, the CCP was not the only health education project run in the same area. The final impact evaluation of the project was carried out by an independent consultant epidemiologist.

Findings, challenges and constraints

The main finding of the impact evaluation was that the WASH arm (specifically the hygiene component) demonstrated significant improvement in nutrition. Whilst these results potentially highlight the importance of WASH programmes to improve child growth, it is important to ascertain whether this result is real or artefact.

The Legambo study faced a number of challenges and constraints. A major challenge was that the project area suffered three consecutive years of drought (2007-2009), resulting in both crop failure and death of livestock. Furthermore, at the start of the project, wasting prevalence in children aged less than 5 years exceeded 15% in all areas.

There were also significant methodological issues for the evaluation due to the effects of confounding and bias on the internal validity of the study, particularly with regard to the lack of randomisation and power. WASH clusters were selected due to demand from district officials, and the villages themselves, whilst the other intervention/ comparison areas were selected by the SC UK project team. The WASH arm was different to the other intervention arms in that the same leader was present throughout the project. This individual was also dynamic and highly involved in community welfare and undoubtedly contributed to stronger community mobilisation and better ‘ownership’ of the intervention.

As is the case for many operations-based research projects, sample size and number of clusters were restricted to the study area. Moving out of the CCP study area would have meant moving into another livelihood zone making comparison more difficult, as well as placing extra burden on the project implementation. However, rather than use power analysis to determine sample size and attempt to minimise a type II error (to detect a difference when there is a difference), exhaustive sampling was used to maximise sample size. Sample sizes in the evaluation for children 6-36 months ranged from 919 in the comparison arm to 535 in the integrated arm; the sample size in the WASH arm was 784. Sample sizes decreased following the baseline survey for all arms (ranging from a reduction of 39.7% in the health villages to 6% in the WASH villages). Determining the sample size to adequately capture change in the prevalence of stunting using cluster sampling and where stunting prevalence is already high meant that in most cases, the sample size would probably have been too small to detect a medium-term effect in reduction in stunting prevalence; about 10 per cent if assuming a 2 per cent decrease per year. Post-evaluation power calculations, using the SAMPSI command in Stata 11 (StataCorp. LP, College Station, TX, USA), revealed all but the WASH intervention was underpowered (power =0.90), although possibly overestimated as this calculation does not take into consideration intra-cluster correlation (ICC)).

Cluster sampling involves the sampling of a predetermined number of community groups or clusters. The number of clusters required is in part dependent on the ICC coefficient. The ICC refers to communitylevel similarities, that is, where responses of individuals within a cluster tend to be more similar than those of individuals of different clusters. A high ICC increases the requirement for not only a larger sample size but a larger number of clusters. For sufficient sample size and valid statistical analysis to detect an intervention effect, the preferable number of clusters per intervention arm should be six or more with the recommendation that "studies with only a few (fewer than four) clusters per arm should generally be avoided". Too few clusters increase the risk of an imbalance in the intervention and comparison arms with respect to known risk factors for the outcome of interest. The more clusters per arm enable detection of a difference due to the intervention, if any, rather than merely community-level differences not related to the study. In the CCP, the number of clusters was two (three for the comparison).

Another important design issue concerned the issue of leakage, where nonbeneficiaries also benefited from the project’s inputs (spill over), or where any of the study group had access to other programmes (contamination) leading ultimately to the dilution of the project effects (due to a reduction in power type II error). Spill over can be minimised by incorporating ‘buffer’ zones between intervention areas to prevent spill over, or dilution, of interventions to other areas (intervention and comparison). This was not the case in this project, however, for a number of reasons: the area is vast and it was thought that there is not much movement between villages and there was only one health clinic that was suitable for delivery of free medication through the CCP project, so the health and integrated intervention villages were selected for logistic reasons. This last point negates to some extent the first point as it is very likely that beneficiaries from these two villages met at the health centre.

Another issue was that due to financial and timing constraints, the evaluation survey was not done at the same time of year as the baseline survey. However, since the main outcome being measured was stunting which is a more chronic manifestation of malnutrition, this was not considered a problem, even though it was not ideal for other types of data affected by seasonality such as childhood morbidity and dietary diversity.

Implementing an operations research project requires a great deal of commitment, under-standing and continuity. The project was designed and set up to be run for five years in a remote and poor part of the country. However, the difficult conditions the SC UK staff faced contributed to high staff turnover resulting in a lack of adequate documentation, poor continuity, with gaps in knowledge between new staff replacing old staff, and overall poor implementation commitments. New staff, without a comprehensive handover with clear objectives and implementation processes to date, had a limited ‘ownership’ of the project and did not fully understand what was expected of them. What could have helped was a wellwritten and visible conceptual framework, or at least a results framework, to be referred to by all staff members.

A further challenge was that chronic food insecurity undermined any potential benefits from improved knowledge and attitudes around complementary feeding and dietary diversity and undoubtedly explains the limited impact on nutrition status (similar to the comparison arm). With hindsight, the nutrition intervention for the CCP was inappropriate and could have been adapted as more evidence came to light; nutrition counselling alone has been shown to be ineffective without resources to purchase food. In the health and integrated arms, logistical constraints resulted in the delay of delivery of free drugs until half way through the project. This may in part explain the lack of significant impact in these areas, effectively shortening the time of the intervention, but does not help explain the observed increase in linear growth retardation.

Educational messages were delivered door to door by trained community animators for all interventions. However, nutrition and WASH messages were also delivered through centre-based training five days every month, which may have helped to establish greater community involvement through regular communication and meeting. Animators in the integrated arm were possibly overburdened with having to deliver more information and this may have led to poor quality delivery. However, the quality of the delivery of educational messages was not examined, so any difference in the quality of delivery and supervisory input, examples of implementation- related effect modification, was unknown, although animators were formally tested on a regular basis on their particular messages. Through focus group discussions with animators and women and men from intervention and comparison arms, it was clear that specific messages were not restricted to specific arms and that the type of knowledge on all topics was similar throughout the CCP project, including that for the comparison arm.

Quality of the anthropometric data was determined using a scoring system, using the Emergency Nutrition Assessment (ENA) (2008) software (ENA for SMART Beta version November 2008: SMART Methodology), which involved looking at indicators such as missing/flagged data, overall sex ratio, digit preference score for weight and height measurements, SDs, skewness and kurtosis. The main problem was with the height for age z score (HAZ) SD, which for both surveys, although slightly better for the endline survey, fell outside the acceptable range (1.10-1.30). This was largely due to problems in collecting the correct ages of the children. In both cleaned baseline and end line surveys, data were similar with respect to skewness and kurtosis but digit preferencing, particularly height measurements (0.0 and 0.5), was observed more in the baseline than endline survey.

A major concern with the baseline data was that child age was collected mainly from date of birth information on registration cards rather than recorded from mother’s recall of months using an ‘events’ calendar. In the evaluation survey, both methods to determine age were used and tested against each other. The result showed a correlation of 0.82 between the two different methods with differences ranging by as much as ±20 months in some cases. For the evaluation survey, age was analysed using the mother’s recall of age in months.

The finding that the only significant impact occurred in the WASH arm could mean different things. It could be that the intervention effect was large enough, given the sample size, to have had an impact or that there were important data quality issues that needed to be addressed. Selection bias that may have occurred within the WASH arm may also have contributed to this finding.

Recommendations

It is important that non-governmental organisations (NGOs) increase their evidence base on nutrition-specific and nutrition-sensitive interventions from more robust research studies in operational environments. Improved institutional awareness is required to develop epidemiologically robust studies as well as document results systematically. Operations evidencebased research is not only important for decision-making but needs to be built into programming proposals. To achieve this and to support better institutional memory, collaborations with academic institutions are becoming increasingly popular.

Project design needs to incorporate evaluation into the design from the start as it is important that a clear hypothesis is tested from the beginning using a causal framework. Causal frameworks can then be used as a backdrop to monitoring and evaluation to determine whether interventions are successful or failing in terms of process and programmes can be altered accordingly. Research projects and evaluations need adequate funding to allow for better study design, especially to ensure randomisation and appropriate power. Building an evaluation into a real-life programme is costly and there is increasing pressure from donors to carry out [robust] impact evaluations.

The fact that in this study the WASH intervention appeared to have a strong impact on stunting is very interesting and suggests an important, and immediate, public health opportunity to tackle stunting in food-insecure areas. However, given the constraints and challenges of this operations research project, repeat studies in different food security settings should be carried out. In the meantime, water and sanitation inputs should routinely be delivered through regular nutrition programmes.

¹Fenn. B (2012. Impact evaluation in field settings: experience from a complex NGO programme in Ethiopia, Journal of Development Effectiveness, DOI:10.1080/19439342.2012.725085