The cost of a knowledge silo: A systematic re-review of water, sanitation and hygiene interventions

Summary of research¹

By Michael Loevinsohn

Michael Loevinsohn is a research fellow at the Institute of Development Studies, UK. An ecologist and epidemiologist, he has worked for many years at the interface between environment, rural society and health, seeking to throw light on the dynamics of systems undergoing rapid change.

Location: Global

What we know: Progress in achieving broad access to water, sanitation and hygiene (WASH) has been slow, particularly for sanitation. A series of systematic reviews of WASH interventions has been conducted, aimed at informing investment and programmatic decisions: all have been from a health perspective and focused predominantly on interventions’ impact on diarrhoea.

What this article adds: The authors test the hypothesis that greater practical understanding of how WASH interventions work and can be made to work is gained when they are evaluated jointly from health and development perspectives. Twenty-seven studies from a systematic review of the impact of WASH interventions on diarrhoea morbidity (Waddington review) were re-reviewed from these two perspectives. In a first stage of review, the authors found evidence in the studies that interventions were more complex than the Waddington review indicated. They often involved more actions than acknowledged and resulted in substantial impacts beyond diarrhoea reduction; in many cases these were achieved by the unplanned actions of individuals, households or communities. In almost 45% of the studies, these additional actions and impacts would likely have affected the intervention’s impact on diarrhoea, including its sustainability and equity, suggesting that the Waddington review mis-estimated these impacts. In a second stage, the authors identify evidence in these studies of six additional impact pathways relating to intervention complexity, direct multiple benefits of interventions, unintended negative consequences, people’s actions favouring food and nutrition, diffusion of innovations within and among communities, and effects of local institutions.

While still tentative, these additional pathways suggest ways in which investments in WASH can more effectively support health and livelihood. The authors recommend at least one systematic review of literature on the different and multiple impacts of WASH interventions to put these suggestions on a firmer footing. Teams evaluating interventions should bring diverse perspectives to bear and have the flexibility to pursue evidence of other impact pathways that emerge. Strengthened research practice will maximise study insights.

Progress in achieving broad access to WASH has been slow, particularly for sanitation. In 2011, some 2.5 billion people were living without access to improved sanitation facilities, and 770 million people were not receiving their drinking water from improved water sources, according to the Global Annual Assessment of Sanitation and Drinking-Water (The GLAAS Report). The report linked these persistent gaps to the toll of diarrhoea, the second leading contributor to the global burden of disease (World Health Organization, 2010). However, understanding this problem has been undermined by the continuing disconnect between the world of health and disease transmission and the wider development world, which considers more diverse perspectives, interests, power and rights. This disconnect hampers comprehensive diagnosis of the problem and the mounting of more effective actions to address it.

The dominant concern with the health and particularly the diarrhoeal impacts of WASH interventions has been reinforced by a series of systematic reviews (SRs), considered the cornerstone of evidence-based policy, aimed at informing investment decisions in the WASH sector. However, their preponderant concern with diarrhoea has not been balanced with SRs of the significant non-health benefits of WASH interventions. These include more time for women and girls to pursue education, income generation, childcare and leisure; more water for kitchen gardens or other productive enterprises; livelihood and food security; saving money spent on private water supply; using these savings for food and other essentials; greater dignity and security with accessible, closed toilets; and increased attendance of girls at school when such facilities are available there. Thus, current SRs, taken together, provide an incomplete picture of the benefits of WASH interventions and limited insight into the contextual factors that can affect the achievement of impact.

Methods

The article’s objective is to test the hypothesis: Can greater practical understanding be gained of the overall impacts of WASH interventions and how they are achieved when examined jointly from health and development perspectives? The authors re-review the studies that were included in the most comprehensive of the recent SRs, the ‘Waddington review’, a Cochrane-standard systematic review of water, sanitation and hygiene interventions’ impact on child diarrhoea morbidity (Waddington & Snilstveit, 2009). The authors employ the methods of ‘realist review’ which sees interventions as being based on ‘programme theories’ which link their delivery to a desired outcome. It assumes that the outcomes actually produced by an intervention are influenced by many actors, not just the intervenors; that these actors respond to their social, political and natural contexts; and that outcomes – positive and negative – often go beyond those intended by the intervention’s designers (Pawson, Greenhalgh, Harvey et al, 2005).

In a first stage of review, the authors searched for evidence in the papers of other impact pathways operating in addition to those assumed in the papers’ programme theories and the Waddington review. The evidence was assessed with respect to three questions relating to these pathways and one overarching question that enabled the authors to test their hypothesis:

(1) Does the intervention involve substantially more actions than considered by the Waddington review?

(2) Are the intervention’s impacts substantially understated if only the diarrhoea morbidity outcome is considered?

(3) Are actions by individuals, households or communities substantially influencing the impacts experienced? and

(4) Would these other impacts and actions substantially affect the level, distribution or sustainability of the diarrhoea morbidity outcome?

Authors evaluated the evidence supporting an affirmative answer to each question: 1) No indication; 2) Possible (substantial additional evidence needed); 3) More than possible (some additional evidence needed); or 4) Likely (little or no additional evidence needed).

In a second stage of review, the authors looked for regularities in the impact pathways they had found in the studies and deliberated on the ‘middle range’ and broader theories that could explain them. They described these regularities and explanations in terms of the context in which they occur, the mechanisms at play and the outcomes they produce: what realist review refers to as ‘CMO configurations’ (Wong, Greenhalgh, Westhorp et al, 2013).

Of the 65 studies that Waddington et al reviewed, the authors excluded 38 on the basis of limited descriptions of context, very restricted space in which people could exercise agency, and for practical reasons (three only available as abstracts and two unavailable in English). The remaining 27 studies describe research conducted between 1982 and 2009. Six were classified by the Waddington review as randomised or cluster-randomised controlled trials and 19 as non-randomised controlled trials. Thirteen described hygiene, four water supply, three sanitation and seven multiple interventions.

Findings

The first stage of review judged that, at the ‘more than possible’ or ‘likely’ level, 22% of interventions involved substantially more actions than the SR’s label indicated; 37% resulted in substantial additional impacts beyond reduced diarrhoea morbidity; and unforeseen actions by individuals, households or communities substantially contributed to the impacts in 48% of studies. In 44%, it was judged that these additional impacts and actions would have substantially affected the intervention’s effect on diarrhoea morbidity, its level, social distribution or sustainability. These impacts and actions would likely be found to be more common in studies not so narrowly selected as were those the Waddington review drew on.

In the second stage of review, the authors identify six impact pathways each present in three or more studies and which were not considered by the original SR. The first relates to intervention complexity: agency staff add actions to an intervention in response to local circumstances; in some cases, interventions are sited where an earlier one had been implemented and build on its effects. In both cases, the intervention-as-experienced involves more than indicated by the label. An example is a Guatemalan program described by the Waddington review as “hygiene education” but which comprised 11 topics including nutrition (promotion of weaning foods, breastfeeding and agricultural diversification) and the recognition and treatment of diarrhoea, in addition to hygiene issues. The study only monitored hygiene behaviours and in relation to diarrhoea reduction (Torun, 1983). The intervention’s impact on diarrhoea may be overestimated if the additional or prior actions are ignored.

The second impact pathway relates to the direct multiple benefits of interventions. By doing one thing, interventions may affect different causes of ill-being. For example, a water supply intervention in Buenos Aires connected shantytown households to the urban system, increasing the quality and reliability of the water they could access, contributing thereby to reduced diarrhoea burdens, and saving them time and money spent fetching and procuring water (Galiani et al, 2008). The non-diarrhoea benefits will likely be valued in their own right and may increase people’s commitment to support and maintain the intervention, enhancing sustainability.

The third pathway relates to unintended negative consequences of interventions. Agencies make operational decisions on where and when to implement interventions affected by political influence, corruption and ease of access. The result may be an anti-poor distribution of benefits, exacerbating inequalities. In the Democratic Republic of Congo, diarrhoea incidence declined significantly in villages to which piped water was delivered compared with those relying on existing sources (Tonglet et al, 1992). Pipes were laid near main roads, apparently for ease during construction, which is also where higher socio-economic status households were concentrated. Thus, those least at risk of diarrhoea captured most benefits. The intervention’s impact was also overestimated because baseline differences between villages were not accounted for.

A fourth pathway was identified in several studies with evidence of unanticipated benefits from people using access to water for food and improved nutrition. Water supply interventions enable beneficiaries not only to avoid water-related diseases but also to irrigate kitchen gardens, throughout the year. Women started to do this as soon as the water arrived in Bangladesh rural water supply intervention (Hoque et al, 1996). In the Buenos Aires study described above, households diverted two-thirds of their savings from purchasing water to purchasing food (Galiani, Gonzalez-Rozada & Schargrodsky, 2008). The authors judged it possible that improved child nutrition resulted in both cases. This could have contributed to reduced diarrhoea morbidity but would be difficult to disentangle from the direct effect of increased access to water. Particularly in Buenos Aires, the reduction in diarrhoea, however it was produced, would have had a markedly pro-poor bias since these householders were among the most marginalised of the city’s residents.

The fifth impact pathway relates to the diffusion of innovations. Interventions are implemented in communities whose members share information though social networks which often reach into neighbouring communities. In the Guatemalan study discussed above (Torun, 1983), the proportion of control households in which at least half of the 27 hygiene behaviours promoted were deemed adequate increased 120%. This was much less than the 560% increase in treatment households but evidence of spontaneous communication among villagers. This ‘contamination’ likely led to an underestimation of the intervention’s impact on diarrhoea, which was calculated as the difference in morbidity between treatment and control households. But rather than considering this solely as an estimation problem, to be designed out or corrected for statistically, it is important to ask how people’s agency can be enlisted to enhance the reach and sustainability of interventions.

The sixth and final impact pathway concerns local institutions. The Waddington review and most of the studies it draws on focus on individuals and households. They largely ignore the adaptive capacity for self-governance of the communities in which interventions are implemented. A few studies however show how emergent and existing institutions can influence the spread, adaptation and retention of interventions. Pattanayak, Dickinson, Yang et al (2007) describe the development of local governance through norms against open defecation in Odisha, India. Experiential learning helps villagers to understand how they and their children are contaminated by open defecation, which often triggers disgust. Social persuasion adds to people’s motivation to progressively improve their latrines or toilets. The poorest are often assisted from within or outside the community to improve their facilities. Related processes are being supported on a broader scale through Community-Led Total Sanitation (CLTS), an approach now being implemented in more than 50 countries.

Conclusions

This re-review provides an enlarged view of WASH interventions and their contexts. There is evidence that other interventions, previous or concurrent, sometimes influence the field in which the intervention and the evaluation operate. Multiple impacts, positive or negative, unforeseen by the intervention’s designers, may be produced, affecting health and livelihood, many of them created or shaped by beneficiaries or by people beyond the intended reach of the intervention. These actions and effects suggest ways in which investment in WASH can better support health and livelihood. They also affect the Waddington review’s conclusions regarding the impact of WASH interventions on diarrhoea, suggesting that these need to be revisited.

Taking account of the limitations of this re-review and the imperative for more joined-up policy across sectors, the authors recommend that donors and commissioning organisations support one or possibly more SRs of literature on the different and multiple impacts of WASH interventions on health and livelihoods.

More broadly, this study contributes important experience to the continuing debate on appropriate methods to evaluate and synthesise evidence on complex interventions. In building evidence, there is an urgent need for studies that can take the measure of operational situations as they exist. Evaluation teams should have the skills and flexibility to pursue evidence of other impact pathways that emerges during research: too often study authors were left to speculate on an unexpected result. Good research practice, such as ensuring baseline data, would have helped to draw more insights from many of the studies reviewed. Better research costs more but a large price is now being paid by not being able to make proper sense of what happens in interventions.

Footnotes