Improving the assessment and attribution of effects of development assistance for health

Location: Global

What we know: It is difficult to relate Overseas Development Assistance for health (DAH) and health outcomes; there are multiple funding resources, the contribution of health systems is typically underestimated and assumptions regarding the impact of external donor investment are overgenerous. Channelling funds through national delivery systems makes attribution to a specific funder difficult.

What this article adds: A set of agreements between donors on the principles underpinning assessment and reporting of the effect of DAH is suggested by the authors of a recent review. There are a number of promising initiatives on impact assessment (NOINE, 3ie, UN interagency groups, HIAs IHP+) to draw upon. Approaches should respond to country needs with appraisal of potential biased reporting; identify contextual factors affecting observed results; take into account confounding factors, such as economic growth or shocks; and adapt to different settings and needs. Lack of definition on what constitutes health systems strengthening activities and expenditures and methodological challenges reinforces the large knowledge gap in assessment of DAH effect. 

The authors of a recent paper in the Lancet suggest a set of agreements between donors on the principles underpinning assessment and reporting of the effect of Overseas Development Assistance for health (DAH).  Their suggestions stems from the methodological challenges in establishing a relation between Overseas Development Assistance for health (DAH) and health outcomes. The validity of the results presented by major funding agencies has been questioned because of weaknesses in models used to estimate outcomes, inappropriate counterfactuals and overgenerous assumptions of investment effects. Attribution of health outcomes to external investments by specific agencies has also proved to be challenging because of the multiplicity of funding sources. Furthermore, methods that attribute health effects to single interventions or specific funds do not reflect the vital contribution of health systems to health outcomes.

Most donors have committed to aid effectiveness principles underpinned by the Paris Declaration (2005) and the Accra Agenda for Action (2008), which stress national ownership and leadership, as well as harmonisation and alignment by donor agencies to national systems. Increasingly DAH is channelled through national delivery systems, making it difficult to attribute results to a specific funder. Yet most donor agencies continue to report their results.

The most commonly used approaches to assess and report the effect of DAH on health outcomes are tracking of trends in mortality, morbidity and coverage indicators; and estimation of lives saved and deaths averted. Methodological challenges include insufficient reliable data and poor disaggregation by age, sex, socioeconomic groups and geographical location. Assessment of the health effect of health systems strengthening (HSS) interventions with use of these measures is particularly challenging, because of the complex causal pathways through which these interventions are likely to have an effect. Furthermore, gaps in tracking input (e.g. HSS expenditures) and output indicators (e.g. measures of service availability, accessibility, and quality) limit the ability of studies to fully describe the sequence of inputs, activities, outputs and outcomes leading to an effect.

Donor agencies also use lives saved or deaths averted metrics to quantify the effect of DAH targeting disease-specific interventions, by comparison of health outcomes after the implementation of an intervention with the outcomes in similar geographical areas that did not receive the intervention during the same period; with outcomes in the same area(s) before the intervention was implemented; and with a hypothetical counterfactual scenario for the same area during the same period. Peer-reviewed models are used to estimate outcomes with these measures on the basis of estimated population coverage of interventions with well documented health outcomes. For example, the Global Fund has estimated lives saved by disease-specific interventions such as anti-retroviral therapy (ART) for AIDS and by insecticide-treated nets that reduce mortality.

Six points of interaction of DAH with national health systems have been identified: governance, finance, health workforce, health information systems, supply management systems and delivery of health services. However, there is no single framework to classify these interactions and research has identified several challenges, e.g. insufficient data, including for contextual factors that might have affected health effects and both the complexity of health system interventions and of the causal pathways leading to health effects.

Two main approaches used by donor agencies when attributing health effects to their funding are determination of plausible counterfactuals and establishment of exclusive attribution. When determining plausible counterfactuals, no intervention has frequently been used to attribute a health effect that could be linked to a specific source of funding, but determination of counterfactuals has been challenging, especially since an ideal control group was difficult to identify. The result attributed to a funder is sometimes assumed to be a proportion commensurate with the donor agency’s contribution to the overall investment for the intervention or country in question, although funding is often pooled at country level.

Promising initiatives on impact assessment       

There is as yet no consensus between key stakeholders on the strategies to improve and harmonise measurement approaches, but they could benefit from refinement and wider application of several promising approaches used by donors outlined below.

Beyond health, the Network of Networks for Impact Evaluation (NONIE) and the International Initiative for Impact Evaluation (3ie) support the use of study designs that enable analysis of interventions across the whole causal chain, from input to effects, and advocate the use of mixed methods for impact assessment to provide a more complete picture of effect.

In an attempt to strengthen country monitoring and evaluation platforms, the UN interagency groups (e.g. UN Evaluation Group, Inter-agency Group for Child Mortality Estimation, and Maternal Mortality Estimate Inter-agency Group) aim to develop new indicators, enhance capacity at country level to improve monitoring systems and strengthen data use, and harmonise work across partners on assessment and reporting of progress towards the Millennium Development Goals (MDGs).

Additionally, the Development Impact Evaluation Initiative by the World Bank provides technical assistance to projects and partners throughout the project cycle to ensure high-quality monitoring and effect evaluation.

One proposed approach, Health Impact Accounts (HIAs), aims to identify crucial value chains (such as infrastructure, transportation, procurement systems, etc) required for effective implementation of interventions (whether disease-specific or HSS interventions ) at country-level, and then apportions credit for effect to all investors, according to their relative contribution to different components of the value chain. 

Recent efforts, including those by International Health Partnership Plus (IHP+), have tried to improve and harmonise approaches used by donor agencies and partner countries to assess the effectiveness of DAH to reduce demands on weak health systems for donor-specific information; strengthen transparency and enhance comparability across different donor agencies; show the contribution of health systems to improvement of health; increase the efficiency of resource use; and provide reliable estimates of benefits of DAH. IHP+ encourages the documentation of results and of health aid effectiveness through in-depth individual country case studies. It also advocates joint annual reviews of country health strategies, including monitoring, evaluation, and review mechanisms, based on country monitoring platforms. However, the extent to which the IHP+ support to improved monitoring of national health strategies produces improved output information that could serve as a basis for further health impact assessment remains to be established. Joint annual reviews by several donors would improve harmonisation and mutual accountability by aligning donor assessment and reporting with national assessment and reporting mechanisms, thereby avoiding parallel donor-specific systems.

Proposed agreements between donors

A set of agreements between donors on the principles underpinning assessment and reporting of the effect of DAH are suggested:

First, approaches to assessment and reporting should mainly respond to country needs (including national governments and end beneficiaries) and emphasis mutual accountability. Country-owned monitoring and evaluation platforms need to be strengthened by external and domestic investments, and include robust additional studies. For example, a national evaluation platform that uses the district as the unit of design for natural, quasi-experimental or experimental studies and is based on continuous monitoring of core indicators can provide a rigorous comparison of effectiveness of interventions and different scale-up approaches.

Strengthening of monitoring and evaluation platforms should include a careful appraisal of potential incentives towards biased reporting, which exist in the relationship between partner countries and donors, especially in performance oriented initiatives such as GAVI’s (vaccine alliance) immunisation services support. Such perverse incentives might affect the accuracy of the data used to assess the effect of DAH, which in turn misinforms donors and their constituencies about their progress and might counteract efforts to strengthen country monitoring and evaluation platforms. Validated and standardised instruments and sampling frames, establishing systems for quality control and periodic surveys on a range of health topics that benchmark country-reported data, could help to address biases arising from perverse incentives.

Second, approaches to assess and report the health effects of DAH should identify the contextual factors both within and outside the health system that contribute to the observed results and might affect their wider applicability.

Photo 1

Third, there should be greater consistency and transparency in the selection of counterfactual scenarios, taking into account potential confounding by factors such as economic growth or conversely, economic shocks that can affect the determinants of health.

Fourth, approaches to assess and report the health effects of DAH should be adapted to different settings and needs. They should draw on a range of methods and designs dependent on what inference is likely to be made from them. They should be tested in both stable and fragile states, with coordinated approaches to monitoring sector performance that foster country ownership and reduce uncoordinated multiple donor programmes.

Substantial knowledge gaps in assessment of the effect of DAH exist, especially for investments in health systems. These knowledge gaps are accentuated by the absence of consensus about the definition of what constitutes HSS activities and expenditures.

Methodological challenges include double counting, underestimation of the contribution of health systems to health effect, and overlooking of system-wide effects of DAH. These challenges could undermine public confidence on the fidelity of the figures presented on the effectiveness of DAH. Exclusively attributing effect to disease-specific interventions supported by donor agencies could lead to channelling of greater financial support to disease-focused interventions than to HSS interventions, which could be particularly damaging in view of the plateauing in DAH since 2010. Some studies have suggested that disease-specific interventions have potential adverse effects on the health system, e.g. distorting national health priorities and diverting health workers from other responsibilities. However, there is paucity of empirical evidence to convincingly conclude whether the effects of disease-specific interventions on health systems have been negative or positive. Rigorous methods are needed to assess whether negative system-wide effects of DAH, including the potential skewing of expenditure (fungibility) are a substantial problem. Donor agencies need to develop their own skills and competencies and work with the research community to further address the methodological challenges inherent in assessment approaches.  

Improved measurement, enhanced comparability across different agencies and strengthened transparency in assessment of the effect of DAH should give credence to the results reported by funding agencies. To advance this agenda and appraise these new approaches in more depth is crucial, in view of the large amounts of funds spent, the growing recognition of the importance of HSS interventions to achieve global health goals and the political pressures on aid budgets at a time of economic austerity. 

Footnotes 

¹Ataya. N et al (2014). Improving the assessment and attribution of effects of development assistance for health. www.thelancet.com, vol 384, pp 2256-2259, Dec 20/27, 2014