Measuring the Impact of Humanitarian Aid

Summary of published paper¹

A recent HPG Briefing Paper reports on research into how the humanitarian community measures and analyses the impact of humanitarian assistance. The study is based on a review of the published and grey literature within the humanitarian sector and more broadly, interviews with aid agency staff and two commissioned papers covering impact measurement in the food and nutrition and health sectors.

The review concludes that the humanitarian system has been poor at analysing impact though promising approaches are now starting to be developed. It states that a major constraint has been the lack of an accepted definition of impact within the humanitarian sector and that the definitions current within the development field may not fully capture the particular nature of humanitarian work. In particular, the concept of positive change is central in developmental definitions of impact, but in humanitarian aid the aim is often to avert negative change (for example to prevent famine). The review also points out that analysing the impact of a humanitarian intervention is not straightforward, particularly in the dynamic and chaotic environments of complex emergencies. The difficulties of the operating environment, the need to act quickly in situations of immediate crisis, an organisational culture that values action over analysis and the fact that there is little consensus around the core objectives of humanitarian aid - all make analysing impact difficult. Techniques that are standard in the social science community, such as the use of control groups, are not widely used, and humanitarian practitioners tend to lack the skills needed to gather and interpret information.

Key findings of the research are as follows;

Moving beyond the project level

• Concern for the impact of humanitarian aid should not be narrowly restricted to the project level. There is a need for greater investment in research, sector and system-wide evaluations that can ask difficult and important questions about; the overall impact and coverage of the humanitarian enterprise, roles and responsibilities for humanitarian outcomes, and the broader political dimensions within which the humanitarian system operates.
• Project-based approaches that focus on determining the impact of a particular inter vention through a causal pathway from inputs to impact should be complemented by approaches that start with changes in people's lives and that situate change in the broader external environment.
• Questions of impact should not be limited to the evaluation process. In the humanitarian sphere, a concern with change in the short term implies a need for impact to be consid ered in ongoing monitoring processes, and through techniques such as real-time evaluation.

Measuring impact: skills, capacity and resources

• Impact in any context is difficult to measure and attribute; this difficulty is exacerbated in the dynamic and chaotic environments of complex emergencies. This does not mean, however that it is impossible, and greater efforts could be made.
• The humanitarian system often lacks the skills and capacity to successfully measure or analyse impact. Greater investment there fore needs to be made in human resources and research and evaluation capacity if the desire to focus more on results is to be realised.

Measuring impact: science and participation

• The humanitarian system has been consistently poor at ensuring the participation of affected populations. Much could be learnt from innovations in participatory approaches in the development sphere, and possibly from customer-focused approaches in the private sphere.
• There is a place for both art and science in impact measurement: scientific, analytical and participatory approaches can often be complementary.

Indicators and objectives

• Analysis and impact could be improved through greater clarity about the objectives of humanitarian assistance, and by more consistent assessment of needs.
• Process indicators can sometimes be used as proxies for impact when there is strong evidence of a link between the action being monitored and an expected impact. There is a need for greater investment in strengthening the evidence base for how activities, such as supplementary feeding or support to health clinics, relate to humanitarian out comes such as reductions in mortality or malnutrition.

Results-based management; potential and dangers

• Results based management systems (focus ing on outcomes and impact rather than outputs and activities) are being introduced in a number of humanitarian organisations. However it is too early to say whether they will significantly improve the measurement and analysis of impact. Experience from elsewhere suggests that there will be a need for caution; in particular, measurement may remain largely focused on outputs and not impact.
• The increased focus on results which comes with such systems carries a risk that the harder-to-measure aspects of humanitarian action such as protection could be neglected.

The way forward

The study suggests that sufficient and appropriate tools and methods exist to provide reliable analysis of the impact of humanitarian aid whatever the context. It is the appropriate use and adaptation of these tools to the particular context and constraints that is lacking as a consequence of insufficient investment in skills and capacity development within the humanitarian sector. The study suggests that addressing this gap would have implications beyond the improved practice of impact assessment but would also lead to clearer objectives for aid, more robust risk and needs assessments, better research into what works and what doesn't and greater emphasis on community participation.

¹Hofmann. Charles-Antoinne et al (2004): Measuring the impact of humanitarian aid. A review of current practice. HPG Research briefing, no 15, June 2004