Evaluation of MSF Holland mission in Afghanistan

Kandahar, Afghanistan, 2002

Summary of Evaluation¹

Last year, MSFH Holland (MSFH) commissioned an evaluation of their activities in Afghanistan between May 2000 and May 2002. The evaluation set out to:

1. understand MSFH's role in Afghanistan and how MSFH adapted nutritional and health programmes in a rapidly changing political and operational context in the cultural setting of Afghanistan
2. examine how MSFH used its proximity in the region to witness violations of humanitarian principles and law and how MSFH positioned itself in its advocacy work.

This summary of the evaluation findings concentrates of the first of these objectives.

Methods

Two external consultants with backgrounds in nutrition and health undertook the evaluation. Field work took place in Kabul, Herat and Kandahar where interviews were carried out with MSFH project staff , personnel from INGOs, local NGOs, UN agencies, and MoPH staff. The consultants also attended a regional workshop in Ashgabad and a medical coordination day in Amsterdam. Phone interviews were conducted with staff in Quetta, Pakistan while a number of MSFH staff no longer involved in the programme were interviewed in London and Amsterdam.

Limitations of the evaluation

Limitations included lack of a central repository for reports and information on the programme. Much information was lost or misplaced during the evacuation. The high staff turnover meant that most MSFH international staff connected to the programme over the time period in question were no longer in Afghanistan at the time of the field visit nor were many key personnel from other INGOs and UN agencies present during the period under question.

Findings

Within a context of the appalling health status of women and children and poor access to basic health facilities in Afghanistan, the focus of MSFH's operational activities was affected by 'what the Taliban would allow', security , a developing drought induced food crisis, lack of reliable information and how MSFH perceived its role (i.e. whether it should focus on needs arising out of crisis, human rights monitoring, or health needs of the most vulnerable). Over the time period in question the focus changed from improving health care access in rural areas to focusing on health needs in cities and eventually to safe motherhood in remote areas.

MSFH's primary analysis of the impact of the drought was in keeping with many other agencies, i.e. that there was a developing food crisis, which required emergency food and nutrition interventions. Although MSFH's perception was that a General Food Distribution (GFD) was the most appropriate response, staff were conflicted about whether MSFH should take on an implementing role. Ultimately the decision was taken (based on many factors and previous experience) that MSFH would not 'do' GFDs. MSFH therefore focused on advocating for improved GFD and later implemented Blanket Food Distributions (BFDs) as stop gap measures. Although much of the advocacy work was important, MSFH were largely unable to influence or affect the mobilisation of GFDs in areas where they were implementing BFDs.

MSFH implemented traditional emergency feeding programmes in a number of open settings partially to collect data which would be useful in advocacy but also to prevent severe malnutrition. However, there was limited evidence of 'alarming' levels of malnutrition and traditional feeding programmes were found to be inappropriate to the cultural setting so that eventually these were either discontinued or adapted (e.g. mobile SFP).

Thorough investigation of outcomes in terms of malnutrition and mortality was not rigorously undertaken by MSFH. However where other agencies managed to undertake thorough representative investigation mortality was found to be high and related to disease while levels of malnutrition were lower than expected in the context of a food crisis.

There was a considerable need within Afghanistan for credible information on how the drought was impacting the health and nutrition of the population. As a health agency with historical capacity in nutrition the evaluators suggested that MSFH could have capitalised more on their comparative advantage in the health sector by analysing and highlighting the health component of the crisis and by focussing their interventions more fully on the health sector, i.e. investing more resources in further strengthening and expanding health programmes and integrating nutrition components (where necessary) into these health programmes. Also MSFH was well placed in the regional paediatric ward to strengthen and support health information systems and analysis and strengthen links in referring clinics, however this was not done. This may have been a better option than focussing on a sector (food aid) where MSFH were dependent upon the commitment and response capacity of other agencies.

MSFH placed too much confidence in rapid MUAC assessments which it saw as a viable alternative to weight for height surveys. Rapid MUAC assessments found alarming rates of Global Acute Malnutrition (GAM) which were not the case where weight for height surveys were undertaken. Similar prevalences of GAM were not identified in a limited number of surveys reviewed where both indicators were measured. The correlation between weight for height measurements and MUAC requires further analysis in this setting before MUAC can be used with confidence in place of weight/height to define prevalence of malnutrition. Trend analysis was also affected by assessments carried out on changing populations and using varying age and height cutoffs.

Weighing supplementary rations, Mazlak IDP camp, Herat 2002

The high dependency on expatriate staff with sometimes limited experience and short contracts, for project management led to lack of continuity of approach. The difficulties experienced by MSFH in human resource management were often related to inability to fill positions by expatriates. Yet MSFH's experience has shown that management capacity can be accessed locally.

At the time MSFH did not have policies which addressed some of the issues that arise in connection with working in the type of longer- term health crisis that exists in Afghanistan. Such a context raises issues on project management, how to work with, through or independently of local partners and infrastructure, the degree of dependence on expatriate staff, coherence of health strategies and methods of withdrawing from programmes. Arguably, this absence of policy led to a modus operandi that was not always appropriate for the context.

Key recommendations

The evaluation contained a number of recommendations related to the above findings. Key recommendations that related specifically to nutrition and food security aspects of the programme were as follows;

• MSFH could further clarify its policy and strategy on implementation of general food distributions based on an analysis of past experience and the internal debate on this issue that has taken place over the past decade. This clarified policy should set out options on what to do in circumstances where food insecurity is severe and undermining MSFH health programmes.
• MSFH could increase capacity to support epidemiological analysis in areas of operation as well as competence in integrating food security, health status and nutritional status analysis.
• Unless the current consensus changes (as evidenced by new research) MUAC assessments and surveys should be treated with the appropriate caution.

¹O'Reilly F., Shoham J. MSF Holland in Afghanistan Mission Evaluation: May 2000-May 2002