The dangers of rapid assessment

By Steve Collins

The food economy of Northern Darfur is complex, with people depending on a wide variety of food sources. Morney Camp, Darfur.

Dr. Steve Collins is a Nutrition Consultant working for Valid International. He undertook this assignment on the request of Save the Children UK and visited Darfur between 28th April and 13 May 2001. He specialises in assessments and setting up emergency nutritional interventions, evaluations and researching severe adult malnutrition and community therapeutic feeding. His previous work experience has included emergencies in Sudan, Somalia, Angola, Liberia, Burundi, Rwanda, DPRK, Balkans and Haiti.

I was recently in Northern Darfur, Sudan, where SCF-UK had employed me to analyse and present data from five nutritional surveys and combine this with food security / economy data collected by their early warning system (EWS)¹. In my opinion, the data SCF had collected was exceptional, both in terms of its high quality and its broad historical and geographical scope. They had undertaken one complete nutritional survey in each of five separate food economy zones, all conducted by well trained teams and implemented according to internationally recognised standards. Detailed food economy and food security data stretching back over ten years supplemented this. The historical records of market prices, terms of trade, harvests, and other sources of income, etc. provided a baseline and enabled the cross-sectional nutritional data to be set in context. This facilitated a clearer understanding of the situation.

The results were very worrying; 24% global malnutrition, six months to the next harvest and clear signs that coping capacities had been exhausted (see graphs). The following table outlines the prevalence of malnutrition throughout the state.

Normal livelihood patterns

The food economy of Northern Darfur is complex, with people depending upon a wide variety of food sources. There is also a wide range of mechanisms that the population employs in order to cope with a variable pattern of food security. People farm cereal, raise livestock, collect wild foods, farm cash crops and traditionally migrate to find work. A strong kinship system, where richer members of clans support their relatives, cements the coping mechanisms together. Interestingly, local crop production is not the most significant source of food in any of the 6 food economy zones.

The October 2000 food economy analysis indicated that there was a food deficit of 26,057 MT. The deficit was anticipated to be greatest in the pastoral, goz and tombac food economy zones and among poor households in these zones. In these three zones, poor households were predicted to not be able to meet over one third of their food needs, indicating a very serious situation.

Erosion of Coping mechanisms in 2001

Crop production

Analysis from the early warning system data indicated that after a bumper year in 1998, crop production in both 1999 and 2000 was low. This is illustrated in the following graph. In addition to crop failures, the drought caused wide spread failure of Koreb, the predominant wild food in Darfur.

Market prices

Analysis from data collected in the six principal markets in Darfur indicated a deteriorating situation. All prices included in this report are unadjusted for inflation.

The following graph presents the average market prices of millet and goats in the six major markets in Darfur. The market price for millet had risen by almost 50% over the previous four months and in March 2001 was higher than during the peak of the hungry season during the crisis in 1997. The signs indicated that the rate of increase in millet prices would continue. The animal market had fared slightly better and although prices were low they had not yet reached the depths of 1997.

Terms of trade between grain and goat were slightly better than it was at the peak of the crisis in September 1997, but were declining rapidly as demonstrated in figure 3:

In the pastoral food economy zone, market prices are intimately related to the rate of malnutrition, and changes in market prices tend to precede changes in malnutrition. The following graph clearly illustrates this fact.

Labour wage rates

The past six months had seen the average wage for labouring decrease substantially. The concurrent high price of millet meant that in March 2001, 27 days of labour were required to buy one sack of millet, compared to 13 in March 1999. Trends in purchasing power in relation to labour payments are outlined below.

Kinship and the capacity of the rich to support their relatives

Support from family / kin represents the last and most profound element in the population's coping strategy. Information clearly indicated that this final mechanism was breaking down. This breakdown was occurring at the beginning of the hungry season, six months before the next harvest.

The principle factors undermining coping strategies are illustrated as follows:

Initiating response

On returning to London we approached the donor organisation DfID, to present the findings and a proposal for an intervention aimed at maximising the amount of food aid to be delivered to Darfur before the rains rendered much of the state inaccessible. Given the comprehensiveness of the SCF data, I expected a broad agreement on the need for action and a positive outcome from the discussions. I was therefore rather taken aback when DfID responded that the SCF data didn't agree with the findings of another NGO that had recently conducted a rapid assessment in the state.

Differing methodologies with conflicting assessment results

This other assessment consisted of a team of two expatriate doctors, a local health assistant and a driver visiting 27 locations across a state the size of France in 21 days. The team held meetings with the local authorities, visited the health facilities, and water points, held discussions with families and screened under-5 children using MUAC (Mid-Upper- Arm Circumference) measurements taken via 'convenience samples' from groups thought to be at high risk (e.g. displaced). The team systematically tried to focus its attention on the most vulnerable areas and families with the aim of describing the situation of the most at-risk rather than giving a general picture of the situation. Given their attempts to focus on the most vulnerable their results were surprising. Of the 424 children that they measured, only 1% had a MUAC <110mm, 5% was between 110 - 125mm and 12.5% between 126 - 135mm. A very different picture to the results of the SCF surveys.

Rapid assessments and 'selective' sampling

A likely explanation for these differences is that the assumption that the displaced are the most vulnerable was false. In Darfur, this assumption was an oversimplification as, the displaced living around the wadis are those who still have cattle remaining, and are in fact the richest segment of the population. This clearly illustrates the dangers of rapid assessments and convenience sampling. One erroneous assumption can completely alter the interpretation of the whole data set.

I recognise that in many situations there may be no time for systematic sampling and we all at one time or another resort to 'convenience samples'. Although we are all aware of the biased nature of these samples and put a "disclaimer" that the results cannot be generalised etc., how many of us then come to believe our own results, especially once they are analysed and presented in a nice table? This was true of this rapid assessment in northern Darfur which initially stated clearly that the MUAC results were not "statistics and do not pretend they are representative of the general nutritional status". Later this position changed in the report to "Even if these numbers do not represent statistics, it shows the low presence of severely malnourished children even among populations facing very hard condition of life for several months (or years)." I wonder whether the same conclusions would have been drawn if they had realised that they were selectively sampling the richer households.

In the field such misleading results are bad enough, they can do even more harm when they filter back to donors as they can be used to justify political positions. In this case the findings from this small non-representative sample negated the results of a random stratified sample of 3779 children using weight for height measurements and complemented by ten years of involved early warning data. Happily the DfID staff later went to Darfur for themselves and quickly decided to fund a response.

Valuable lessons highlighted

In my opinion, there are important and simple lessons to learn from this. Rapid assessment and convenience samples will and should continue to form an important element in emergency responses. Often there is no time to wait for a formal survey and to do so would waste precious time and delay response. However, it is imperative that such assessments are conducted as broadly as possible (i.e. draw on a wide variety of sources) and the results obtained discussed with others in the field who may have valuable complementary insights, knowledge and data. Dissemination and sharing at field level should occur before any reports filter back to central offices and thence to donors. Although, statistical and nutritional acumen and intelligence are necessary elements in any rapid assessment team, these cannot compensate for local knowledge. A good rapid assessment is one that accesses as broad a cross-section of local knowledge as possible. In Darfur, there was a wealth of information available and closer consultation with other agencies operating in the state, both during and after the assessment, could have prevented much confusion and delay.

References

Save the Children Fund (UK) and Development and Rehabilitation Committee of North Darfur, October 2000 Village and Household Survey and Food Needs for 2000/2001 Save the Children (UK). How bad does it have to get?. The nutritional status in N Darfur in the spring of 2001, El Fasher data from SC-UK EWS/Nutrition Unit, El Fasher, N. Darfur report compiled by Steve Collins

¹El Fasher, Kutum, Mellit, Nyala, El Geneina & Umkeddada

²Dotted lines are projected prices based upon changes over the previous four months.

³Dotted line is projected terms of trade based upon rate of change during the past four months.