Review of survey methodology in emergencies

Published:

01 September 2007

Summary of published research¹

Spinning the stick to select clusters during a UNHCR survey in Bangladesh

A recent paper set out to identify common methodological errors in nutrition and mortality surveys conducted in humanitarian emergencies, to examine trends over time and to provide recommendations on how to improve surveys in future.

The sample of surveys was selected from 948 reports of nutrition surveys received by the Nutrition Information in Crisis Situations (NICS) between October 1993 and April 2004 from 34 countries. Of these, 17 countries were selected using a random number generator and all of the survey reports in these countries were reviewed for analysis. Survey reports were evaluated for validity of sampling methodology, precision of estimates, quality of measurements and calculation of prevalence of acute malnutrition and mortality rates.

Three hundred and sixty eight survey reports conducted by 33 non-governmental organisations (NGOs) and international agencies in 17 countries were eventually evaluated.

Criteria for sampling validity were met for 85.9% of surveys. All of the random sample surveys that used sample sizes of 450 children for random and systematic sampling and 900 children for cluster sampling were sufficiently precise. However, cluster surveys that sampled <900 children were insufficiently precise - half of these were conducted in Sudan with many of them occurring in 1999. The vast majority of surveys correctly included children 6-59 months or 65-110 cm. Measurements met the quality criterion in 57.1% of surveys, the remainder that did not meet criterion due mainly to missing information. Regarding oedema, 8.4% of surveys did not include oedema in the definition of acute malnutrition. Three-quarters (76.1%) of the surveys provided the percentage of oedematous children. Incorrect interpretation of results occurred in 15.5% of surveys. Overall, 42.4% of surveys met the criteria for correctly calculating prevalence of acute malnutrition.

Just over three-quarters (76.6%) of surveys were both valid and precise, 51.3% were valid, precise and met the quality of measurement criteria. Finally, 35.3% were valid and sufficiently precise, met the criteria for quality of measurement, outcome definition and calculation.

Measles vaccination coverage could be assessed in 57.1% of surveys. Most (72.3%) used card examination and history of vaccination. Just over half (52.4%) of nutrition surveys had an associated mortality survey. Of these, 81.3% assessed Crude Mortality Rate (CMR) and under-five mortality rates (U5MR) while only 18.1% assessed U5MRs.

Among the 158 surveys that assessed CMRs, 55.1% met criteria for sampling validity and the same percentage met precision criteria, 38.6% of surveys were both valid and precise and only 3.2% were valid, precise and met the calculation criteria.

The proportion of nutrition surveys that met criteria for sampling validity, precision, measurement, definition and calculation rose significantly from 11.1% in 1993-4 to 51.7% in 2003-4. The implementation of CMR surveys associated with nutrition surveys increased significantly over the years but the proportion of CMR surveys that met criteria for sampling validity, precision and calculation did not differ.

The lack of validity of the mortality surveys was mainly due to a lack in specificity in the reports as to how household selection occurred. The lack of precision in the CMR surveys was most likely because the sample sizes often used the same sample size calculated for nutrition surveys. Sample sizes for each key variable in a survey must be calculated independently. Many reports lacked structured methodology and results sections and had insufficient detail to allow for proper analysis and evaluation.

The quality of calculations depended upon the software used for analysis. Approximately 33% of calculations performed with EpiInfo 6 were incorrect because oedematous children had not been correctly taken into account. No comprehensive guideline to analyse nutrition surveys using EpiInfo 6 was available before 2004 when a manual that extensively describes the procedure was released.

Only approximately half of surveys assessed measles vaccination. All nutrition surveys among children should include a measles vaccination coverage component and should clearly disaggregate in the report the percentage according to both history and vaccination card as well as card alone.

In the study, precision of nutrition and mortality surveys was analysed assuming a design effect of 2.0 for cluster-sampled nutrition and mortality surveys. However, recent research shows that a design effect of some nutrition and non war-related mortality surveys was below 2.0 and closer to 1.5. If a design effect of 1.5 had been used, the proportion of nutrition and mortality surveys for which the sample size was sufficient would have been higher. Conversely, war-related mortality surveys might have a higher design effect and might require a higher sample size.

The study concludes that several factors are essential to the strengthening of the quality of nutrition and mortality surveys. Appropriate guidelines are now available and need to be widely disseminated; the increased availability and access to the internet in remote areas has improved their distribution. However, the most crucial factor is the availability of well-trained and experienced staff.

The authors cite a number of initiatives to improve and standardise survey methodology, e.g. FAO, through the Food Security Analysis Unit (FSAU) for Somalia since 1994 and the Emergency Nutrition Coordination Unit (ENCU) in Ethiopia. Furthermore, an international inter-agency initiative was launched in 2002 to Standardize Monitoring and Assessment of Relief and Transitions (SMART).

There are still a number of methodological issues in nutrition and mortality surveys in humanitarian emergencies that need further study. These include, but are not limited to, the mapping of the affected populations for first stage cluster sampling and the selection of households for the second stage, documenting different design effects among various outcomes in different situations, and different methodologies to record deaths, births, and migration in mortality surveys. Furthermore, there is a need to consider standardised age distribution in order to compare over different situations for mortality surveys. The introduction of the recently released WHO growth standards suggests important differences in the diagnosis of wasting compared to the National Centre for Health Statistics (NCHS) growth reference and will thus make comparison of trends in malnutrition in different regions more complicated.

Finally, there is a need for a central and accessible site on the internet, not only for survey reports but also the actual data. This repository will help to ensure quality of surveys and aid in future research.

¹Proudhon C and Spiegel P (2007). A review of methodology and analysis of nutrition and mortality surveys conducted in humanitarian emergencies from October 1993-April 2004. Emerging Themes in Epidemiology 2007, Vol 4:10