Contextual data collection in nutrition surveys in Ethiopia

Summary of analysis¹

The study described in this article was carried out by NutritionWorks in collaboration with the Emergency Nutrition Coordination Unit (ENCU) of the Early Warning and Response Directorate/Disaster Risk Management and Food Security Sector (EWRD/DRMFSS), Government of Ethiopia. Funding was provided by the Global Nutrition Cluster and OCHA Ethiopia.

NutritionWorks (a partnership of international nutritionists based in UK) have recently completed a review of 'contextual'² data that are routinely collected during nutrition surveys in Ethiopia. Currently, nutrition surveys collect a large quantity and range of data on context but guidelines on methods and indicators are unspecific³.

The objectives of the analysis were:

• To evaluate the methods and indicators used to collect 'contextual' data in nutrition surveys conducted in Ethiopia.
• To evaluate the recommendations made from the data.
• To examine the use made by agencies or programmes of the survey data collected.
• To support further development of the Government of Ethiopia's interim Guidelines for Emergency Nutrition Surveys 2008 by determining whether standardised methods and indicators for contextual data collection and analysis can be employed.

Data collection

Data collection was carried out in two stages. First, all reports of nutrition surveys carried out in Ethiopia from January 2003 to December 2008 and available at the Emergency Nutrition Coordination Unit (ENCU) office in Addis Ababa were examined. Summary survey reports without details of methods were excluded. Information was extracted and summarised on an EXCEL spreadsheet with 15 sheets. Each sheet summarised information on the following areas:

• Objectives
• Vaccination
• Methods
• Water and sanitation
• Background
• Health
• Anthropometry
• Infant feeding
• Micronutrients
• Food security
• Mortality
• Coping strategies
• Morbidity
• Conclusions
• Recommendations

The data were then transformed into STATA files for tabulation and description.

The second stage involved interviews with key informants from agencies actively involved in carrying out nutrition surveys and using the results for programming purposes and advocacy in Ethiopia. Interviews with national-level staff based in Addis Ababa and with regional-level staff in Amhara and SNNPR (Southern Nations, Nationalities, and People's Region) were carried out. Those interviewed included staff from government departments (Health and Food Security), donors, United Nations (UN) agencies and Non-Governmental Organisations (NGOs).

Table 1: Sample sizes of different methods used to collect 'contextual' information
	Range of sample size	Comments
Household questionnaires	87 - 972	Clustered around 90, 180 and 300 households in sample, depending on sampling method.
Community questionnaires	26 - 45	Clustered around 30 and 36 communities
Focus group discussions	6 - 36	Clustered around 30 focus group discussions
Key informant interviews	30 - 45	Few surveys reported sample size

A total of 291 surveys were included in the final analysis and a total of 64 interviews with key informants were carried out. Of the 291 surveys analysed, the majority were carried out in SNNP and Oromia regions. Five agencies carried out over 70 % of all surveys examined. These were Concern, GOAL, the government (ENCU), Save the Children UK and World Vision Ethiopia (WVE).

Findings

The most common objectives for carrying out surveys as cited in survey reports were to assess the prevalence of acute malnutrition, mortality rates and vaccination coverage. The objective of assessing the causes of malnutrition was included in just over half the surveys.

A variety of methods were used to collect 'contextual' data. These included household questionnaires, community questionnaires, focus group discussions and key informant interviews.

There were considerable differences in the sample sizes used to collect household and community information (see Table 1). This was because different sampling procedures were used to identify samples. For example, sometimes every third household was selected to complete a questionnaire, sometimes every fifth household or sometimes another process was adopted.

Contextual indicators

A large number of 'contextual' indicators were collected. Table 2 presents the range of 'contextual' indicators that were collected ordered into four categories: health, care, food and coping strategies. A total of 46 of the most commonly collected indicators are presented. Many surveys collected additional indicators.

'Contextual' indicators were not necessarily collected in a consistent manner. For example, information on type of water source was collected in 63 % of surveys but as Table 3 illustrates using results from six example surveys, the data were not comparable.

Correlations between anthropometric and contextual indicators

The dependent variable, global acute malnutrition (GAM), was correlated with 93 independent contextual variables. Of these, 1 in every 1,000 will be statistically significant with P values of <0.0001 by chance alone. As Table 4 shows, in total 13 statistically significant correlations were found (P<0.001) which exceeds the number expected by chance alone.

Table 2: Range of 'contextual' indicators collected
	Indicator	% of surveys collecting data (n=291)
Health	Measles (coverage card and recall)	96.6
	Sick (in last 2 weeks)	90.4
	BCG (coverage)	88.3
	Water source (type)	63.2
	Vitamin A supp. (coverage)	60.0
	Water source (distance to)	47.7
	Access to safe water	28.9
	No. of health facilities accessible	27.8
	Health clinic (time to)	26.1
	Latrines (households with)	10.0
	Mosquito nets (used)	7.6
	Health clinic (took sick child to)	2.1
	Water (cost)	1.4
	Wash hands	0.3
Care	Maternal literacy (illiterate mothers)	25.4
	Meal priority (for children)	22.7
	Feeding practice in last 24 hours (reported)	22.0
	Weaning* (introduced food at 6 months)	17.9
	Breastfeeding (starting feeding within 1 hour of birth)	12.7
	Breastfeeding (exclusive to 6 months)	12.0
	Breastfeeding (gave colostrum)	3.4
Food	Source of food (own production etc.)	74.9
	Source of income/cash	74.9
	Livestock condition (below average)	67.7
	Main staple food	65.3
	Source of food in future	57.7
	Pasture condition (below average)	48.1
	Last rainfall (worse than normal)	47.4
	Last harvest/crop production (worse than last year)	41.2
	Livestock death/herd size reduced	38.5
	Livestock prices lower	37.4
	Change in staple food from 'normal'	34.0
	Food prices higher	33.7
	Receiving relief food	30.6
	Market access (time to)	16.5
Coping strategies	Migration (unusual)	46.4
	Livestock sale	39.9
	Consumption of unusual/wild foods	33.0
	Meal reduction	29.2
	Asset sale	23.0
	Wage labour	6.2
	Dietary diversity score (adults)	5.8
	Dietary diversity score (children)	5.8
	School drop out	5.5
	Borrow food or money	2.1
	CSI scores	1.0

* Complementary feeding

Table 3: Example from six survey reports of the different parameters collected on source of water
(% of households reporting access to different water sources)	Survey 1	Survey 2	Survey 5	Survey 108	Survey 110	Survey 272
River		5.7	13.9	21.5	7.3	23
Ponds		3.4	4.4		21.7	2
Irrigation canal					41.3
Wells				0.7		27
Springs				66.2
Springs (unprotected)		25.7	5.0		3.7
Springs (protected)		6.3	2.8
Piped		33.7	70.6
Pump				3.3
Boreholes
Cistern/rain harvest			3.4		9.7
Purchase/water tankering		25.1				13
Other
Total safe water			73.4		16.3	29

 

Table 4: Significant correlations between the dependent variable GAM and independent contextual variables
Independent variables	r (correlation coefficient)	P (statistical significance)	N (no. of paired values)
POSITIVE
Crude mortality rate	0.4909	0.0000	288
Under five mortality rate	0.5509	0.0000	288
Main source of food in future: borrowed	0.5147	0.0000	38
Main source of food in future: food aid	0.4049	0.0000	106
Female headed household	0.3320	0.0003	117
Livestock sold as coping strategy	0.3961	0.0000	116
Unusual migration as coping strategy	0.4051	0.0000	135
NEGATIVE
BCG scar (6-59 months)	-0.4285	0.0000	257
Measles vaccination (card and/or caretaker recall)	-0.2704	0.0000	281
Vitamin A coverage (caretaker recall of vitamin A in last 6 months )	-0.3688	0.0000	174
Source of food: own production	-0.3179	0.0000	200
Main source of food in future: own production	-0.3458	0.0000	200
Price of livestock has fallen (lower)	-0.3637	0.00001	109

 

Seven of the correlations were positive. This means that a high prevalence of the contextual indicator is associated with a high GAM rate. For example, high mortality rates were significantly associated with high GAM rates. Six of the correlations were negative. This means that a low prevalence of a variable like prevalence of BCG scar is associated with high GAM rates.

Surveys frequently reported on other important contextual factors based on secondary sources rather than primary data collection from the survey. The percentage of surveys that mentioned these other contextual factors, either in the introduction or in the discussion of the report, are presented in Table 5.

Aggravating factors

A classification of overall severity based on GAM rates plus aggravating factors has been developed for Ethiopia. This is based on a similar classification system developed by the World Health Organisation (WHO). There are five categories of aggravating factors described in the ENCU Guidelines for Emergency Nutrition Surveys in Ethiopia 2008 (see column 1 of Table 6). Indicators are not specified for these aggravating factors in the guidelines.

From the 46 commonly collected indicators listed in Table 2, potentially relevant indicators for the aggravating factors were identified and are listed in column 2 of Table 6. Column 3 notes the percentage of surveys that collected these indicators. While over 90% of surveys collected information relevant to aggravating factors category 2 (epidemics of communicable diseases) and category 4 (vaccination coverage), far fewer surveys collected data relevant to category 1 (household food security) and category 5 (water and sanitation). No data were collected which related to category 3 (shelter).

Recommendations in nutrition surveys

Recommendations were grouped into 21 intervention types. Table 7 lists these interventions with the percentage of surveys that made a related recommendation. The most common recommendations related to vaccination, general health and WASH (water, sanitation and hygiene). Many of the recommendations tended to be general and were usually not SMART⁴ (Specific, Measureable, Attainable, Relevant and Timebound).

Specificity and categorisation of recommendations

There were different categories of recommendation ranging from the very specific to the very vague. Recommendations could call for:

• Immediate action
  e.g. "Implement blanket supplementary feeding for all the under five children, pregnant & lactating women for one/two months
• Longer-term actions
  e.g. "Education intervention on maternal and child care in particular and females' education in general should be enhanced."
• Be directed at a specific agency
  e.g. "Samaritan purse should strengthen its Community Therapeutic Care (CTC) Programme to address the problems of severe acute malnutrition; 203 under five severely malnourished children were estimated to be eligible for programme"
• Be directed in general
  e.g. "Efforts should be made by all development actors in improving the use and distribution of insecticide-treated bed nets to minimize the effects of malarial disease and attention should be given [to] under 5 children and pregnant & lactating mothers"
• Recommend direct nutrition interventions
  e.g. "The EOS/TSFP⁵ programme should be strengthened to treat moderately malnourished children as 579 children were estimated to be eligible to EOS/TSFP and Outpatient Therapeutic Programmes (OTP) and it should be implemented on a monthly basis".
• Recommend interventions indirectly linked with nutritional outcomes
  e.g. "Regional cooperative office should introduce small scale saving and credit scheme (in collaboration) with Amhara credit and saving organisation so as to minimize asset depletion, especially in peasant associations (Pas) that have been producing surplus produce in order to fill food gaps during drought seasons."

Relationship between GAM and recommendations

In 11% of surveys where GAM rates fell below 5%, there was a recommendation to introduce or continue general food distribution. In 8% of surveys where GAM rates exceeded 20%, general food distribution was not recommended.

Similarly, in 8% of surveys, supplementary feeding was recommended though rates of GAM were below 5% while in 8% of surveys no recommendation was made despite GAM rates exceeding 20%.

Calculation of programme requirements

Supplementary and therapeutic feeding requirements were increasingly calculated by surveys over the time period of investigation; 6% in 2003 and reaching 78% by 2008, as Table 8 shows.

Table 5: Background information reported from secondary sources
Indicator	% of surveys collecting data (n=291)
Livelihood of population	94.2
Health and nutrition programmes functioning in area	84.8
Agro-ecological zones	73.2
Seasonality	64.3
Access to health services	50.5
Access to education services	28.2
Female headed household	0.7

 

Table 6: Categories (1-5) and description of aggravating factors and potential indicators to assess them
Categories and description of aggravating factors*	Relevant indicators collected	% of surveys collecting this type of data
1. Poor household food availability and accessibility (due to a poor harvest, poor pasture conditions, high market prices, insecurity, or inadequate general distribution in a camp setting, etc.)	1. Harvest worse than last year (yes/no)	41.2
	2. Pasture conditionbelow average (yes/no)	48.1
	3. Price of food increased (yes/no)	33.7
	4. Receiving food aid (% of households)	30.6
2. Epidemics of measles, cholera, shigella and other important communicable diseases	5. Sick in past 2 weeks (% of children aged 6-59 months)	90.4
3. Inadequate shelter and severe cold
4. Low levels of measles vaccination and vitamin A supplementation	6. Measles vaccination verified by card (% children)	93.1
	7. Measles vaccination verified by card or caretaker report (% children)	96.6
	8. Vitamin A supplementation (% chidlren)	60.0
5. Inadequate safe water supplies (quality and quantity) and sanitation	9. Access to safe water (% households)	28.9
	10. Access to latrines (% of households)	10.0

*Taken from: Revised Guidelines for Emergency Nutrition Surveys in Ethiopia were produced by the ENCU in September 2008.

Use of nutrition survey data

Purpose of nutrition surveys

Respondents were asked the reasons why nutrition surveys were carried out in Ethiopia. The reasons given ranged from purposes linked directly to 'emergency response' to more general purposes linked to 'longer-term response'⁶. Most surveys were carried out for more than one overall purpose. Other reasons included:

Confirmation and identification of 'hotspots': to confirm the severity of a situation. Nutrition surveys are sometimes done to confirm a problem after a rapid nutrition assessment has shown potentially high levels of acute malnutrition based on mid upper arm circumference (MUAC), or woreda officials have reported suspicions of high malnutrition which require objective verification. Nutrition survey data are also used for classification purposes. The ENCU has a system for defining 'hotspots' (areas/woredas affected by current food insecurity due to drought, disease, flood, etc.) based on GAM rates and other factors.

Advocacy: To convince donors or government to provide more funds. Nutritional information is highly sensitive in some regions of Ethiopia.

Entry and exit: To provide evidence that either there is a need to intervene or that the situation has stabilised and emergency interventions can be handed over by the NGO to the government. This is particularly relevant for therapeutic care programmes that are opened in times of acute stress for relatively short periods of time (3-6 months).

Resource requirements and coverage: To quantify resource requirements, e.g. amount of therapeutic foods, and to estimate the coverage of existing programmes. This information is useful to Operations Managers' to check on end-point delivery and coverage. For example, findings can help quantify need for community therapeutic care and outpatient therapeutic programmes.

Baseline and monitoring: To have baseline information at the start of a programme or intervention. Follow-up surveys are used to monitor the situation regularly while interventions are being carried out.

Use of 'contextual' indicators collected in nutrition surveys

Most respondents recognised that a lot of the data collected in nutrition surveys are never used:
"We do almost nothing with that contextual data....."
"20% of data can be used for immediate purposes, 30% can be used for long-term programmes and 30-40% of data collected in surveys can never be used"

Data considered most important for decision-making were:

• Mortality
• Acute malnutrition
• Current source of food
• Recent harvest

Data considered least important were:

• BCG coverage
• Distance/time to water
• Access to latrines
• Hygiene habits
• Distance/time to health centre
• Literacy of mother

Some respondents reported that recommendations in nutrition surveys were generally very weak. They were vague, not time bound, not directed at a particular agency, and no idea was given of how improvement should be effected, or whether there would be extra funding.

One view expressed was that ultimately, whether a recommendation was carried out or not depended upon funding. For example, community based management of acute malnutrition (CMAM) is widely supported in Ethiopia but SFP is not. Therefore there is minimal funding for SFP, even if it is recommended. The general view was that whatever was recommended, the response was going to be "always food".

Table 7: Recommendations by intervention type
	% surveys
General ration (introduce or continue)	46.7
Supplementary feeding (introduce or continue)	40.9
Therapeutic care (introduce or continue)	47.4
Distribution of seeds	10.6
Agriculture general	14.1
Livestock feed	6.5
Veterinary services	14.4
Livestock general	12.7
Prevention/treatment of malaria	12.7
Vitamin A supplementation	26.5
Vaccination	71.1
Health/nutrition education	40.5
Health general	70.8
WASH programmes	66.7
IEC(Information, Education, Communication) on feeding practices	25.4
Employment generation	19.6
Livelihood support	24.7
Capacity development	37.8
Repeat nutrition survey	35.7
Monitor nutrition and/or food security situation	58.8
Other*	27.5

*Examples of 'Other' included: support for school feeding and improved water provision, introduce small-scale savings and credit scheme, construction of flood protection structures, strengthen re-settlement programme, encourage population in asset building.

Table 8: Calculation of therapeutic and supplementary programme requirements
(% of surveys n=291)General ration (introduce or continue)	Year of survey
	2003	2004	2005	2006	2007	2008
Yes	6.3	2.3	21.7	36.5	51.3	78.7
No	92.1	97.7	73.9	59.6	48.7	17.0
Missing data	1.6	-	4.4	3.9	-	4.3
Total	100	100	100	100	100	100

 

Discussion and recommendations

Lack of standard methods for collecting 'contextual' information

While standard methods were employed to collect malnutrition and mortality information, standard methods were not employed to collect 'contextual' information. A variety of tools were used that were not standardised and differed in terms of sampling procedures and therefore sample size, methods for data collection and indicators collected.

The new 2008 ENCU interim guidelines based on SMART methods provide detailed guidance on sampling and data collection methods for anthropometric and mortality data. Very little detail is given about contextual data collection, so those doing assessments have to rely on the 2002 ENCU guidelines. The 2002 ENCU guidelines provide broad but not specific guidance. It is therefore not surprising that agencies adopt a variety of different methods to collect contextual information.

Standard methods for contextual data collection would help to ensure that results were comparable, while clear recommendations on sample sizes required for particular types of context data could lead to considerable saving of resources.

Large numbers of contextual indicators collected

A very large number of contextual indicators are collected in each nutrition survey. Forty six indicators were commonly collected and the number of questions asked usually exceeded 50. The total number of data points collected was probably on average close to 100.

While it was not possible to measure the time required to collect, record, analyse and report the large number of contextual indicators collected, it is likely to take days. This time is wasted if contextual indicators are not used. A smaller number of useful contextual indicators would reduce time and costs.

Lack of standard contextual indicators

Four contextual indicators were defined in a consistently uniform manner and collected in the majority of nutrition surveys in Ethiopia. These indicators were:

• Vitamin A supplementation coverage (care-taker report that child had received supplement in last six months)
• BCG vaccination coverage (scar present)
• Measles vaccination coverage (card only; card and/or caretaker report)
• Child sickness in last two weeks (caretaker report)

For the remaining indicators, however, there was practically no consistency in which indicators were collected, definitions of indicators, cut-off points or methods of data collection.

Methods and results were not described or presented in any standard way in reports.

There is, as yet, no agreement amongst stakeholders in the emergency nutrition sector (either within Ethiopia or internationally) as to which contextual indicators should be collected. Respondents who filled out the indicator scoring sheet were almost unanimous in agreeing that mortality and acute malnutrition were key indicators that should always be collected in nutrition surveys. There was little agreement about which other indicators were essential to collect and which were not. Judgements were made using different criteria. For example, some indicators were given a low rank because they were judged to be unreliable, others because they didn't change much over time, others because they weren't going to have an impact on programming decisions in the short-term.

The result of these findings is that it is not possible to compare the survey findings (over time or between different places).

Problems of interpreting contextual indicators

Ideally, contextual data should provide an indication of the following:

• Severity of the current situation
• Future trends
• Programme needs
• Causes of malnutrition
• Coping strategies and capacity of population

Currently, however, the interpretation of contextual data is largely subjective and depends on the views of those carrying out the survey. A number of problems in interpretation of contextual information were identified.

Lack of baseline or comparative data

Baseline or comparative data for a particular indicator were rarely reported. No baselines for contextual indicators are included in the 2002 or 2008 guidelines. This makes it impossible to know whether the situation is 'normal', or better/worse than 'normal'.

Lack of trend data

Trend data for contextual indicators are not usually presented in nutrition survey reports. The 2002 guidelines do not provide sources for potential trend data available from other sectors. Trend data for contextual data would provide some kind of benchmark to be able to interpret data. In particular, it would provide information on normal seasonal changes.

Thresholds for response

In the 2002 guidelines, there are no cut off points or thresholds for the contextual data discussed above for which some kind of response is recommended. The only guidance on thresholds is linked to 'aggravating factors'. However, there are no clear definitions of aggravating factors given or cut-off points for response for these factors. They are, therefore, of little use in their present format for decision-making.

Failure to collect essential information for interpretation

Background information essential for interpretation was often not reported. For example, seasonality was not mentioned in 32% of surveys although this is crucial for interpretation of acute malnutrition figures. According to the ENCU guidelines 2008, in order to be able to interpret correctly the malnutrition rates from a survey, it is necessary to consider the following factors:

• Determine seasonal variations.
• Compare the results to previous surveys in the same area or livelihood zone at the same time of year.
• Using the livelihood analysis framework, explore how the levels of malnutrition can be explained by a change in the livelihood of the population compared to the baseline livelihood zone profile.
• Interpret all results in their cultural, socioeconomic and agroecological context, together with other supporting data such as indicators on health, food supply, markets, etc.
• Analyse mortality rates in the survey area.
• Look at what interventions are already being implemented in the survey area.

These essential pieces of information were frequently not mentioned in reports at all or mentioned in passing only. There is a need routinely to collect and present essential background information in a standard way.

Non-conformity in report formats

There is no consistency in the way that contextual data are presented in reports. Much of the data are not reported upon at all and the decision to report on some data but not others leaves room for bias. Interpretation of data could be greatly improved if clear guidance was provided on how to present and interpret context information. This would stipulate a minimum set of background information that should always be presented (e.g. seasonality) and a set reporting format including tables for contextual indicators where results are presented in a uniform way.

Importance of aggravating factors in identification of 'hot spots'

A major purpose for conducting surveys is to identify 'hot spot' woredas. At present the identification of hotspots is based on largely subjective indicators. Given that the majority of nutrition surveys find similar rates of malnutrition (nearly half the surveys found a GAM rate between 5 and 10% while three quarters of GAMs fell between 5 and 15%), the availability of accurate information on aggravating factors is crucial. However, data on aggravating factors are frequently not collected at all.

It would be useful to develop a core set of contextual indicators that can provide more objective information on aggravating factors. This information could be used in the decision- making process in relation to 'hotspots' and ensure greater accountability and reduce bias.

Contextual indicators not used for taking decisions

It is clear that a huge amount of contextual information is collected but a lot of it is never used. This has time and cost implications. The collection, coding, analysis and reporting of contextual data take time. As most NGOs do not keep a survey team, staff who could be used for other activities are employed doing surveys instead. Furthermore, additional data collection has a cost. The average nutrition survey costs around $10,000. A total of 509 surveys were carried out over a 9 year period in Ethiopia according to the ENCU. This averages out at more than one per week at a cost of more than half a million US$ per year. It is essential that time and money are not wasted on collecting information which is not used for decision-making.

Lack of linkage between survey results and action

While there was some relationship between survey results and subsequent interventions, there were some odd exceptions. For example, the relationships between GAM levels and general food distributions (GFDs) and SFPs. In general, recommendations are weak in nutrition survey reports. Guidance on how to write SMART recommendations that link directly with results is sorely needed.

Importance of specifying the purpose of nutrition surveys

The underlying reasons (as opposed to the specific objectives) for carrying out nutrition surveys are often not explicit and there is a lack of clarity as to what a survey can or cannot achieve. This needs to be addressed in guidelines. For example, nutrition surveys are currently not designed to measure impact or coverage of interventions and are unable to identify causes of malnutrition, though they may provide a crude indication of associated factors. Some nutrition surveys have purely short-term objectives while others are done for long-term purposes.

For more information, contact: Fiona Watson, email: fiona.watson6@btopenworld.com

---

¹An Analysis of Nutrition Surveys in Ethiopia Background Paper for workshop on contextual information collected in emergency nutrition assessments in Ethiopia, Addis Ababa 22nd and 23rd September 2009

²'Contextual' data refers to all information collected in nutrition surveys with the exception of anthropometric and mortality data. These data are sometimes referred to as 'non-anthropometric' and are usually collected to help understand the causes of malnutrition.

³Revised Guidelines for Emergency Nutrition Surveys in Ethiopia were produced by the ENCU in September 2008. The section on non-anthropometric data is brief and unfinished. The previous Guidelines of 2002 include an extensive description of non-anthropometric data collection but do not specify particular methods or indicators. International guidelines such as SMART and Sphere do not currently specify standard methods and indicators for the collection of 'context' data.

⁴SMART is usually applied to objectives but it could usefully be applied to recommendations.

⁵Enhanced Outreach Strategy-Targeted Supplementary Feeding Programme

⁶Note that ENCU is the Governments Emergency Nutrition Coordination Unit and is within the Early Warning Unit/DPPA. It is therefore only involved in emergency assessment and response.