Practical challenges of evaluating BSFP in northern Kenya

Summary of published study¹

Beneficiaries of BSFP in Kenya

A mass or ‘blanket’ supplementary feeding programme (BSFP) was implemented by the World Food Programme (WFP) and partners in five northern districts of Kenya between January and April 2010. It was undertaken due to fears of an increase in the incidence of malnutrition as a result of seasonal food insecurity exacerbated by persistent drought. The five programme districts of Mandera, Marsabit, Samburu, Turkana and Wajir cover 45% of Kenya's total land area (Figure 1) but at the time, contained only 4.5% of the population of 28.87 million recorded in the 1999 census².

An attempt to evaluate the impact of the food on children's anthropometric status was put in place in three districts. A recently published study set out to assess the quality of the data on the cohort of children studied in the evaluation and to propose methods by which it could be improved to evaluate future blanket feeding programmes. Reasons for the poor quality of the evaluation are identified.

BSFP intervention

The primary stated aim of the programme was to protect the nutritional status of an estimated 300,000 children aged 6-59 months, or 20% of the 1999 census population³. All children <110 cm in height were eligible for a ration of food plus any taller children whose mother insisted that they were <60 months of age.^4,5

Rations of food provided by WFP were distributed on four occasions, each about 30 days apart, beginning in January 2010. They consisted of 7.5 kg of corn-soy blended flour (CSB) and 0.75 kg of vegetable oil to provide an average of 1,000kcal/day/child. The food was distributed by non-governmental agencies (NGOs) at the sites of pre-existing feeding programmes and at some extra sites, to improve local access.

An evaluation was undertaken to try to detect evidence for an effect of the rations on the anthropometric status of children in three main ways:

• By comparing the anthropometric indices of newly recruited children at the second and third food distributions with children enrolled at the first distribution, to assess if their anthropometric status was getting worse during a period when food security was supposedly poor or deteriorating.
• By comparing the weight change of children who received two, three or four rations of food during the programme, in order to detect a dose-response relationship.
• By comparing the weight change of singleton children with children matched for age and sex in households of two or more children, based on the premise that if a ration was shared it would be less effective than if it was given to an only child, and assuming that the ration was not shared outside the household.

Evaluation sites selection and process

The rations of food were distributed initially at 540 sites in the districts of Mandera (99 sites), Marsabit (55), Samburu (101), Turkana (162) and Wajir (123) (Figure 1) by a group of eight NGOs led by Save the Children, UK (SCUK).

An arbitrary number of 25 food distribution sites were randomly selected for study in each of two adjacent districts, Mandera and Wajir, 26 sites operated by SCUK and 24 by Islamic Relief. At the request of the National Nutrition Technical Forum, 25 sites were also randomly selected in Turkana district, which is in a different livelihood zone and contains a different ethnic group, the Turkana (most people in Mandera and Wajir are Somali). Four agencies were responsible for collecting data in Turkana: Merlin (10 sites), Samaritan's Purse (4), IRC (1) and World Vision (10). Because of a delay in funding, World Vision did not collect data at their 10 sites.

The staff of each NGO was responsible both for distributing the food and for collecting data for the evaluation. All members of staff were given one day's theoretical training by SCUK on the BSFP, community mobilisation, organising distribution sites and on the evaluation methods, including sampling children and administering questionnaires. The five NGOs then organised two days practical training for their field staff according to an agenda specified in the programme guidelines⁶. All the NGO staff were nurses or nutritionists who were supposedly practiced at making anthropometric measurements, so no specific training on anthropometry was arranged.

The aim was to recruit up to a 10% sample of children at the first food distribution at each study site and then at the same sites, recruit all new children who were brought to claim a ration at the second and third food distributions, as they should not yet have received any supplementary food.

A power calculation using Stata 11⁷ indicated that a sample size of 3,022 children could detect a 4% difference in the prevalence of wasting from 26% (the average prevalence reported in three previous surveys)⁸ over the period of intervention. This allowed for a design effect of 2 due to the clustering of children around distribution sites, 25% drop-out, and assuming a power of 80% and a two-sided statistical significance of P<0.05.

Each child was weighed to a precision of 0.1 kg on electronic scales (Uniscale, UNICEF) and measured to a precision of 0.1 cm, supine if <87 cm and standing if =87 cm on locally made stadiometers, according to Kenya Government guidelines⁹.

Each caregiver was given a ration card for the child with a unique identification number created from the site code and the child's serial number. These numbers were also recorded in a register book for each site and on the data forms for each child at each visit to collect a ration and were used to link data. Ration cards were given only at sites taking part in the evaluation.

The date of birth and date of visit were used to estimate each child's age in months at enrolment and z-scores of height-for-age, length-for-age and weight-for-height were calculated using a macro for Stata 11¹⁰ published by the WHO¹¹. This flags values of weight-for-age that are >5 and <-6 S.D., values of height-for-age that are >6 and <-6 S.D. and values of weight-for-height that are >5 and <-5 S.D. because the underlying data are likely to be wrong.

Assessment of evaluation data quality

In order to assess the quality of the data collected in the evaluation, five indicators were used:

• The name of the child recorded on both occasions on different data forms. The names were judged to be the same, different or possibly different.
• The age distribution of children aged 6 - 59 months, which should be similar to the distribution reported in the last district census.
• The number of z-score values that were flagged by the WHO anthropometry macro in Stata, and the difference in months between the age estimated at the first and last visits. For the purpose of analysis, a difference of ±3 months was arbitrarily taken to be acceptable.
• The difference in length or height of each child between the first and last measurements. An acceptable range was taken to be -1 cm to +4 cm. This is a combination of measurement error and rounding (which was evident in the data) of ±1.0 cm; changes in measuring children from supine to standing of 0.7 cm, plus a possible gain in height of up to 2.7 cm rounded up to 3.0 cm. This height gain is the maximum possible gain for a nearly 5 year old boy who is 3 S.D. above the median in height according to WHO growth references. A change greater than 4 cm or less than -1 cm should not have been possible.

Findings of study

Of the 3,544 children enrolled, 483 (13.6%) did not return to collect a fifth ration. Of the 3,061 children who did return, 196 (6.4%) had a different name and 200 (6.5%) had a possibly different name, indicating that perhaps up to 13% of mothers had brought a different child to collect the last ration. There were three names missing.

Figure 2 shows the age distribution of 3,397 children aged 6-59 months whose age was recorded at enrolment compared with the expected age distribution based on the 2009 census in the same three districts¹². The expected number of children aged 6-11 months was estimated by dividing by two the numbers recorded for children aged 0-11 months. Figure 2 shows that there were 89% more children than expected aged 12-23 months and 56% fewer children aged 48-59 months, suggesting a bias towards younger children. Only 93 children (2.63%) were older than 60 months (not shown in Figure 2), which seems unlikely if the entry criterion to the programme was based on a height of <110 cm rather than age and should have included older but stunted children. There were no statistically significant differences in the mean reported age of children enrolled at the first, second or third food distributions.

The WHO macro to calculate anthropometric indices flagged baseline values of weight-forheight, height-for-age and weight-for-age for 237 children (6.67%). of these, 67 (2.56% of the total) were weight-for-height, suggesting that a measurement of weight or height was incorrect. The same values were flagged for fewer children at the fifth food distribution: 113 (3.18%) had any index flagged while 35 (1.17%) had the value of weight-for-height flagged.

Figure 3 shows the distribution of the difference in age in months recorded for 3,061 children at enrolment and at the fifth food distribution, an average of 97 days later (range 16-135 days), depending on when children were enrolled. Only 21.23% of children were recorded as having the same month of age, 23.72% were 1 to 3 months younger or older, 25.22% were 4 months or more younger, and 29.79% were four months or more older. In summary, 44.95% of children were within ±3 months of the same age and 55.05% were =4 or =4 months different in age.

Figure 4 shows the distribution of the difference in height of 3,032 children measured at enrolment and the fifth distribution of whom 66.09% were within a range of >-1 to <4 cm, 15.77% were >1 cm shorter, and 18.14% were =4 cm taller.

Of the 2,640 children who were considered by their name to be the same on both occasions, data on only 902 children (34.17%) were considered to be acceptable based on both their stated age (±3 months different) and length or height (>-1 or =4 cm different) at the two instances they were seen. This meant that data on nearly two thirds of children were of questionable quality. Because of these discrepancies, no further analysis was done to assess the impact of the feeding programme.

Any attempt to estimate the impact of supplementary food on weight gain requires that each child is measured twice, at the start and end of the programme to obtain paired measurements, and that accurate data on age is obtained if anthropometric indices other than weight-for-height are to be calculated. The data collected during the present evaluation in northern Kenya indicated that a large proportion of children were not the same at the first and fifth food distribution and that the age of many children was not given nor estimated consistently and so was probably inaccurate. There are a number of likely reasons for this, including the possibility that mothers did not bring the enrolled child to collect the fifth and final ration, systematic bias or errors in estimating age, inconsistent estimates of age on separate occasions, and errors in making anthropometric measurements, in recording data, and in data entry.

The age distribution shown in Figure 2 is unlike a typical age pyramid and suggests that many mothers, who made up 92% of the caregivers at enrolment, were either not giving or not estimating correctly the age of their child, perhaps to ensure that they obtained a ration of food. The fact that about 40% of all children were either shorter by 1 cm or more or taller by 3 cm or more suggests that a substantial proportion of caregivers did not bring the same child to collect the fifth ration, although some differences could be measurement errors made by busy staff.

It is to be expected that parents will make every effort to obtain a ration of food for their children during a food shortage and the many years of food insecurity in this part of Kenya have led to a degree of dependency on humanitarian assistance. An eligibility threshold of <110 cm in height¹³ was applied to try to eliminate a reason for parents to be untruthful about giving the correct age of their child. This did not seem to work, perhaps because community mobilisers did not understand or make it clear to parents, because mothers did not understand, or because mothers were mistrustful of a different criterion of eligibility for health services from the usual, which is age.

Factors that compromised data quality

Several things compromised data quality:

• The design and implementation was complicated by the request to evaluate the impact in a different a livelihood zone, among a different ethnic group, and by additional NGOs. This increased the sample size, increased the cost, and increased the number of agencies involved from two to six, with consequences for staff training and data quality. As this was the first attempt in Kenya to evaluate the impact of a blanket feeding programme using such methods, it might have been best to focus efforts in two contiguous districts among a single ethnic group and in one livelihood zone, to keep it as simple as possible.
• The pressure to begin distributing rations reduced the time available for training field staff of six NGOs to two days by eight different trainers, adding other factors that may have compromised data quality.
• The personnel doing the evaluation were also responsible for registering and distributing rations to about 500-2,500 beneficiaries at each site during the first food distribution, so the staff were overburdened.
• Supervising the collection of data at food distribution sites spread over an area of 150,000 km2 posed an insuperable problem to the lead NGO. Wajir town is 1,100 km by road to the main town in Turkana. So after being quickly trained, the NGO staff were unsupervised by the lead evaluation agency.

Lessons learned

The experiences described here offer useful lessons that could be applied to improve the quality of data in future evaluations of blanket feeding programmes in Kenya and elsewhere.

Beneficiaries of BSFP in Kenya

First, the evaluation should be put in place as the intervention is being planned so that the evaluation is a part of the programme, not an external component. Both require preparation, swiftness and adequate funding.

Second, community mobilisers need to explain clearly and effectively to potential beneficiaries the criterion for eligibility: height <110 cm, not age <59 months - the usual threshold for health programmes. Ideally this criterion would be used for all programmes because it is simple, objective and transparent, and would include stunted children older than 5y, who could also benefit from most interventions. The disadvantage of using height is that there is no easily calculated denominator to estimate both the numbers of eligible children and coverage, whereas a denominator based on age can be estimated from census data. As an estimate of coverage is an important indicator of the effectiveness of an intervention, a separate survey would be necessary at additional expense.^14,15

Third, every study child ideally should be identified either using a digital photograph or perhaps using a fingerprint reader, either in a personal digital assistant (PDA) or connected to a laptop computer, to confirm their identity at subsequent contacts. Battery powered PDAs could also be used to collect, store and compare data in the field, so that widely differing anthropometric measurements could be flagged and checked immediately. Such devices require a capital outlay, a software programmer and field testing before deployment, which is expensive. But this could improve data quality and speed up the process of analysis and reporting, as well as increasing the validity of the evaluation. If suitable equipment is not available, then key data should be copied onto ration cards to act as a check, including the estimated date of birth and the first height and weight. Neither process would guarantee that the same child is seen on all occasions, but any substitute could be identified on the spot.

Fourth, the staff doing the evaluation should be different from the staff delivering the ration cards, food or other interventions, so that both jobs are done as well as possible in an often chaotic and busy environment in which agitated parents demand attention. A dedicated evaluation team would require additional funding, an issue not addressed here, but the compromised evaluation also represents a waste of resources, as well as the time of staff and mothers. The evaluation staff also require specialised training in anthropometric measurements, even if they have done them many times before, because both accuracy and precision are required and should not be assumed.

Finally, data analysis should be done as quickly as possible in the field, so that systematic errors such as rounding can be identified and rectified by re-training or reorganisation of procedures. If all data are entered in the field onto PDAs, the confirmation of each entry would duplicate the process of double data entry. Data could also be merged from different field teams and analysed quickly in the field using batch files written for statistical software. By reporting the problems and lessons learned from this evaluation of a BSFP, it is hoped that future evaluations will be better planned and implemented and may provide plausible evidence of a benefit to children's nutritional status.

¹Hall A, Oirere M, Thurstans S, Ndumi A, Sibson V, 2011. The Practical Challenges of Evaluating a Blanket Emergency Feeding Programme in Northern Kenya. PLoS ONE 6(10): e26854. doi:10.1371/journal.pone.0026854

²Kenya National Bureau of Statistics (2007). Statistical Abstract Nairobi: Kenya National Bureau of Statistics.

³Save the Children (2009) Blanket supplementary feeding programme monitoring and evaluation guidelines. Nairobi: Save the Children. p58.

⁴See footnote 2

⁵World Health Organization (2000) The management of nutrition in major emergencies. Geneva: World Health Organization. p236.

⁶See Footnote 3

⁷StataCorp (2010) Stata Statistical Software: Release 11.0. College Station, Texas, USA: StataCorp.

⁸ACF USA (2009) Anthropometric and retrospective mortality surveys in the Districts of Mandera, Kenya. NairobiKenya: Action contre la Faim.

⁹See Footnote 3

¹⁰See Footnote 6

¹¹WTO (2011) Macros to analyse growth data for the age group 5-19 years.Geneva: World Health Organization. Available: http://www.who.int/growthref/tools/en/ Accessed 2011 Oct 9.

¹²Kenya National Bureau of Statistics (2011) Kenya National Bureau of Statistics The 2009 Kenya Population and Housing Census.Volume 1C.Population distribution by age, sex and administrative units. Nairobi. 546 p.

¹³See Footnote 4

¹⁴Myatt M, Feleke T, Sadler K, Collins S (2005) A field trial of a survey method for estimating the coverage of selective feeding programmes. Bull World Health Organ 83: 20-26.

¹⁵Sadler K, Myatt M, Feleke T, Collins S (2007) A comparison of the programme coverage of two therapeutic feeding interventions implemented