SQUEAC in routine monitoring of CMAM programme coverage in Ethiopia
By Lily Schofield, Selome Gizaw Lalcha and Terefe Getachew
Lily Schofield has worked in many countries in Africa and Asia as a nutrition consultant. She has been involved in many coverage assessments of selective feeding programmes since 2007. She is currently working as the Evaluation and Research Support Advisor, Concern Worldwide Kenya.
Selome Gizaw Balcha has a BA in Sociology and Social Anthropology and works as a Senior Social Development Officer for the national CMAM programme, Concern Worldwide Ethiopia.
Terefe Getachew has an extensive background in community mobilization and has participated in multiple nutrition surveys and assessments. He is currently working as a Senior Social Development Officer for the national CMAM programme, Concern Worldwide Ethiopia.
The authors would like to thank the Woreda Health Offices and Health Centre Staff and all Health Extension Workers and Community Volunteers who took part in the survey, as well as the mothers and caregivers who participated as respondents.
The development of community-based management of acute malnutrition (CMAM) was a significant advance in the treatment of severe acute malnutrition and associated child mortality. One of its primary innovations was the decentralization of services closer to beneficiary homes by treating children through out-patient therapeutic programmes (OTPs) rather than as inpatients. This translated into the potential for significantly higher coverage because travel time for beneficiaries was shortened and community mobilization improved1. However, in order to achieve this potential, regular monitoring of coverage levels and barriers to coverage are needed so that community outreach and mobilization strategies can be adapted accordingly.
The challenges of monitoring programme coverage
Measuring coverage of CMAM programmes presents special challenges when compared with other health services. Even during acute food emergencies, severe acute malnutrition (SAM) only afflicts a small percentage of the total under 5 population. Therefore, a random sample of all children under 5 is unlikely to include enough SAM children to estimate CMAM coverage with any precision. Secondly, traditional survey methods that rely on sampling proportional to population size may bias coverage estimates upwards, as areas of high population are more likely to be close to main roads and health centres (HCs)2. In response to these challenges, an alternative survey method based on area sampling was developed for assessment of CMAM coverage. This method, called Centric Systematic Area sampling (CSAS), overcomes the biases described above. However it remains a resource and time intensive methodology, typically requiring 12 to 18 enumerators for 7 to 10 days of data collection and significant logistical support. It is therefore a poor tool for routine monitoring of coverage, as only the most well resourced programmes can afford to conduct regular CSAS surveys.
The challenge of monitoring coverage becomes more acute when an integrated CMAM programme is considered. While many of the early CMAM programmes were implemented by non-governmental organisations (NGOs) in emergency settings, more and more countries are beginning to integrate the treatment of SAM into basic preventative and curative health services offered through Ministry of Health (MoH) facilities3. These programmes possess a high potential for sustainability, but are also resource constrained both in terms of staff and finances. The result is that in most integrated CMAM programmes, coverage is never assessed and so the proportion of SAM children being missed is never known. This also means the community mobilization cannot be evaluated and improved.
Valid International and partners recently developed a new methodology (SQUEAC) aimed at integrated CMAM programmes that is less resource intensive and requires only basic technical skills. This methodology is based on a combination of qualitative and quantitative data collection to triangulate information about coverage and barriers to access. This information is then confirmed through small area surveys that used Lot Quality Assurance Sampling (LQAS)4 techniques to confirm or refute initial hypotheses about adequacy of coverage.
Pilot of new methodology in Ethiopia
Ethiopia has long been at the forefront of developments in CMAM. One of the first places where CMAM programmes were initially trialled and the approach developed,5,6Ethiopia is currently working to integrate CMAM into primary health care facilities around the country. As of the beginning of 2008, CMAM services were offered in tertiary health facilities in four regions of the country. Given the scale of integrated CMAM in Ethiopia, there is an immediate need to identify ways of monitoring coverage on a regular basis that can be carried out by the MoH with limited or no external support.
This article describes a pilot test of SQUEAC in one region in Ethiopia to determine the feasibility and functionality of SQUEAC as a tool for use by the MoH to regularly monitor coverage of integrated CMAM services.
The SQUEAC pilot was undertaken in 23 health centres (HCs) across 8 woredas (districts) in Tigray Region, Ethiopia in November, 2008. A second round of SQUEAC surveys were conducted in the same HCs in July, 2009.
Study area
Tigray region is the Northern most region of Ethiopia bordering Eriteria on the East, Sudan on the West and Afar and Amhara regions to the South. A mountainous region, the primary livelihoods are subsistence agriculture, animal husbandry and wage labour. There is also significant reliance on safety nets and food aid among the poorer segments of the population7.
Collecting information of home location of admission and defaulters from OTP cards (Hewane HC)
CMAM activities in Tigray Region began in June 2007 with initial training and support provided by Concern Ethiopia. The programme was launched in nine HCs in eight woredas throughout the region. This was soon expanded to 10 more HCs in 10 additional woredas. In 2009, the regional health bureau (RHB) began further decentralization through a programme of on-the-job training. At the time of the first round coverage assessment, OTPs were functional in 81 HCs in 38 woredas across the region.
Pilot Methodology
The SQUEAC assessment was based on a two stage sampling method. In the first stage, routine programme data such as admissions, recovery rate, defaults, etc, were used to identify a region of each HC catchment area that was likely to have low coverage. This data was also used to generate a list of reasons why coverage may be low in this area.
In the second stage, a small area sample in the suspected low coverage area served to confirm or refute the original hypothesis that coverage was low. Low coverage was defined as below the appropriate SPHERE standard for selective feeding programmes (50% for rural areas, 70% for urban areas and 90% for camps). Teams also collected qualitative information8 from key informants on perceptions of the programme, malnutrition and reasons for non attendance to further triangulate data.
During the first round of SQUEAC, a cascadetraining model was used. MoH staff from all participating HCs as well as representatives from the woreda health offices received a two-day training on the theory and practice of SQUEAC. HC staff then trained Health Extension Workers (HEWs) and Community volunteers (CVs) from their catchment area in data collection for the small area surveys. During the second round a one-day refresher orientation was given to MoH staff from town administrations, woreda health offices and HCs.
Selection of woredas/Health Centres for pilot
All woredas with >50% of their area falling within the catchement of a HC offering OTP services were included in the pilot. Within these woredas, all HCs offering CMAM and who had received support from Concern's National CMAM programme participated in the assessment.
Sampling for small area surveys
Once the suspected area of low coverage was identified during the first stage of SQUEAC, a random sample was taken from the villages in that area for the small area sample. Estimates of malnutrition prevalence and average village size were used to calculate the number of villages to sample in order to find a minimum of 10 SAM cases9. Experience from the first round showed that due to large village sizes in Tirgray, sub-villages (a smaller population unit whose boundaries are locally known, but often are not listed in government records) were a more useful sampling unit. Subvillages were therefore used as the sampling unit in all HCs during the second round.
Within village sampling was done in one of two ways in the first round; active and adaptive case finding10 or house-to house screening of all children 6 to 59 months in selected villages/subvillages. Based on first round experience, it was decided that house-to-house screening was the most effective method. HEWs and CVs were already familiar with this technique from immunisation campaigns and could complete a village/sub-village in a reasonable timeframe. House-to-house was therefore used by all teams in the second round.
SAM was defined according to programme admission criteria as a child whose mid upper arm circumference (MUAC) was less than 11 cm and/or the presence of bilateral pitting oedema.
Data collection
Individual HCs were in charge of organizing their teams for data collection. On each team, a HEW was in charge of leading the team and recording all data on the data collection forms. The CVs on the team were responsible for MUAC measurement and oedema assessment.
For all SAM cases found that were not currently attending OTP, a questionnaire was administered to the caretaker to determine the reasons for non-attendance. All uncovered cases were also referred to the nearest OTP for treatment.
Data analysis
Data analysis was conducted using a simple compilation sheet that summed the information gathered from all villages within a HC catchment area. The total number of SAM cases found in all villages surveyed and the total number of covered cases was tallied. The achieved sample size (the number of SAM cases actually found in the survey) was then used to calculate a decision value for the survey. The following rule of thumb formula was used to calculate the decision value:
D= [N x p/100]
where p was the target coverage proportion (for example 50% in rural areas).
If the number of covered SAM cases was greater than the decision value, then coverage in the area surveyed was classified as above the target. If the total number of covered cases was less than or equal to the decision value, then coverage was classified as below the target. Defaulted cases and relapsed cases were considered uncovered because they were not attending OTP at the time of the survey.
Results from qualitative data collection (focus groups and semi-structured interviews) were summarised by main points. These points were compared to the results of the structured questionnaires collected during the small area surveys. The information from both the qualitative data collection and the small area questionnaires were used to identify barriers to coverage. Action points to address these barriers were then outlined by health centre staff.
Evaluation of the pilot
To assess the success of the pilot and the usefulness of the methodology for MoH use in Tigray region, a series of debriefings were held to capture lessons learned. First, the staff at several HCs were interviewed about their experience and opinions of the SQUEAC methodology and process. Secondly, a debriefing was held with Concern staff who supervised the survey and lastly, a debriefing was held with representatives from the federal and regional MoH and Concern nutrition staff to present results and brainstorm about next steps. A follow-up debriefing was held after completion of the second round with Concern staff who oversaw implementation of the second round.
The pilot test was largely successful with all 23 participating HCs able to obtain information about CMAM coverage and barriers to services during the first round and 22 HCs in the second round.
Implementation
Implementation of the entire SQUEAC assessment took six days in the first round and five days in the second round, including training but excluding travel days. Fieldwork for small area surveys and qualitative data collection took two days within each health centre area. During the first round, a large number of data collectors were used to minimize data collection time. However it became clear that these high numbers were not required for efficient data collection. Therefore numbers were reduced significantly in the second round (see Figure 1), with no significant negative impact on data collection time or quality.
Outcomes
In the 23 HC catchments surveyed in the first round, all except one were below their target coverage level (50% for rural areas and 70% for one town administration) (Table 1). In the second round, three additional HCs had increased their coverage to above their target level.
| Table 1: Results for standard indicators measured on IYCF practices | ||||||||
| First round - November 2008 | Second round-July 2009 | |||||||
| Health centre | # cases | # covered | Decision value | Coverage | # cases | # covered | Decision value | Coverage |
| E/Bahre | 22 | 6 | 11 | <50% | 16 | 4 | 8 | <50% |
| Abi Adi | 9 | 1 | 4 | <50% | 7 | 2 | 3 | <50% |
| Wukro Mariam | 19 | 2 | 9 | <50% | 11 | 2 | 5 | <50% |
| Chila | 9 | 0 | 4 | <50% | 7 | 2 | 3 | <50% |
| Mayasmi | 20 | 1 | 10 | <50% | 18 | 2 | 9 | <50% |
| Hayelom | 9 | 0 | 4 | <70% | 19 | 1 | 9 | <70% |
| E/Homus | 28 | 6 | 14 | <50% | 2 | 1 | 1 | <50% |
| Wukro town | 4 | 0 | 2 | <50% | 3 | 1 | 1 | <50% |
| Dinglet | 13 | 2 | 6 | <50% | 0 | 0 | - | <50% |
| Zala | 26 | 7 | 13 | <50% | 12 | 1 | 6 | <50% |
| Sobeya | 16 | 4 | 8 | <50% | 18 | 3 | 9 | <50% |
| Fatsi | 15 | 0 | 7 | <50% | 18 | 3 | 9 | <50% |
| Zalambesa | 4 | 0 | 2 | <50% | 5 | 4 | 2 | >50% |
| Freweni | 14 | 1 | 7 | <50% | 5 | 2 | 2 | <50% |
| Abi Kebeles | 22 | 9 | 11 | <50% | NA | NA | NA | <50% |
| Dengolat | 11 | 0 | 5 | <50% | 6 | 2 | 3 | <50% |
| Mehoni | 5 | 4 | 2 | >50% | 5 | 3 | 2 | >50% |
| Kukuftu | 4 | 2 | 2 | <50% | 8 | 5 | 4 | >50% |
| Hreko | 7 | 1 | 3 | <50% | 4 | 2 | 2 | <50% |
| Korem | 8 | 3 | 5 | <70% | 0 | 0 | - | <70% |
| Hiwane | 7 | 1 | 3 | <50% | 7 | 0 | 3 | <50% |
| Adi Godum | 3 | 1 | 1 | <50% | 6 | 4 | 3 | >50% |
| Ado Keyeh | 4 | 1 | 2 | <50% | 5 | 0 | 2 | <50% |
NA=data collection postponed due to schedule conflict with training for community based-nutrition
The main reasons for non-coverage were largely the same in both rounds and are presented in Figure 2. In both rounds, lack of awareness of the CMAM programme was the primary barrier to coverage, followed closely by distance and lack of awareness about the signs and symptoms of malnutrition. While awareness of CMAM and of malnutrition requires further improvement, there is evidence that the action steps implemented by the ministry following the first round had a positive impact. The percentage of mothers with severely malnourished children surveyed who did not know about the programme decreased from 39.2% to 21.7% in the second round. Similarly, the percentage of caregivers who did not think their child was malnourished decreased from 14.5% in the first round to 10.9% in the second round (Figure 2).
Uptake by MoH actors/Sustainability
There was a high level of interest and investment by the MoH staff, particularly at a regional and woreda level. The identification of barriers to coverage and implementation of action steps to address these barriers was prompt. Before commencement of the second round, Concern held discussion with the woreda health offices about decreasing the financial and logistic support in the second round and eliminating financial support in the third round. There was a high level of ownership of the process at the woreda level, though HC staff still expressed a need for external support to complete any future assessments.
There was a low rate of staff turnover in the seven months between the first and second rounds, facilitating a rapid refresher for HC staff in the second round. Unfortunately, low staff turnover is not the norm for Ethiopia. Increased inputs in the form of refresher training may be required before successive rounds of SQUEAC, if it is used in other areas of the country.
Discussion
The methodology was well understood by the MoH and was successfully implemented in a short period of time. SQUEAC assessments required much less time and resources than the traditional CSAS method. As evidenced by the positive changes seen in coverage in several of the surveyed areas, the assessment also provides information on barriers to access and coverage that can be translated into effective remedial actions.
Challenges
A challenge that emerged in the second round was when an area sample found no severely malnourished children. Because the methodology identifies a relatively small area for sampling, focusing time and resources on suspected areas of low coverage carries the risk, particularly in areas were SAM prevalence is low, that no children eligible for OTP will be found. If no children are identified, then coverage cannot be classified as above or below the target. One possible solution to this would be to select backup villages in areas were the SAM prevalence is low. If the first sample did not find any SAM cases, then sampling could continue through the replacement villages. While this would require more time for data collection it would ensure that coverage could be classified.
While the methodology worked well in rural areas, it was less robust in the urban context. Three of the 23 HCs included in this pilot were town administrations, which cover a very small geographic area but have high population densities relative to the rural areas. The urban centres also tended to have lower malnutrition rates, which had implications for sample size. Because of this, identifying a sub-area of low coverage within the town administrations was not meaningful. In the three towns included, the entire town was taken as the sampling frame. Even when the entire town was taken as a sampling frame, identifying meaningful sampling units was difficult. Divisions within towns were largely administrative and did not correspond to population groupings, mixing both business districts and homes. Trend analysis and beneficiary home mapping were also more difficult to interpret in towns as only a handful of cases attending OTP at the town HCs came from the town itself. The majority of attendees came from outside the catchment area of the HC.
This pilot also highlighted that given the health workers limited experience with qualitative data methods such as focus groups and key informant interviews, a key component that was missed during the first round assessment was adequate training and guidance on conducting the qualitative part of the assessment. This was addressed in the second round by developing a short worksheet. This worksheet helped ensure uniformity of qualitative data collection and helped enumerators plan the qualitative work.
Even with these challenges, the assessments provided clear indications of where the community mobilisation systems were breaking down. Based on this, health staff were able to define immediate actions steps to improve coverage.
Conclusions
This methodology has the potential to become a useful tool for routine management and monitoring of CMAM activities by the MoH. There was a high level of ownership at the woreda level and HC staff were able to carry out the assessments and utilise the resulting information to address barriers to coverage.
While resource requirements for these assessments were minimal compared to alternative coverage survey methods, some costs remain. Whether these costs can be successfully incorporated into MoH budgets and a regular schedule of coverage assessments maintained without external support remains to be seen. However, this pilot clearly demonstrates that providing simple, easy to use tools for programme assessment that can be implemented with limited resources and external support is key to improving the quality of integrated CMAM programmes.
For more information, contact: Lilly Schofield, lilly.schofield@gmail.com
1Collins S, Dent N, Binns P, Bahwere P, Sadler K, Hallam A. Management of severe acute malnutrition in children. Lancet 2006. Dec 2; 368(9551): 1992-2000.
2Myatt M, Feleke T, Sadler K, Collins S. A field trial of a survey method for estimating the coverage and selective feeding programmes. Bull World Health Organ 2005 Jan; 83(1): 20-6.
3Gatchell V, Forsythe V, Thomas PR. The sustainability of community-based theraputic care (CTC) in non-emergency contexts Food Nutr Bull 2006 Sep 27; 27(3):S90-S98.
4Myatt M. SQUEAC: Low resource method to evaluate access and coverage of programmes. Field Exchange 2008 Jun; 33: 3-5
5Collins S, Sadler K. Outpatient care for severely malnourished children in emergency relief programmes; a retrospective cohort study. Lancet 2002 Dec 7; 360: 11770-11773.
6Chaiken M, Deconinck H, Degefie T. The promise of a community-based approach to managing severe malnutrition: A case study from Ethiopia. Food Nutr Bull 2006 Jun 27;27(2): 95-104.
7Tigray Region Livelihoods Profiles: Regional Overview. Livelihood Integration Unit. DPPA, January 2008.
8Though ideally qualitative data collection happens before areas are chosen for small area surveys, because of the impending EOS screening and other schedule constraints, qualitative data collection and small area surveys were conducted simultaneously.
9Personal communication with M. Myatt 10 See footnote 2.
Imported from FEX website