Using SLEAC as a wide-area survey method

By Ernest Guevarra, Saul Guerrero, and Mark Myatt

Ernest Guevarra leads Valid International's coverage assessment team. He has formal training as a physician and a public health practitioner and invaluable informal training as a community worker from the communities with whom he has worked. He has worked in the Philippines, Uganda, Somalia and Sri Lanka.

Saul Guerrero is the Evaluations, Learning and Accountability (ELA) Advisor at Action Against Hunger (ACFUK). Prior to joining ACF, he worked for Valid International Ltd. in the research, development and roll-out of CTC/CMAM. He has worked in many countries in Asia and Africa

Mark Myatt is a consultant epidemiologist and senior research fellow at the Division of Opthalmology, Institute of Opthalmology, University College London. His areas of expertise include infectious disease, nutrition and survey design.

The authors would like to thank the Sierra Leone Ministry of Health and Sanitation for managing the survey and for allocating key ministry personnel to conduct the survey, Statistics Sierra Leone for providing relevant data and appropriate maps used during the survey and UNICEF Sierra Leone for funding the survey. The authors' appreciation also goes to the people of Sierra Leone, without whose support and assistance this survey would not have been possible.

SLEAC stands for Simplified LQAS¹ Evaluation of Access and Coverage. It is a quick and simple method for assessing coverage in a programme area such as a health district.

In 2010, UNICEF approached VALID International Ltd. to design and conduct a national coverage survey of the government-run community based management of acute malnutrition (CMAM) programme in Sierra Leone. Discussions with UNICEF and the Sierra Leone Ministry of Health indicated that a spatially exhaustive set of SLEAC surveys (i.e. a SLEAC survey performed in every health district) augmented by one or two targeted SQUEAC investigations would provide the information needed by both UNICEF and the Sierra Leone Ministry of Health. The idea of using the two methods together in this way is to use SLEAC to identify district programmes achieving low and high coverage and to use SQUEAC to investigate the reasons for the observed levels of coverage. Two variants of this model are outlined in Figure 1 and Figure 2.

This article describes how we used the SLEAC method to perform a wide-area coverage survey of the national CMAM programme in Sierra Leone. It also describes the SLEAC method in general terms.

The SLEAC method described: The simplified LQAS classifier

The SLEAC method classifies programme coverage for a service delivery unit such as a health district. A SLEAC survey does not provide an estimate of overall coverage with a confidence interval for a single service delivery unit. Instead, a SLEAC survey identifies the category of coverage (e.g. low, moderate, or high) that best describes the coverage of the service delivery unit being assessed. The advantage of this approach is that relatively small sample sizes (e.g. n = 40) are required in order to make accurate and reliable classifications.

SLEAC uses the same simplified LQAS (Lot Quality Assurance Sampling) classification technique that is used in SQUEAC small-area surveys. The differences between how the simplified LQAS classification technique is used in SQUEAC and SLEAC are:

• The SLEAC survey sample is designed to represent an entire district rather than a small area.
• SLEAC surveys have no prior hypothesis regarding coverage. This means that SLEAC surveys require larger sample sizes than SQUEAC small area surveys.
• A target sample size for SLEAC surveys is decided in advance of data-collection. This is usually about n = 40 severe acute malnutrition (SAM) cases.
• SLEAC surveys may classify coverage into three (or more) classes.

Analysis of data using the simplified LQAS classification technique involves examining the number of cases found in the survey sample (n) and the number of covered cases found:

• If the number of covered cases found exceeds a threshold value (d) then coverage is classified as being satisfactory.
• If the number of covered cases found does not exceed this threshold value (d) then coverage is classified as being unsatisfactory.

The threshold value (d) depends on the number of cases found (n) and the standard (p) against which coverage is being evaluated. A specific combination of n and d is called a sampling plan. The following rule-ofthumb formula may be used to calculate a suitable threshold value (d) for any standard (p) and any sample size (n):

For example, with a sample size (n) of 40 and a standard (p) of 70% the appropriate value for d would be:

It is unlikely that a SLEAC survey will return the target sample size (n) exactly. If a survey does not return the target sample size (n) exactly then the classification threshold value (d) should be recalculated using the achieved sample size. For example:

Target sample size : 40

Achieved sample size : 43

Standard : 70%

Coverage is classified using the same technique as is used for SQUEAC small-area surveys. For example:

n : 43

d : 30

Covered cases found : 34

Coverage classification : Satisfactory (since 34 > 30)

The simplified LQAS classification technique provides binary or two-tier classifications. The method is usually extended to provide more granular classifications in SLEAC surveys. Three classes are sufficient for most SLEAC applications. Three-tier classifications require two sampling plans which are created using the rule-of-thumb formula presented earlier.

For three-tier classifications there are two coverage proportions:

p1: The upper limit of the ‘low coverage’ tier or class
p2: The lower limit of the ‘high coverage’ tier or class The ‘moderate coverage’ class runs from p1 to p2. For example:

Two classification thresholds (d1 and d2) are used and are calculated as:

Classifications are made using the algorithm illustrated in Figure 3.

This three-tier classification works well with small sample sizes (e.g. n = 40) provided that the difference between p1 and p2 is greater than or equal to about 20 percentage points.

Here is an example of the calculations required:

Sample size (n) : 40

p1 : 30%
p2 : 70%

Figure 4 shows a nomogram for finding appropriate values for d1 and d2 given n, p1 and p2 without the need for calculation.

Classifications are made using the algorithm illustrated in Figure 3.

If a survey does not return the target sample size (n) exactly then the classification thresholds (d1 and d2) should be recalculated using the achieved sample size. For example, a set of SLEAC surveys classifying coverage in individual clinic catchment areas using a target sample size of forty SAM cases (n = 40) for each catchment area and the class boundaries p1 = 30% and p2 = 70% returned the data shown in Table 1. The target sample size was applied to each of the clinic catchment areas separately. This allowed coverage classifications to be made for individual clinic catchment areas. These coverage classifications could be presented as a map as in Figure 5.

Table 1: Results of SLEAC surveys in eight clinic catchment areas using a target sample size of forty (n = 40) cases for each catchment area and the class boundaries p1 = 30% and p2 = 70%
Clinic catchment area	Sample size	d1*	d2*	Number of covered cases	Classification
Chawama	38	11	26	29	High
Matero	32	9	22	18	Moderate
Makeni	43	12	30	36	High
Chipata	35	10	24	15	Moderate
Ngombe	42	12	29	14	Moderate
Kalingalinga	37	11	25	10	Low
Mtendere	39	11	27	5	Low
Kanyama	42	12	29	23	Moderate
All	308	92	215	150	Moderate

* d1 and d2 calculated after data collection using achieved sample sizes

SLEAC can estimate coverage over several service delivery units. Coverage is classified using SLEAC surveys in individual service delivery units. Data from the individual service delivery units are then combined and coverage for this wider area is estimated from the combined sample. The details of calculating a wide-area estimate from a set of SLEAC surveys are not covered in this article.

The SLEAC method described: Sample Design

SLEAC uses a two-stage sample design:

First stage sampling method: This is the sampling method that is used to select villages to be sampled. CSAS coverage assessments use the centric systematic area sampling or quadrat method to select villages to be sampled (Figure 6). A similar method is often used to select villages to be sampled for SLEAC surveys. The number of quadrats drawn on the map may be much smaller than would be used for a CSAS assessment (this is the same as using larger quadrats). The villages to be sampled are selected by their proximity to the centre of each quadrat (Figure 7). The number/size of quadrats should be selected so as to spread the sample of villages over the entire survey area. Another approach is to stratify by clinic catchment area and systematically select villages from a complete list of villages sorted by clinic catchment area (Figure 8). This approach may be used with any areas (e.g. administrative areas) for which complete lists of villages are available. This first stage sampling method should be a spatial sampling method that yields a reasonably even spatial sample from the survey area. Cluster sampling using population proportional sampling (PPS), such as that used for SMART surveys, is not appropriate. The approaches outlined here can provide a reasonably even spatial sample using village lists and do not require the use of maps. It is important to note that sampling should not stop when the survey has reached its required sample size. Sampling only stops after you have sampled all of the selected villages.

A within-community sampling method: This will usually be an active and adaptive case-finding method or a house-to-house census sampling method (see Box 1). These methods find all, or nearly all, current and recovering SAM cases in a sampled village. Sampling should be exhaustive. This means that you only stop sampling when you are sure that you have found all cases in the community. Sampling should not stop when you have met a quota or the wider survey has reached its required sample size.

 

 

This is a two-stage sample because a sample of villages in the survey area is taken first (stage one) and then a ‘census’ sample of current and recovering SAM cases is taken from each and every one of the selected villages (stage two).

The SLEAC method described: Sample size

SLEAC uses a target sample size (n) which, together with prevalence and population estimates, is used to decide the number of villages (nvillages) that should be sampled in order to achieve the target sample size. A target sample size of n = 40 cases from each service delivery unit in which coverage is to be classified is usually large enough for most SLEAC applications.

The target sample size (n) together with estimates of the prevalence of severe acute malnutrition (SAM) in the survey area and population data is used to calculate the number of villages (nvillages) that will need to be sampled in order to achieve the target sample size:

SAM prevalence refers to the average SAM prevalence in the catchment area of the service delivery unit. It is unlikely that SAM prevalence will be known or known with good precision. SAM prevalence estimates may be available from (e.g.) previous SMART surveys. SAM prevalence varies throughout the year (e.g. prevalence is usually higher before harvests than after harvests). This means that you should use the results from a nutritional anthropometry survey undertaken at the same time of year as the current SLEAC assessment.

Note that prevalence here is the estimated prevalence of the programme's admitting case-definition. This will usually not be the weight-for-height based ‘headline’ prevalence estimate reported by a SMART survey. The required estimate will usually be found in the needs assessment section of a SMART survey report.

If you do not have nutritional anthropometry survey results for the same time of year as the current SLEAC assessment then you should use results from the most recent nutritional anthropometry survey and adjust them using (e.g.) seasonal calendars of human disease, calendars of food-availability, agricultural calendars, long term admissions data from nutrition programs, and long term returns from growth monitoring programmes.

The formula for the calculation of the minimum number of villages that need to be sampled in order to achieve the required sample size shown above assumes that the case-finding method being used will find all, or nearly all, current and recovering SAM cases in sampled villages. If you are unsure of this then you should sample a larger number of villages.

Once these calculations have been made, sampling locations can be identified and the survey undertaken (e.g. as shown in Figure 6 and Figure 8). A standard questionnaire, such as that shown in Box 2, should be applied to carers of non-covered cases found by the survey. Data collected using the standard questionnaire can be presented using a Pareto chart (i.e. a bar-chart with bars ordered by and with lengths proportional to the frequency of the reported barriers).

Background to the Sierra Leone national SLEAC survey

The CMAM approach to treating cases of severe acute malnutrition (SAM) in government health facilities was piloted in four districts of Sierra Leone in 2008. The programme was expanded to provide CMAM services in selected health centres in all fourteen districts of the country in 2010. This report describes the application of SLEAC to the assessment of the coverage of this national CMAM programme. The work reported here took place in March and April 2011.

SLEAC sample design

SLEAC was used as a wide-area survey method to classify coverage at the district level. The district was selected as the unit of classification because service delivery of the national programme was managed and implemented at the district level.

The primary sampling units (PSUs) used in the SLEAC surveys were census enumeration areas (EAs). In rural districts, EAs were individual villages and hamlets. In urban and peri-urban districts, EAs were cityblocks or compounds. In rural districts, lists of potential PSUs were sorted by chiefdom. In urban and peri-urban districts, lists of potential PSUs were sorted by electoral ward (sections). This approach ensured a near even spatial spread of the selected villages across rural districts and a near even spatial spread of selected EAs across urban and per-urban districts. The structure of the district-level samples is shown in Figure 9.

A target sample size of n = 40 current SAM cases was used in both rural and urban districts. This is the standard SLEAC sample size for large populations.

The number of PSUs (nPSU) needed to reach the target sample size in each district was calculated using estimates of average EA population and SAM prevalence using the following formula:

Average EA population was estimated as:

using data from the most recent (2004) Sierra Leone Population and Housing Census.

The percentage of the population aged between 6 and 59 months was estimated as 17.7%. This is a national average taken from the Sierra Leone 2004 Population and Housing Census. This estimate is used by Sierra Leone government departments, UN agencies, and NGOs.

SAM prevalence rates were taken from reports of SMART surveys of prevalence in each district that had been undertaken in the lean period of the previous year. The prevalence of SAM using MUAC and oedema was used since this matched programme admission criteria.

The Sierra Leone Central Statistics Bureau provided information on the total district populations and total number of EAs in each district. The Sierra Leone Central Statistics Bureau also provided lists of enumeration areas for the Western Area (urban and peri-urban) districts and largescale maps (see Figure 10) of the EAs that were selected for sampling.

PSUs were selected using the following systematic sampling procedure:

Step 1: The lists of EAs were sorted by chiefdom for rural districts and by section for urban and peri-urban districts.

Step 2: A sampling interval was calculated using the following formula:

Step 3: A random starting PSU from the top of the list was selected using a random number between one and the sampling interval. The random number was generated by coin-tossing (see Box 3 for details).

 

The PSUs selected by this procedure were sampled using a case-finding method tailored to the particular district:

• In rural districts, a district-specific case-finding question was developed from the base case-finding question:

Where can we find children who are sick, thin, have swollen legs or feet, or have recently been sick and have not recovered fully, or are attending a feeding programme?

This question was adapted and improved using information collected from traditional birth attendants, female elders, traditional health practitioners, carers of children in the programme, and other key informants to include local terms (in all local languages) and local aetiological beliefs regarding wasting and oedema. This question was used as part of an active and adaptive case finding method (see Box 1).

• In urban and peri-urban districts, house-to-house and door-to-door case-finding was used. This was done because it was felt that active and adaptive case-finding would not work well in these districts. Sampling was aided by the use of large-scale maps showing enumeration area (EA) boundaries (see Figure 10).

After all of the selected PSUs in a district had been sampled, the survey team met at the district headquarters for data collation and analysis. The simplified LQAS classification technique was applied to the collated data. Coverage standards:

Low coverage: Below 20%.
Moderate coverage: Between 20% and 50%.
High coverage: Above 50%

were decided centrally by MoH and UNICEF staff before the start of the surveys. These standards were used to create decision rules using the rule-of-thumb formulae:

where n is the sample size achieved by the survey, p1 is the lower coverage threshold (i.e. 20%), and p2 is the upper coverage threshold (i.e. 50%).

Coverage in each district was classified using the algorithm presented in Figure 3.

Table 2 presents the results of the surveys.

 

Figure 11 presents the same results as a map of per-district coverage.

A short questionnaire, similar to that shown in Box 2, asking about barriers to coverage was administered to carers of non-covered cases found. Data were tabulated from the questionnaires using a tally-sheet and presented as a Pareto chart (see Figure 12A and Figure 12B).

The SLEAC implementation process

The process as described above was completed in eight weeks (44 working days) staffed by fifteen mid-level health management staff and a principal surveyor provided by VALID International Ltd.

Three survey teams with five members each were used. The teams were divided into two sub-teams. A survey team was headed by a ‘captain’ who was in charge of managing the subteams, organising travel and survey logistics, and co-ordinating survey activities with the principal surveyor.

Each district was divided into three segments. Segmentation was informed by logistics with each segment being served by a road (when possible).

Each survey team was assigned to one of the three segments and provided with:

• A list of PSUs (sorted my chiefdom) to be sampled.
• A list of the locations of CMAM programme sites.
• A list of the names and home villages of chiefs and chief's assistants for each PSU to be sampled.

Each survey team started case-finding in the farthest PSU and then moved to the next-farthest PSU for case-finding and so-on. At the end of each day, the survey teams lodged in health centres, local guesthouses, or in villagers' homes. They restarted case-finding on the following day. This continued until all the PSUs had been sampled. The survey teams then came together at the district headquarters for data collation and analysis and results were shared with district-level health management staff.

Upon completion, the survey team was able to:

• Classify coverage in each district (Table 2)
• Map coverage by district for the whole country (Figure 11)
• List barriers to coverage ranked by their relative importance (Figure 12A and Figure 12B)

An overall coverage estimate was calculated but not reported. Figure 13 shows the calculation of the overall coverage estimate using spreadsheet software.

A single SQUEAC investigation was carried out in the peri-urban ‘Western Rural’ district. This district was chosen because a large number of cases were found in the SLEAC survey, coverage was low, and it was conveniently close to where the survey team was at the end of the SLEAC surveys. Figure 14 shows a concept map summarising the key findings of the SQUEAC investigation.

Conclusion

The work reported in this article supports the use of SLEAC for the assessment of coverage of CMAM programmes over wideareas up to national scale. The application of the SLEAC survey method in Sierra Leone proved to be easy to set-up and supervise. The simplified LQAS technique for classifying coverage and the use of tally-sheets to analyse and present ‘barriers’ data allowed the survey team to analyse results of the survey for each district as soon as each SLEAC survey was completed. Feedback was immediately provided by the survey team to district Ministry of Health officials on their programme's level of coverage and the barriers to access and service uptake. Such ‘real-time’ analysis and reporting of results is unique to SLEAC and has the potential for real-time action and programme reform to be implemented. These findings demonstrate the usability of the method by Ministry of Health staff and make SLEAC the coverage assessment method of choice when evaluating the coverage of CMAM programmes at a regional or national level.

For further information, contact: mark@brixtonhealth.com

---

¹Lot Quality Assurance Sampling