SQUEAC: Low resource method to evaluate access and coverage of programmes

By Mark Myatt

Mark Myatt is a consultant epidemiologist and senior research fellow at the Division of Ophthalmology, Institute of Ophthalmology, University College London. His areas of expertise include infectious disease, nutrition, and survey design.

The author would like to acknowledge the contributions of VALID International Ltd, Concern Worldwide, World Vision, and UNICEF to this work.

Centric Systematic Area Sampling (CSAS) was developed to estimate coverage of selective feeding programmes. It provides an overall estimate and a spatial distribution map of programme coverage, and a ranked list of programme-specific barriers to service access and uptake. As CSAS is resource intensive, it tends to be used in programme evaluation rather than in planning. This means CSAS findings arrive too late in the programme cycle to institute effective remedial action. In addition, the community-managed model of service delivery is now being adopted in developmental and post-emergency settings, which tend to suffer from considerable resource-scarcity, compared to emergency response programmes implemented by nongovernmental organisations (NGOs). There exists, therefore, a need for a low-resource method capable of evaluating programme coverage and identifying barriers to service access and uptake. This article describes some aspects of such a method¹ currently being developed by VALID International, in collaboration with Concern Worldwide, World Vision, and UNICEF.

Outline of the proposed method for evaluating access and coverage

The method is called SQUEAC (Semi- Quantitative Evaluation of Access and Coverage) and uses a two-stage screening test model:

STAGE 1: Identify areas of probable low and high coverage, and reasons for coverage failure using routine programme data, already available data, quantitative data that may be collected with little additional work, and anecdotal data.

STAGE 2: Confirm the location of areas of high and low coverage and the reasons for coverage failure identified in Stage 1 using small-area surveys.

Data sources and their methods of analysis (STAGE 1)

Routine programme data

Experiences with Community Therapeutic Care (CTC) programmes in a variety of emergency settings show that programmes with reasonable coverage show a distinctive pattern in the plot of admissions over time. As can be seen in Figure 1, the number of admissions increases rapidly. These may then fall away slightly before stabilising and finally dropping away, as the emergency abates and the programme is scaled down and approaches closure. Major deviations from this pattern in the absence of evidence of (e .g.) mass migration or significant improvements in the health, nutrition, and food-security situation of the programme's target population indicates a potential problem with a programme's recruitment procedures. For example, Figure 2 shows a plot of admissions over time in an emergencyresponse CTC programme that had neglected to undertake effective community mobilisation and outreach activities. Admissions initially increased rapidly and then fell away rapidly. Such a pattern is indicative of a programme with limited spatial coverage, relying on self-referrals.

The pattern of admissions in a developmental setting is likely to be more complicated and, once the programme has been established, should vary with the prevalence of acute undernutrition. Making sense of the plot of admissions over time in such settings requires information about the probable prevalence of acute undernutrition. This can be determined using seasonal calendars of human diseases associated with acute undernutrition in children (e.g. diarrhoea, fever, and acute respiratory infection (ARI)) and food availability. Figure 3 shows a plot of admissions over time with seasonal calendars of human diseases and food availability. Deviation from the expected pattern indicates a potential problem with a programme's recruitment procedures.

Using admissions over time ignores the problem of defaulters. Defaulters should be in a programme and are, by definition, coverage failures. Figure 4 shows a standard programme indicator graph from a CTC programme. This graph shows an increasing defaulting rate. This was due to the programme having too few Outpatient Therapeutic (OTP) sites. More cases were found and admitted as the programme's outreach activities were expanded. However, more of these cases defaulted after the initial visit because beneficiaries and carers had to walk too far to access services.

The home location of the beneficiary is usually recorded on the beneficiary record card. Mapping the home locations of beneficiaries attending each OTP site is a simple way of defining the actual (rather than the intended) catchment area of each OTP site. Figure 5, for example, shows the home location of each beneficiary, attending an OTP site, that was admitted to the programme in the previous two months. This plot suggests that the programme has limited spatial coverage with coverage restricted to areas close to OTP sites or along the major roads leading to OTP sites.

Mapping is also a useful way of assessing outreach activities. It is also useful to map the home locations of defaulting cases. Figure 6, for example, shows the home locations of beneficiaries who defaulted in the previous two months. Most defaulting cases come from villages far from the OTP site suggesting that lack of proximity to services (either to the OTP site or to outreach and support services) is a leading cause of defaulting. It may also be useful to record and map DNA (did not attend) referrals (i.e. probable current cases that did not attend the programme despite having been referred to the programme) that you find by referral monitoring (see below). Following-up of defaulting and DNA cases (i.e. with home visits) should also be undertaken in order to identify reasons for defaulting and non-attendance.

All of the mapping work outlined in this article can be performed with a paper map of useful scale, clear acetate sheets, adhesive masking tape, Post-itT notes, and marker pens. Figure 7, for example, shows a coverage assessment worker mapping the home locations of beneficiaries attending an OTP site.

DNA referrals are more likely than defaulters to be current cases. This means that high DNA rates are associated with low programme coverage. DNA rates can be calculated by monitoring referrals. Mapping of DNA cases can provide information about problems of proximity to services and other barriers to service access and uptake that may also be spatially distributed (e.g. ethnic or religious groups). Defaulting and DNA rates may also be analysed (classified) using the Lot Quality Assurance Sampling (LQAS) technique presented later in this article. The SPHERE minimum standard for defaulting rates is 15% (maximum). This standard may also be used for DNA rates.

Anecdotal data

Three methods of collecting anecdotal data from a variety of sources are used in SQUEAC assessments. These are:

1. Informal group discussions with:
   • Carers of children attending OTP sites.
   • Relatively homogenous groups of keyinformants (e.g. community leaders and religious leaders) and lay-informants (e.g. mothers and fathers).
   • Programme staff.
2. Semi-structured interviews with keyinformants such as:
   • Programme staff.
   • Clinic staff.
   • Community-based informants such as schoolteachers, traditional healers, and traditional birth attendants.
   • Carers of defaulting and DNA cases.
3. Simple structured interviews, undertaken as part of routine programme monitoring and during small-area surveys, with:
   • Carers of defaulting and DNA cases.
   • Carers of non-covered cases found by small-area surveys.

Other methods of collecting anecdotal data (e.g. formal focus groups and more structured and indepth interviews) may also prove useful in some contexts. The collection of anecdotal data should concentrate on discovering reasons for both nonattendance and defaulting.

Validating and analysing anecdotal data

It is important that the collected anecdotal data is validated. In practice, this means that data is collected from as many different sources as possible. Data sources are then cross-checked against each other. This process is known as triangulation. The data collected from routine programme data and anecdotal data, when combined, provide information that can be considered as a set of hypotheses that can be tested. The SQUEAC method uses small-area surveys to confirm or deny these hypotheses. Hypotheses about coverage should be stated before undertaking small-area surveys. Hypotheses about coverage will usually take the form of identifying areas where the combined data suggest that coverage is likely to be satisfactory and areas where the combined data suggest that coverage is likely to be unsatisfactory.

Figure 8, for example, shows an area of probable low coverage identified by mapping home locations, analysis of outreach activities, defaulter follow-up, and anecdotal data. The hypothesis about coverage in this area was that coverage was below the SPHERE minimum standard for coverage of therapeutic feeding programmes in rural settings of 50% due to:

• A mismatch between the programme's definition of malnutrition (i.e. anthropometric criteria and problems of food security) and the community's definition of malnutrition (i.e. as a consequence of illness, particularly diarrhoea with fever).
• Patchy coverage of outreach services particularly with regard to the ongoing follow-up of children with marginal anthropometric status.
• Distance to OTP sites and other opportunity costs.

A small-area survey was undertaken in this area to confirm this hypothesis. The findings of the small-area survey confirmed, in general terms, the hypothesis under test. It also identified a problem with the application of case-definitions leading to some cases being admitted to the wrong programme.

Small-area surveys (STAGE 2)

SQUEAC small-area surveys use the same incommunity sampling and data-collection methods as CSAS surveys. Cases are found using an active and adaptive case-finding method (see Box 1). When a case is found, the carer is asked whether the child is already in the programme. A short questionnaire is administered if the malnourished child is not already in the programme.

Severe acute undernutrition is a relatively rare phenomenon. This means that the sample size (i.e. the number of cases found) in smallarea surveys will usually be too small to estimate coverage with reasonable precision (i.e. as a percentage with a narrow 95% confidence interval). It is possible, however, to classify coverage (i.e. as being above or below a threshold value) with small sample sizes using a technique known as Lot Quality Assurance Sampling (LQAS).

Analysis of data using the LQAS technique involves examining the number of cases found (n) and the number of covered cases found. If the number of covered cases found exceeds a threshold value (d) then coverage is classified as being satisfactory. If the number of covered cases found does not exceed this threshold value (d) then coverage is classified as being unsatisfactory. The value of d depends on the number of cases found (n) and the standard against which coverage is being evaluated.

 

 

 

The SPHERE minimum standard for coverage of therapeutic feeding programmes in rural settings is 50%. The following rule-of-thumb formula may be used to calculate a value of d appropriate for classifying coverage as being above or below a standard of 50% for any sample size (n):

 

 

The [ and ] symbols mean that you should round down the number between the [ and ] symbols to the nearest whole number. For example:

 

With a sample size (n) of 11, for example, an appropriate value for d would be:

 

For standards other than 50%, the following ruleof- thumb formula may be used to calculate a suitable value for d for any coverage proportion (p) and any sample size (n):

 

For example, with a sample size (n) of 11 and a coverage proportion (p) of 70% (i.e. the SPHERE minimum standard for coverage of therapeutic feeding programmes in urban settings) an appropriate value for d would be:

An alternative to using the simple rule-of-thumb formulae presented here is to use LQAS sampling plan calculation software. Consideration of the classification errors associated with candidate LQAS sampling plans should be informed by the two-stage screening test model used by SQUEAC².

Figure 9 shows the data collected in the small-area survey of the area shown in Figure 8. The survey found 12 cases and 3 of these cases were in the programme. The appropriate value of d for a sample size (n) of 12 and a coverage standard of 50% is:

Since 3 is not greater than 6, the coverage in the surveyed area is classified as being below 50%.

The LQAS technique may also be used to classify defaulting and DNA rates. For example, using the data presented in Figure 10 and a standard for DNA rates of 15% (maximum):

In this example there are 7 DNA cases from 15 referrals. Since 7 is greater than 2, the DNA rate for referrals from this particular community based volunteer (CBV) is classified as being unsatisfactory (i.e. above 15%).

SQUEAC use-studies

The first SQUEAC use-study examined the ability of the LQAS method to classify coverage correctly and was undertaken by VALID International and World Vision in Ethiopia in March 2007.

Six small-area surveys were undertaken:

• Six OTP sites were selected at random.
• Three villages (kebeles) were selected at random from each OTP site's catchment area.
• Five localities (gotts) where then selected at random from each of the selected villages.
• Each locality was sampled using active and adaptive case-finding.

The true coverage for each OTP site was estimated using data from a CSAS survey undertaken at the same time as the small-area surveys. The results from these small-area surveys are shown in Table 1. The LQAS method correctly classified coverage in each of the six OTP catchment areas.

Table 1: Results from six small-area surveys from the first SQUEAC use-study
OTP Site	True Coverage	Cases Found (n)	Covered Cases (c)		Is c > d?	Classified Coverage
Bezena Benara	> 50%	15	9	7	Yes	> 50%
Jore	> 50%	5	3	2	Yes	> 50%
Yebu	< 50%	16	6	8	No	< 50%
Funto	< 50%	21	7	10	No	< 50%
Demboya	> 50%	5	3	2	Yes	> 50%
Adilo	< 50%	9	2	4	No	< 50%

Data courtesy of World Vision

 

The second SQUEAC use-study was undertaken in a Concern Worldwide CTC programme in the Democratic Republic of Congo in November 2007. This concentrated on the use and availability of routine programme data and anecdotal data to identify areas with either low attendance rates or high defaulting and DNA rates. It also looked at the ability of local programme staff to collect and analyse data from these sources and to plan, undertake, and analyse data from small-area surveys. Local staff proved capable of using routine programme data to identify probable areas of low coverage and reasons for non-attendance and defaulting when these data were available and presented using simple graphs, tables, and maps. They had no difficulties mapping routine programme data. Local staff also had no difficulty undertaking small-area surveys and analysing survey data using the LQAS technique. However, the collection and analysis of routine programme data needed to be improved and standardised. Furthermore, collection and analysis of anecdotal data by local programme staff proved problematic, particularly with informal group discussions. This situation may improve with careful selection and training of local staff. Further usestudies will concentrate on resolving this issue.

The SQUEAC method is suited to being used within a clinical audit framework³. For more information, visit: http://www.brixtonhealth.com/squeaclq.html

Estimating and classifying 'headline' or overall programme coverage

Using LQAS techniques to derive a 'headline' coverage classification requires:

A first stage sampling method: This is the sampling method that is used to select the villages to be sampled. CSAS assessments use the centric systematic area sampling or quadrat method to select villages to be sampled. If only a 'headline' coverage classification is required, then a similar method could be used to select villages to be sampled. The number of quadrats drawn on the map can be smaller than would be used for a CSAS assessment (this is the same as using larger quadrats). The villages to be sampled would then be selected by their proximity to the centre of each quadrat as is done in a standard CSAS survey (Figure 11). Such an approach would be appropriate for classifying coverage over a wide area, such as a health district. In developmental settings it may be desirable to classify coverage over a wide area and also for individual clinic catchment areas. In such situations, a stratified (i.e. by clinic catchment area) sample could be taken with villages selected at random from a complete list of villages within each catchment area (Figure 12).

A within-community sampling method: This will usually be an active and adaptive casefinding method (see Box 1) or a house-to-house census sampling method.

A LQAS sampling plan: This provides a target sample size (n) which, together with prevalence and population estimates, is used to decide the number of villages to sample (see below). The target sample size (n) will usually be larger than is required for the small-area surveys used in the SQUEAC method. Suitable sampling plans can be selected using an LQAS sampling plan calculator. Figure 13 shows an LQAS sampling plan calculator being used to select a sampling plan. This LQAS sampling plan calculator is available, free of charge, from: http://www.validinternational.org/pages/sub.cfm?id=1780

A sample size calculation: This is the target sample size (n) from the LQAS sampling plan which, together with estimates of the prevalence of severe acute undernutrition in the survey area and population data, is used to calculate the number of villages that will need to be sampled:

Once these decisions and calculations have been made, sampling locations can be identified and the survey undertaken. A standard questionnaire should be applied to carers of noncovered cases found by the survey.

It is unlikely that a survey will return the target sample size (n) exactly. So, a value for the LQAS classification threshold value (d) for the achieved sample size should be calculated using the rule-of-thumb formulae presented earlier in this article. For example:

LQAS sampling plan calculation software could also be used (Figure 13).

Coverage is classified using the same technique as is used for SQUEAC small-area surveys. For example:

Analysis of data collected in individual clinic catchment areas is analysed in the same way although errors may be considerable if sample sizes are very small.

Conclusions

The early work presented in this article suggests that the SQUEAC approach is likely to prove a suitable method for frequent and ongoing evaluation of programme coverage and identification of barriers to service access and uptake. The SQUEAC approach is also capable of providing a similar richness of information as is provided by the CSAS method.

The LQAS method may be used to provide classifications of 'headline' or overall coverage over wide areas. The low sample size requirement means that surveys using the LQAS method are likely to be less resource intensive than standard CSAS surveys. Such surveys would, however, be limited in their ability to provide a detailed map of the spatial distribution of programme coverage.

For further information, contact: Mark Myatt, at http://www.brixtonhealth.com/squeaclq.html

---

¹A more detailed report on the SQUEAC method can be found at http://www.brixtonhealth.com/squeaclq.html

²For more information on this, visit: http://www.brixtonhealth.com/squeaclq.html

³Clinical audit is a quality improvement and monitoring method that seeks to improve service delivery through systematic review against specific criteria and standards and the implementation of change.