Sampling in insecure environments: Field experiences from coverage assessments in Afghanistan
By Ben Allen, Mark Myatt, Nikki Williamson, Danka Pantchova and Hassan Ali Ahmed
Ben Allen has been Global Coverage Advisor for Action Against Hunger UK for the past two years. He has now left to pursue further study in public health. From 2010 Ben worked with ACF-UK in evaluation, learning and, more recently, on methodologies to assess the coverage of CMAM programmes. He has directly supported two SQUEAC assessments in Afghanistan and provided remote support to many others.
Mark Myatt is a consultant epidemiologist and senior fellow at University College London. His areas of expertise include infectious diseases, nutrition and survey design.
Nikki Williamson is Senior Project Officer at Action Against Hunger UK, specialising in the coverage assessment of CMAM programmes. Previously she was SLEAC Programme Manager in Afghanistan, managing the implementation of the five SLEACs addressed in this article. Nikki has also conducted a regional SLEAC and district SQUEAC in Uganda.
Danka Pantchova is Nutrition Surveillance and Prevention Advisor with Action Contre la Faim France. Previously she technically supported all ACF nutrition programmes in Afghanistan, including coverage assessments.
Hassan Ali Ahmed is the Nutrition Surveillance Head of Department with Action Contre la Faim Afghanistan. He has worked on nutrition surveillance projects in Kenya, Somalia and Afghanistan for the last seven years and has wide experience of conducting nutrition assessments, including SQUEACs, SLEACs, SMART and Rapid SMART surveys.
Location: Afghanistan.
What we know: Afghanistan is a challenging place to implement community-based management of acute malnutrition (CMAM) and coverage assessments, due to persistent insecurity.
What this article adds: Coverage assessments are important but not life-saving interventions; exposure to risk should be managed and minimised for staff. Insecurity typically affects spatial representation of villages and achieving minimum samples. Including only safe and accessible villages is likely to inflate coverage estimates; bias introduced due to adaptations should be clearly reported. Qualitative information from identified cases and additional qualitative studies can also valuably inform programme reform and, to some extent, compensate for the limitations to the survey sample. A number of steps can help achieve the basic principles of data collection while ensuring the safety of staff and informants. These include balancing risks, triangulation and monitoring of security information, use of local staff, additional context-specific staff training, strong communication protocols and documentation of challenges for wider lesson learning.
When evaluating access and coverage of CMAM services in insecure environments, restrictions on data collection limit assessments. This article describes the challenges faced in sampling during coverage assessments of severe acute malnutrition (SAM) treatment services across Afghanistan and provides some methodological guidance for obtaining the most reliable information while maintaining staff safety. The article draws on experiences from five different SLEAC 1 assessments (in Laghman, Badakshan, Jawzjan, Bamyan and Badghis) and three SQUEAC 2 assessments (in Samangan, Paktya and Kunar) supported by Action Against Hunger (ACF) and the Coverage Monitoring Network 3 (CMN) in Afghanistan 4. Details on these coverage methods are available at: www.coverage-monitoring.org/training-centre/
The assessments faced two broad categories of conflict related to insecurity. First, hostile environments largely caused by conflict between government forces and armed opposition groups (AOGs) or inter-ethnic and tribal conflict. Specifically, this entails unpredictable fighting with a risk of crossfire incidents, checkpoints and kidnap. Second, direct hostility towards staff and users by armed groups known to be hostile towards government, United Nations (UN) agencies, and staff of national and international non-governmental organisations (NGOs) was also experienced. Health workers may be perceived to be part of government and data collection activities are viewed with further suspicion (considered as intelligence gathering). The possibility that family planning activities are being implemented attracts even wider suspicion (i.e. not just from armed groups) of local and/or international NGO activity.
In Afghanistan, war and insecurity are a barrier to accessing healthcare services (MSF, 2014) in terms of both demand (users seeking care) and supply (partners providing care). For example, one in two people are said to be concerned about making dangerous journeys to receive healthcare (MSF, 2014), and implementing partners are under great strain to provide reliable supplies of commodities (ACF, 2016; see this report for more details on the nature of insecurity and how it impacts CMAM service delivery). Awareness of insecurity patterns can help with planning and implementing assessments (see Box 1 for examples).
A key guiding principle is that, whilst coverage assessment is key to improving programme performance, assessment should not expose survey teams or informants to unacceptable risk: the level of risk, and the tolerance of risk to survey teams, should be low.
Handling insecurity in SLEAC and SQUEAC assessments
In order to achieve a reliable classification or estimation of coverage in a given area, a minimum sample size of SAM children is required, from a spatially representative selection of villages. To achieve this, both SLEAC and SQUEAC stage 3 (wide-area surveys) use a two-stage process, sampling and case finding. The impact of insecurity in Afghanistan on both processes is shown below.
Stage 1: Sampling villages
Two sampling methods are commonly used to select villages: the ‘list method’ and CSAS (grid/quadrat) method. For both, engagement with security advisors and informants familiar with the area (e.g. programme and survey staff) was essential to assess the level of insecurity in each village. Some subjective interpretation (including understanding of unacceptable risk and assessment of the reliability of information required) was required, but triangulation by source (e.g. use survey staff, programme staff, health centre staff and locals) strengthened this appraisal. During each assessment, at least three informants were openly asked to identify villages that they knew to be unsafe, to give the reasons why, and to identify other potentially unsafe/inaccessible villages. Unsafe villages were monitored.
Villages identified as unsafe and inaccessible from all sources were considered dangerous; where lists did not concur, further data/informant discussion clarified the situation. The end result was a full list of villages with security status and reasons for insecure classification. Insecure villages were not visited and the security situation for selected villages was monitored for change. If there was any doubt, case finding in that village was postponed until the security situation had improved (if within the survey period) or was abandoned. Table 1 shows the number of villages that were removed (during the initial security review) and then abandoned (due to a change in the situation) for each SLEAC assessment undertaken in Afghanistan. A rigorous, open and participatory process in the security review and on-going monitoring was important to ensure staff trust in the final security decisions, whilst benefiting from the team’s local knowledge and experience.
Stage 2: Case finding at village level
Once villages had been selected, it was necessary first to identify an appropriate survey team to go to each village, and second to identify safe routes into and out of villages. To increase acceptance, local survey staff were employed who knew local customs (e.g. dress codes, the need for chaperones for female staff), were aware of whether communities accepted non-local women, and were familiar with local perceptions of data collection activities. Survey team members for each village were then carefully considered to ensure familiarity. If at least one member of a survey team was already known to local authorities (e.g. governors, police commanders, paramilitary commanders) and local leaders, then activities at village level were facilitated. Teams and individuals unfamiliar with an area or different ethnicity could be at higher risk of being threatened, told to leave or kidnapped. In order to identify safe routes into and out of villages, a risk assessment was undertaken, again triangulating various sources of information. Security on roads is often subject to rapid change (e.g. due to the use of roadside bombs or checkpoints) and required close monitoring.
On the day of visiting the village, a security assessment was conducted using available sources (programme staff, drivers and friends) and, when possible, village leaders were called in advance to confirm safe access. If no village-level contact was available but all available information indicated safety, then teams proceeded and contacted village leaders upon arrival. When information at base was not available, survey teams assessed the security situation while travelling to the village and upon arrival. In Afghanistan, village leaders would advise survey teams to leave if hostile forces were present in the village or nearby. Teams should and would abort field activities if there is anything above a low level of risk.
Once the team and routes had been identified, the following considerations were made to ensure ongoing monitoring of the security situation:
- Work with survey teams and teams from other programmes to monitor the on-going security situation;
- Liaise with local security services and paramilitary organisations;
- Monitor local radio traffic covering UN, NGO, police and paramilitary frequencies and in some settings, local broadcast radio stations; and
- Maintain a radio or telephone network to monitor the locations and status of teams in the field.
Depending on the existing staff capacity, additional resources (for example, a specialist in radio and security) may be needed to ensure this level of communication is maintained.
In each village, active and adaptive case finding (which involves local knowledge and local informants to find suspected cases and assumes a level of social cohesion) and door-to-door case finding (going to every house) were used to identify SAM children. Once a suspected case was found, the child was screened using mid-upper arm circumference (MUAC) and for bilateral pitting oedema. The caregiver was asked whether the child was in treatment and all cases were interviewed in order to identify boosters and barriers to access.
Due to the security risk, assessment leaders, who were often international staff, were not permitted to travel to villages to supervise data collection, and field training was not possible. To compensate, training included extra practical classroom-based exercises, including role-play. A tailored form ensured that teams recorded numbers of children screened, houses visited, cases found, and households refusing entry in each village.
During fieldwork, communication was also enhanced. Mobile telephone or radio contact was sustained during village visits for assistance and remote supervision. In the earlier stages, teams often called the survey leader on arrival and on concluding case finding to crosscheck the course of action decided. When possible, end-of-day debriefings, in person or by phone, were conducted and information gathered (including challenges faced and solutions to overcome them) was shared amongst the entire team for collective learning.
Consequences of adapted sampling
The insecure environment in Afghanistan challenges spatial representation of villages and achieving a minimum sample of children – two key aims of SLEAC surveys and stage 3 (wide-area survey) of SQUEAC surveys.
The sample of villages that are both accessible and safe is unlikely to be spatially representative of the entire intended programme catchment area; identified insecure villages will tend to be isolated. If spatial representivity is likely to be badly affected, alternative villages (or contingency clusters) could be selected as close to the original village as possible. In the Afghanistan assessments, alternative villages were not used due to time restraints, resulting in a smaller sample size than planned and therefore less accurate and precise coverage classification or estimates. The removed villages are typically places where service delivery and access are more affected, and where coverage is likely to be lower. Including only safe and accessible villages is likely to inflate coverage estimates. The sampling method and any potential bias should be clearly noted when reporting the assessment.
In order to compensate for loss in survey coverage and to access information in inaccessible areas, additional quantitative and qualitative studies can be conducted. Quantitative analysis can compare numbers of admissions or defaulting rates from insecure areas with those from secure areas to look for differences and provide some indication of coverage levels.
Interviews can be conducted in nearby health centres with residents that have travelled from inaccessible areas, and with health staff that cover those areas. Alternatively, specific survey staff members for whom the risk is deemed acceptable may be able to visit an otherwise inaccessible area. In these circumstances, the staff member should be carefully prepared and consulted, together with security advisors. Box 2 outlines details of a qualitative study on insecurity conducted in Jawzjan.
Box 2: Summary of a small study on insecurity conducted during a SLEAC assessment in Jawzjan
Introduction
During the SLEAC assessment in Jawzjan, large areas of the province and entire districts were deemed to be inaccessible for the survey team. In order to understand more about coverage and access of SAM treatment in these areas, a small study was designed.
Methodology
Qualitative data was collected through structured interviews with programme management staff, health facility staff and patients. Three health facilities where surrounding villages were deemed insecure were visited. In each health facility, interviews were conducted with at least two staff members (engaged with CMAM) and two to four patients (visiting for CMAM treatment). The following considerations were made to maintain the safety of the surveyor:
- Locally known survey team members were sent and local community members and health staff should be informed in advance.
- Advise that the study is about the CMAM programme – do not say it is a study on security since this may raise unnecessary concerns.
- When meeting people, do not ask for interviews (which are often seen as interrogatory), just explain that you are there to talk about the CMAM programme.
- Ensure the conversation is conducted in a secure and relaxed environment where the informant can speak openly and in a free manner.
Themes investigated
Interview guides were developed to support surveyors in collecting useful information on challenges to access and service delivery. Different guides were developed for caregivers and for staff to collect the following information:
Caregivers at the health facility:
- Information about their journey: means of transport, duration, cost, whether there were obstacles, whether insecurity affects the journey and how the decision is made to visit the facility.
- Information about available alternatives when the decision is made not to visit the health facility, such as use of CHW (community health worker) or traditional medicine.
Staff at the health facility:
- Information about the running of the programme: details of any past closures due to insecurity, and whether insecurity affects the logistical running of the programme, for example in ready-to-use therapeutic food (RUTF) supply.
- Information about activities away from the health facility such as alternative strategies to reach SAM cases, e.g. use of CHWs, outreach activities and training and supervision visits.
Findings
These investigations provided information about types of insecurity, as well as the impact of insecurity on community access and on the provision of services.
Impact on community access:
- People are not able to take the road because of checkpoints and threats on the way.
- Women were not able to travel alone, and therefore without a male chaperone (who was not always available) were not able to access services.
- More than half of the informants (five out of nine) spoke of clashes between AOG and government security forces (Afghan National Security Forces (ANSF)) as a factor preventing them from making the journey to the health facility.
- Instead of making a journey, they either use alternative treatments (such as buying medicine from shops or using fortified milk) or simply must wait until security is better before travelling to the health facility.
Impact on provision of services:
- Insecurity inhibits activities required for operating the SAM sites, specifically those involving movement within the district, such as monitoring, training, supervision and supply of RUTF.
- Delivery of RUTF was managed through relationships with local shura, who had strong links with armed groups, which enabled the continuation of treatment services.
- Many areas in the province are too insecure for any monitoring or supervision to be conducted safely. For example, in Qush Tepa and Darzab, which are long-term insecure, the last support visit to the health facilities was possible in 2010.
- In Darzab, the health facility (a sub-centre) has now closed because the staff did not feel safe to work there.
Guidance for practitioners
It is possible to undertake SQUEAC and SLEAC assessments in insecure environments, but this may affect the quality and reliability of the information. The bias introduced by the removal of villages from the sampling frame must be recognised and clearly reported. However, the qualitative information (on barriers and boosters) collected from the cases found during the survey remains useful and can provide evidence for programme reform.
A number of steps can ensure the most reliable and comprehensive information regarding coverage and access, achieving the basic principles of data collection while ensuring the safety of staff and informants. The following ten points should be considered by practitioners undertaking coverage surveys in insecure environments:
1. Adhere to the basic principle that CMAM programmes are child-survival programmes, therefore performance must be assessed and coverage assessments themselves are not life-saving activities throughout the assessment;
2. Ensure triangulation of security information with survey team members and other local sources related to the accessibility of villages;
3. Use a rigorous and participatory process to review the security status of villages;
4. Ensure constant assessment of the security situation, especially in sampled villages and access routes;
5. Use local survey staff who both know the area and are known in the area;
6. Where possible make contact with the village leader prior to travelling to the village;
7. Allow for extra time training and supervising survey teams and include role play activities;
8. Ensure regular communication with survey teams to monitor their safety and provide close supervision, including daily debriefings;
9. Develop context-specific qualitative and quantitative studies to investigate factors affecting access in the inaccessible areas; and
10. Document all limitations, challenges and adaptations to the methodology in the final report.
For more information, contact: Nikki Williamson, email: n.williamson@actionagainsthunger.org.uk
References
MSF, 2014 Afghanistan: Between rhetoric and reality – the ongoing struggle to access healthcare. Médecins Sans Frontières. www.msf.org.uk/article/afghanistan-between-rhetoric-and-reality-–-ongoing-struggle-access-healthcare.
ACF, 2016 A review of SAM management in Afghanistan: Lessons from 2013-2016. ACF International. www.cmamforum.org/Pool/Resources/Afghanistan-Review-of-SAM-management-2016.pdf
1Simplified LQAS (lot quality assurance sampling) evaluation of access and coverage.
2Semi-quantitative evaluation of access and coverage.
4See www.coverage-monitoring.org/country/afghanistan/ to access the reports.