Is there a systematic bias in estimates of programme coverage returned by SQUEAC coverage assessments?
This is a poscript for research article 'Is there a systematic bias in estimates of programme coverage returned by SQUEAC coverage assessments?' by Mark Myatt and Ernest Gueverra, which you can read here.
By Sheila Isanaka, Rebecca F. Grais, and Ben G.S. Allen
We thank Mark Myatt and Ernest Gueverra for adding this important work to the ongoing discussion surrounding appropriate coverage methodologies in the management of acute malnutrition. A 2015 review of coverage methodologies highlighted uncertainty in the use of currently recommended methods for coverage assessment and the need for more peer-reviewed evidence to inform global guidance (Epicentre, 2015). We are delighted that our work (Isanaka et al 2018) may have motivated additional consideration of these important issues and hope that stakeholders and policy makers continue to insist on high-quality, evidence-based experience to inform nutrition programming.
The report by Myatt and Gueverra (2019) shows that coverage estimation using the SQUEAC methodology can yield biased estimations in either direction. For the first time, Myatt and Gueverra quantify the magnitude of this bias, showing only a moderate correlation between the prior and likelihood estimates (Pearson correlation coefficient 95% confidence interval: 0.67 to 0.78). This new evidence is consistent with our findings and supports our conclusion that conflicts between prior and likelihood modes are possible and can lead to biased and imprecise coverage estimates. As discussed by Myatt and Gueverra, the risk of such bias is low when trained staff conduct a survey using appropriate SQUEAC methods. Our work supports this finding, as we similarly show no conflict when an external support team estimated the prior. Our experience, however, goes further than what is possible in the secondary analysis of Myatt and Gueverra to suggest that in resource-limited settings, where sufficient capacity and resources may not be available and the correct methodology may not be faithfully executed, conflict and bias may be more common.
We wholeheartedly agree that coverage assessments should be done using appropriate methods as outlined in the SQUEAC manual (Myatt et al. 2012). Our SQUEAC assessment followed this guidance, including triangulation by source and method, use of the BBQ tool, sampling to redundancy and a small survey. We used booster and barrier weighting by caregivers of severely acutely malnourished (SAM) children as reported in other contexts (Blanárová et al. 2016) as one component in developing the prior mode. The various prior modes in our analyses were combined to simulate different potential scenarios, including situations where external support is not available. This was done for the pedagogic purpose of the study, and as discussed in the paper does not necessarily reflect typical SQUEAC procedures.
We note that the database employed in the analysis of Myatt and Gueverra is likely comprised of surveys conducted by experienced coverage consultants (those provided, for example, by the Coverage Monitoring Network or Valid International) and therefore include prior modes developed by dedicated consultants using gold standard methods that may be less likely to conflict. The analyses further include data from supplementary feeding programmes and does not standardise calculations of coverage estimate according to current guidance to use single coverage (Balegamire S, 2015), analytical choices which may influence the extent to which conflicts and bias were detected. Nonetheless, we welcome the new evidence presented by Myatt and Gueverra (2019) as an important step towards better understanding of the implications of using the SQUEAC methodology for valid coverage estimation.
Overall, we look forward to continued evidence-based and peer-reviewed discussion of appropriate coverage methodologies. Several methodologies are available to monitor programme coverage, and the appropriate study design should be selected in consideration of team capacity, resources and reporting requirements. SQUEAC can be a technically demanding method and requires the appropriate capacity to avoid the potential for bias. As both we and Myatt and Gueverra have shown, conflict and biased coverage estimation are possible and should be considered in selecting the appropriate study design and allocating appropriate resources for assessment.
References
Balegamire S, Siling K, Alvarez Moran JS, Guevarra E, Woodhead S, Norris A, Fieschi L, Paul Binns P, and Myatt M, A single coverage estimator for use in SQUEAC, SLEAC, and other CMAM coverage assessments, 2015, Field Exchange 49 Full text available from: https://www.ennonline.net/fex/49/singlecoverage
Blanárová L, Woodhead S, and Myatt M, Community weighting of barriers and boosters in Democratic Republic of Congo, 2016, Field Exchange 52 Full text available from: https://www.ennonline.net/fex/52/barriersandboostersdrc
Epicentre. Open review of coverage methodologies: comments, questions and ways forward. London: Coverage Monitoring Network; 2015.Isanaka S, Hedt-Gauthier BL, Grais RF, Allen BG, Estimating program coverage in the treatment of severe acute malnutrition: a comparative analysis of the validity and operational feasibility of two methods, Population Health Metrics, 2018,16:100,1-9 Full text available from: https://pophealthmetrics.biomedcentral.com/articles/10.1186/s12963-018-0167-3
Myatt M, Guevarra E, Fieschi L, Norris A, Guerrero S, Schofield L, Jones D, Emru E, Sadler K (2012) Semi-Quantitative Evaluation of Access and Coverage (SQUEAC) / Simplified Lot Quality Assurance Evaluation of Access and Coverage (SLEAC) Technical Reference, Food and Nutritional technical Assistance III Project (FANTA-III), FHI 360 / FANTA, Washington, DC, October 2012 https://www.fantaproject.org/monitoring-and-evaluation/squeac-sleac