A review of research methods used to study specialised nutritious foods

View this article as a pdf

By Kristine Caiafa, Maria Wrabel, Devika Suri, Ye Shen, Shelley Walton, Beatrice Lorge Rogers and Patrick Webb.

All authors are affiliated with the Food Aid Quality Review (FAQR) project at the Friedman School of Nutrition Science and Policy at Tufts University.

The authors are grateful to the U.S. Agency for International Development (USAID) and the USAID Office of Food for Peace (FFP) of the Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA) (AID-OAA-C-16-00020) for funding this activity, and to Breanna Langlois for her invaluable contributions to the development of this manuscript.

Location: Global

What we know: Research on specialised nutritious foods (SNFs) has increased in the past two decades, but this has not resulted in commensurate advances in policies and practice.

What this article adds: The goal of this exercise was to identify key factors in SNF study methodologies that could be strengthened to develop a more rigorous evidence base. A search of the literature and clinical-trial registries was conducted to identify studies using SNFs to influence anthropometric outcomes, and information about the research methods used was collected. Among the 114 studies identified (89 published and 25 ongoing), impediments to a robust evidence base included research bias, heterogeneous study design and insufficiently reported study details. A list of specific actions was developed to be taken by global agencies, research funders, researchers and practitioners to build a higher-quality evidence base for translating research on SNFs into policy and practice. Supplemental tables are included in an online version of this article.

Introduction

Specialised nutritious foods (SNFs), which include lipid-based nutrient supplements (LNSs), ready-to-use therapeutic foods (RUTFs), ready-to-use supplementary foods (RUSFs), fortified blended foods (FBFs), micronutrient powders (MNPs) and locally produced analogs of these products, are food products specially formulated to treat, prevent or mitigate undernutrition. Scientific research on SNFs has expanded rapidly in the past two decades, driven by an intent to improve nutrition outcomes. While much has been learned, global practice standards for using SNFs as a class remain elusive. The challenges inherent in studying these products and a lack of aggregate emphasis on study quality has generated an evidence base considered in recent reviews to be of low or moderate quality (Webb, 2015; Lazzerini et al, 2013; Schoonees, 2013).

An excellent model for how to move forward can be found in the methods used to develop standards for the management of acute malnutrition (WHO, 2013; WHO, 2012). These global policies were made possible by corralling a robust evidence base, largely through the use of systematic reviews. Similar evidence synthesis for SNFs would require high-quality studies; i.e., using designs that are appropriate to the research question and which mitigate risk of bias and threats to validity, and that are collectively similar enough in study characteristics so that findings are comparable. Research generalisability, or applicability to larger populations from which a study sample is drawn, is also critical to this type of evidence synthesis.

The aims of this review are threefold: first, to identify common methods used in a sample of SNF research; second, to highlight the methods that influenced quality, comparability and generalisability; and third, to propose actions for a stronger evidence base.

Methods

This review included studies based on two factors: 1) the study used an SNF in at least one intervention arm, and 2) the study specified at least one anthropometric outcome. Studies were not excluded based on characteristics of participants, comparisons or study design. Studies published between January 1 2011 and April 1 2017 were identified through searches conducted in English of PubMed and Web of Science in April 2017. These were compiled and screened for inclusion by one analyst; first by title, then abstract, then content. A second analyst independently reviewed any studies in question, first by abstract and then by content. Eighty-nine publications were identified for this review.

Ongoing trials were identified in December 2017 using REFINE (Research Engagement on Food Interventions for Nutritional Effectiveness; www.REFINEnutrition.org), a public platform that maps SNF research. Twenty-five ongoing studies were identified by one analyst, who recorded information from trial registrations and publicly available study protocols.

In preparing this manuscript, the authors followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist (Higgins and Altman, 2008) to the extent applicable.

Findings and discussion

Description of studies

Overall, we reviewed 89 publications and 25 ongoing studies (114 studies total). Based on terminology used in each study, predominant outcomes of interest among studies treating acute malnutrition (n=60) were recovery (as defined by the study) and weight gain. Other studies included stunting as a categorical variable (61%; n=33); height/length-for-age (included in 72%; n=39); wasting as a categorical variable, defined by either mid-upper arm circumference (MUAC) or weight-for-height z-score (WHZ) (44%; n=24); weight-for-age (59%; n=32); underweight as a categorical variable (31%, n=17); linear growth (39%, n=15); birth outcomes (17%, n=9); and biomarkers of undernutrition (20%, n=11). Most studies (87%, n=99) targeted children under five years of age, with focus on multiple sub-groups in that age range.

Research design

Eighty per cent (n=92) of studies were intervention studies, in which investigators assigned (randomly or not) participants to receive a specific intervention. Of these, most (92%, n=88) had multiple study arms. Twenty per cent (n=22) of studies were observational, in which investigators observed the outcomes of participants who had received an intervention not assigned by the investigators.

Studies typically explored one of three questions: 

1. Is a given SNF effective at improving nutrition-related anthropometric measures? 
2. Which SNF is comparatively more effective than other SNF(s) at improving nutrition-related anthropometric measures?
3. Do complementary activities in addition to SNF improve nutrition-related anthropometric measures?

To effectively answer these questions, researchers must be transparent about research design limitations and interpret findings only in a way that the study is designed to answer. For example, single-armed research (which accounted for 18% of all studies) can offer valuable hypothesis-generating information, but is relatively limited in answering the above questions.

Avoidable bias

We assessed study methods against the forms of bias included in the Cochrane Collaboration’s study quality evaluation protocol (Higgins and Altman, 2008; Ryan and Hill, 2016; Schünemann et al, 2013):

Selection bias – Among intervention studies (n=92), 42% (n=39) passively recruited participants (i.e., drew from community screenings or enrolled self-selecting individuals), 39% (n=36) actively recruited participants from a target community, and 8% (n=7) randomly sampled participants from communities. Although passive recruitment is decidedly convenient, it can introduce selection bias insofar as some segments of a target population may be unintentionally but systematically excluded due to exogenous characteristics, leading to baseline differences in the groups that are compared. Consequently, enrolled participants may also not be representative of the underlying population (which also impedes generalisability). Active recruitment, in which study investigators directly recruit participants, can reduce this bias.

Performance and detection bias – Of multi-armed studies, less than half (47%, n=41) used a form of blinding or blinded outcome assessors or data analysts (43%, n=38). Not blinding the treatment, enumerators or analysts can lead to awareness of the treatment group that might cause altered outcomes (due, for example, to more careful SNF preparation, adherence to feeding regimens or weight measurements). To this point, differences in SNF packaging, appearance, taste, texture and preparation should be as discreet as possible, and participants, researchers and staff should be blinded to the treatment and non-treatment groups to the extent possible. At a minimum, data analysts should be blinded to intervention groupings.

Most multi-armed studies did not note whether or how intervention implementation differed between arms. To mitigate performance bias, or differences between study arms relating to how interventions are carried out, study arms should be implemented using the exact same methods, except for the elements designed to differ. For instance, SNFs should be distributed in the same time increments (e.g., monthly) and enumerators should be identically trained and supervised.

Attrition bias – To avoid bias, study attrition and differential rates of attrition between arms should be expected and accounted for in study planning and analysis. Most publications reported some degree of attrition (52 out of 68 publications reported baseline and endline sample sizes); however, only three explicitly reported differential attrition. Determining whether attrition rates differ based by age, sex, SNF or distance from treatment centre can inform whether there was any bias related to those who remained enrolled compared to those who did not.

Reporting and publication bias – Twenty-one per cent of all studies did not state pre-specified research outcomes, introducing potential bias regarding whether outcomes have been selectively reported to alter the conclusions drawn. Reports should: i) state all pre-specified primary and secondary outcomes; ii) examine those outcomes; and iii) present all pre-specified outcome data, regardless of positive, negative or null results. Reporting and publication bias can be further mitigated by reporting every research activity to a trial registry. Keeping information updated upholds high standards of transparency and accountability to participants.

Comparability

Study comparability was impeded by inadequately detailed study design information, as well as inconsistent parameters and definitions. A subset of studies on severe acute malnutrition (SAM) treatment illustrates the lack of comparability in this sample. Among these (n=28), there were six similar (but not identical) target age groups: “under 6 months,” “6-23 months,” “6-24 months,” “6-59 months,” “6-60 months,” and “6 months to 5 years.” A child treated from “6 months to 5 years” could mean that child receives the intervention from six months until their fifth birthday or until any timepoint within their fifth year; without more explicit cut-off information or commonly-agreed definitions, it is impossible to know. Outcome variables, while also generally aligned, were not equivalent. Although 57% (n=16) identified the primary outcome as “per cent of children recovered from SAM,” different definitions of “recovery” were used: seven studies based their determination of recovery on MUAC, compared with nine that used WHZ.

To enhance comparability, it would be useful for all studies to report on all relevant outcomes. Direct comparability is impossible; differing resource constraints, geography, infrastructure, political environments and social factors require (and even favour) flexibility in SNF programming choices. Nevertheless, research leaders are well positioned to establish a set of “reference cases” like those developed for economic evaluation of health technologies in low- and middle-income countries (Wilkinson et al, 2016). For instance, for studies considering the effect of an SNF on wasting recovery, there would be a standardised set of research designs with acceptable definition and indicators for “recovery”. The intention should be to reach a minimum level of research alignment for study comparability, while still enabling creative scientific inquiry.

Generalisability

Generalisability requires representative sampling of the population of interest. Sample size should always account for variability in the target population and be large enough to draw conclusions about that population with a specified level of confidence. Random selection and assignment of participants allows researchers to verify that observed effects are not a result of differential characteristics of the groups being studied. In this sample, random assignment was common: of intervention studies with multiple study arms (n=88), 92% (n=81) randomly assigned interventions to either clusters (42%, n=37) or individuals (51%, n=45).

Ideally, a study investigating the effects of an SNF on the incidence of moderate acute malnutrition (MAM) recovery in a specific country would randomly select children with MAM from across the country. In practice, however, such defined sampling frames do not exist and implementing an experimental intervention at the national level would not be feasible – and it would be impossible to monitor these programmes closely. Cluster randomisation, in which interventions are assigned to clusters rather than individuals, is an approach that allows clinic and community-centred feeding programmes to continue to operate with minimal disturbance. Participants can be sampled in a way that is representative of the target study population in terms of demographic, geographic and other relevant community characteristics; blinding can be addressed; and the risk of treatment crossover is greatly reduced (Friis et al, 2015).

Reporting

Reporting methods predetermine the quality and utility of an evidence base. In this sample, study methods were often not reported with the capacity for replicability, comparability and external assessment of quality. Twenty-five per cent (n=28) of publications did not provide SNF nutrient composition; 72% (n=82) of all studies did not specify SNF dosage; and 69% (n=79) did not state dose distribution frequency (e.g., weekly, monthly, etc.). Of published intervention studies, 94% (n=85) did not report study-arm sample sizes at both baseline and endline and 26% (n=66) of publications did not report statistical power calculations. As information about these important design elements is critical to assess bias and replicate studies, researchers should always use accepted checklists when preparing reports¹.

Reporting also has implications for systematic reviews, which are widely used tools for decision-making. To be included in a systematic review, a study must first include the relevant search terms, then meet narrowly defined inclusion criteria assessed through reported study methodology. We observed inconsistent terminology for common potential search terms, like ready-to-use foods (RUF) and lipid-based nutrient supplements (LNS). To avoid study exclusion due to nominal differences, norm-setting bodies (such as the United Nations organisations, academic institutions, and prominent, consortium-based communities of practice such as the State of Acute Malnutrition and the Food Security and Nutrition Network) should standardise the language around SNFs and nutrition, beginning with product terminology.

Conclusions

Despite the many challenges inherent to SNF research, there are opportunities to support a stronger research practice and evidence base. A global body responsible for coordinating SNF research would be ideally placed to develop and institutionalise specific guidance on SNF research methods, supporting both researchers and funders in their efforts to produce rigorous evidence. In the absence of such a body, we suggest a list of actions as a starting point for discussion and revision (Box 1). We hope that this is a first step for decision-makers in this community to take this effort forward and own a more active role in coordinating SNF research to inform global policy.

For more information, please contact Maria Wrabel at maria.wrabel@gmail.com

Box 1: Actions for SNF research

Actions for normative agencies

1. Establish reference cases for commonly studied topics in food aid research to improve how studies are conducted and reported.
2. Establish standards for terminology and corresponding abbreviations.

Actions for research funders

1. Require research awardees to identify all the components listed under Study reporting, below.
2. Require research awardees to follow relevant reporting checklists.
3. Require research awardees to register trials in a trial registry and keep the information up-to-date.

Actions for researchers and research staff

Study design and analysis

1. When possible, actively recruit study subjects.
2. Always blind data analysts to the intervention group. To the extent possible, blind participants, researchers, intervention implementers and outcome assessors to the different interventions and intervention groups.
3. Anticipate attrition and account for it in the sample size calculation. Record attrition overall, across and within study arms, and investigate differential attrition rates by participant and intervention characteristics.
4. Identify all outcomes of interest at the outset of the study and investigate all of them.
5. Use the same intervention methods in all study arms.
6. Use random selection and assignment of study participants. When appropriate, consider randomised cluster-based sampling.

Study reporting

1. Register every trial with a trial registry. Keep the information up-to-date.
2. Follow accepted research-reporting checklists.
3. Describe the study with the intention for replicability, comparability and external assessment of quality. Include:

• Study details: year(s) trial took place; country in which trial took place; context (emergency/protracted emergency/development; rural/urban/semi-urban); target population; nutrition problems studied; all planned outcomes of interest; outcome indicators.
• Study design: intervention (randomised controlled trial, cluster-randomised control trial, non-randomised study, etc.)/observational (cross-sectional, retrospective, prospective)/ descriptive; inclusion of control group or comparator; all blinding used.
• Participant selection and sampling strategy: inclusion criteria; exclusion criteria; active or passive recruitment; randomisation strategy (clustered, individual, non-randomised); total sample size; sample size per intervention arm; sample size and power calculations.
• Intervention assignment: parallel; factorial; single group; crossover.
• Intervention design: SNF studied; nutrient composition of SNF; dose provided; frequency of product distribution; total number of distributions over the course of the intervention; duration of intervention; implementation methods used to carry out study arms, noting differences; descriptions of how foods were prepared or how instructions were given to prepare foods.
• Analysis: statistical methods used; assessment of baseline comparability across study arms; detectable effect sizes and an explicit comparison to actual sample size; desired and actual sample size; attrition rates overall, across and within study arms; possible explanations for attrition.
• Outcomes: results for all pre-specified outcomes.

Endnotes

¹CONSORT 2010 checklist www.consort-statement.org/media/default/downloads/consort 2010 checklist.pdf; STROBE checklist www.strobe-statement.org/fileadmin/Strobe/uploads/checklists/STROBE_checklist_v4_combined.pdf; MOOSE checklist www.elsevier.com/__data/promis_misc/ISSM_MOOSE_Checklist.pdf and SQUIRE www.squire-statement.org/index.cfm?fuseaction=page.viewpage&pageid=471

References

Friis H, Michaelsen KF, Wells JC. Choice of Design and Outcomes in Trials among Children with Moderate Acute Malnutrition. Food Nutr Bull. 2015 Mar;36(1_suppl1):S35–40.

Higgins JP, Altman DG. Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions: Cochrane book series. 2008:187-241.

Lazzerini M, Rubert L, Pani P. Specially formulated foods for treating children with moderate acute malnutrition in low- and middle-income countries. Cochrane Database Syst Rev [Internet]. John Wiley & Sons, Ltd; 2013; Available from: http://doi.wiley.com/10.1002/14651858.CD009584.pub2

Ryan R, Hill S. How to GRADE the quality of the evidence [Internet]. 2016. Available from: http://cccrg.cochrane.org/sites/cccrg.cochrane.org/files/public/uploads/how_to_grade_revising_1_ december_2016.pdf

Schoonees A, Lombard M, Musekiwa A, Nel E, Volmink J. Ready-to-use therapeutic food for home-based treatment of severe acute malnutrition in children from six months to five years of age. In: Volmink J, editor. Cochrane Database of Systematic Reviews [Internet]. Chichester, UK: John Wiley & Sons, Ltd; 2013. Available from: http://doi.wiley.com/10.1002/14651858.CD009000.pub2

Schünemann H, Brozek J, Guyatt G, Oxman A. Handbook for Grading the Quality of Evidence and the Strength of Recommendations Using the GRADE Approach [Internet]. The GRADE Working Group; 2013. Available from: http://gdt.guidelinedevelopment.org/app/handbook/handbook.html

Webb P. How Strong is Our Evidence for Effective Management of Wasting? A Review of Systematic and other Reviews. Food Nutrition Bulletin [Internet]. SAGE PublicationsSage CA: Los Angeles, CA; 2015; 36(1_suppl1):S65–71. Available from: http://journals.sagepub.com/doi/10.1177/15648265150361S111

WHO. Guideline: Updates on the management of severe acute malnutrition in infants and children. Geneva: World Health Organization; 2013.

WHO. Technical note: supplementary foods for the management of moderate acute malnutrition in infants and children 6–59 months of age. Geneva, World Health Organization, 2012.

Wilkinson, Thomas, Mark J. Sculpher, Karl Claxton, Paul Revill, Andrew Briggs, John A. Cairns, Yot Teerawattananon et al. “The international decision support initiative reference case for economic evaluation: an aid to thought.” Value in Health 19, no. 8 (2016): 921-928.