CMAM Surge: understanding costs and potential contribution to CMAM’s cost-effectiveness

View this article as a pdf

By Kate Golden and Simone van Dijk

Kate Golden is Senior Nutrition Adviser at Concern Worldwide. Her career started in Ethiopia in 2003 in one of the early pilots of community-based management of acute malnutrition. Since then, she has worked in South Sudan, Sudan and Lebanon and supported Concern nutrition programming and strategy development in roughly 20 countries across Africa and Asia as a global adviser.

Simone van Dijk is a freelance consultant with over 12 years of experience in the humanitarian and development sector at the intersection of research and policy development. Simone has worked for the Red Cross Movement (Netherlands, Colombia, International Federation of the Red Cross), Save the Children (UK and International) and other smaller international non-governmental organisations.  

The authors would like to acknowledge Concern Worldwide’s Ethiopia and Niger teams who carried out data collection for the Concern community-based management of acute malnutrition (CMAM) Surge cost-effectiveness studies and Jean Christophe Fotso of EVIHDAF and Mark Myatt for leading the studies.

Introduction           

The community-based management of acute malnutrition (CMAM) Surge approach aims to optimise the efficiency of the delivery of wasting treatment services over time by helping health systems better anticipate and prepare for peak periods of service demand. The approach builds on the premise that appropriate, early action is generally more cost-effective than a traditional, large-scale response launched once an emergency is underway (Idris, 2018). As part of a broader effort to evaluate the added value of CMAM Surge, Concern Worldwide (Concern) developed a Value for Money (VfM) framework to help examine whether CMAM Surge is likely to be more cost effective than routine delivery modalities for CMAM and conducted two cost-effectiveness analyses (CEA) in contexts where CMAM Surge was being implemented.

Many different approaches to CEA and terminology exist. Concern used the widely recognised definitions of VfM and CEA outlined by the United Kingdom’s Foreign, Commonwealth and Development Office (UK FCDO)¹ as the basis for this analysis.² Recognising some limitations of the model, UK FCDO emphasises the importance of using qualitative methods alongside quantitative ones to contextualise and understand VfM results (DFID, 2013).

This article outlines key factors to consider when assessing the cost-effectiveness of the CMAM Surge approach, shares results and some challenges from the two CMAM Surge CEAs carried out to date and suggests priorities for future cost analysis related to CMAM Surge.

What we know about CMAM’s cost-effectiveness

A recent review of five CMAM cost-effectiveness studies suggests that CMAM is cost-effective (AAH & SCI, 2020). Several limitations, however, must be considered when interpreting the results including some inconsistency in methods across studies and contextual aspects that were not always well accounted for but likely influenced cost-efficiency and cost-effectiveness. Box 1 defines the CMAM CEA measures commonly used. The review found that the cost per disability adjusted life year (DALY) ³ averted by the five CMAM programmes assessed ranged from USD23 to USD53. This is within the international benchmark for a highly cost-effective intervention set by the World Health Organisation (WHO), i.e., the cost per DALY was less than the national annual Gross Domestic Product (GDP) per capita for that country (Marseille et al, 2015).⁴ Other cost effectiveness metrics generated by the studies include cost per child treated, cost per child cured and cost per death averted for children with severe acute malnutrition (SAM) and the values for these measures (such as cost per DALY) were also broadly similar. 

The review concluded that more cost-efficiency and cost-effectiveness analyses of CMAM using standardised methods are needed, particularly for large-scale and longer-term community-based programmes as well as for new approaches to delivering SAM treatment services. It also recommended that future studies better account for the influence of key contextual aspects including the scale of the programme, population density, wasting burden and health system factors.

Box 1: CMAM CEA measures explained

Cost per child treated is how much it costs to treat each child, usually regardless of the treatment outcome (although in some cases, including in the studies below, children who defaulted were excluded).

Cost per child cured/recovered represents the cost for each child who reached the criteria for recovery/being cured of acute malnutrition.  

Cost per DALY averted is the cost of averting loss of one life year that has been adjusted for any additional disability incurred. These can be compared to a) the single fixed standard of less than USD100 at the time of analysis and/or b) the per capita GDP of the country in which the intervention is implemented.

Cost per death averted is the cost required to avert a death in the population of interest. Unlike DALYs, this does not take into account morbidity and associated disability.

What we know about CMAM Surge’s cost-effectiveness

We are in the early stages of understanding if and how the CMAM Surge approach can improve the cost-effectiveness of CMAM itself and the cost-effectiveness of health systems in preventing under-five mortality. To date, Concern has tried to frame how CMAM Surge could improve CMAM cost-effectiveness in the form of the VfM framework and carried out two cost-effectiveness analyses of the approach in Ethiopia and in Niger in 2018/19. Learning from these studies is outlined below.  

The CMAM Surge Value for Money (VfM) framework

The CMAM Surge VfM framework was developed in 2016 using UK FCDO’s approach to VfM which focused on economy, efficiency and effectiveness (see Box 2). The VfM framework suggested two main comparisons to focus on when assessing the cost-effectiveness of CMAM Surge: 1) CMAM Surge versus routine CMAM service delivery via the government health system and 2) CMAM Surge versus a more traditional, non-governmental organisation (NGO)-led emergency nutrition response. The VfM framework also outlines the key costs and effects that need to be measured to make these two comparisons. The first comparison is more straightforward, given how widely available wasting treatment services now are within government health systems. The second comparison requires finding (and seizing) an opportunity to assess the cost and outcomes of a more traditional, external emergency CMAM response, ideally in the same context and at the same time that CMAM Surge is being implemented. Alternatively, it requires building a theoretical picture of what such a traditional response would look like and cost. Based on the experience of trying to implement the VfM framework, as outlined below, the framework is currently being updated by Concern and will be available soon.

Box 2: Value for Money and cost-effectiveness definitions

Efficiency: How much do you get out in relation to what you put in, measuring the efficiency in delivering the expected outputs.

Effectiveness: The optimal use of resources to achieve intended outcomes.

Cost analysis: Cost analysis is a broad category that evaluates the cost of delivering an intervention and the components of the cost to help identify major cost drivers, e.g., the cost of ready-to-use therapeutic food (RUTF).

Cost-efficiency analysis: A type of cost analysis that analyses cost per programme output. In the context of treatment of wasting, this is usually cost per child admitted for treatment, regardless of the outcome.

Cost-effectiveness analysis: Cost-effectiveness analysis combines cost data with a programmatic outcome, e.g., the cost per child who recovered following treatment. This is usually expressed in a cost to effectiveness ratio.

CMAM Surge cost-effectiveness study in Ethiopia

This study was carried out as part of a broader evaluation of a CMAM Surge pilot in the Amhara Region of Ethiopia in 2018/2019 with funding from the United States Agency for International Development (USAID) (Fotso & Myatt, 2019). The study followed the logic of the VfM framework and was designed to compare the cost per child cured and the cost per DALY averted by treatment services for wasting delivered via CMAM Surge with the standard intervention in two woredas over a 12-month period. The outcomes of the study were the number of children cured, the number of deaths averted (i.e., the number of lives saved) and the number of DALYs averted. The CMAM Surge approach was implemented in one woreda (Bati – the ‘intervention woreda’) and a more traditional emergency nutrition response was expected to be triggered in the second woreda (Dewa Cheffa – the ‘comparison woreda’). The expected emergency response included a six-month support package of staff, transport, supervision and a strengthened supply chain for ready-to-use therapeutic food (RUTF).  

In practice, the expected emergency response was not triggered in the comparison woreda as planned. This was because the situation was not classified as a ‘hot spot 1’ woreda by the government but rather a ‘hot spot 2’ woreda, meaning it was not prioritised for the full emergency response package via central humanitarian funding.⁵ Thus, Concern provided a more basic package that included refresher CMAM training and some basic logistics support. As such, the study reverted to a more basic comparison between CMAM Surge and ‘routine CMAM’.  

The study found that the CMAM Surge and routine CMAM arms were both highly cost-effective at USD21.58 and USD10.75 per DALY averted respectively – well within the WHO benchmark of three times the national per capita GDP which, for Ethiopia, was USD772 in 2018 according to the World Bank. CMAM Surge, however, turned out to be less cost-effective than the routine CMAM package delivered via government health services (USD21.58 (95% confidence interval: 16.38 – 28.20) for CMAM Surge and USD10.75 (95% confidence interval: 8.47 – 14.88) for ‘normal’ CMAM). The same was true for the cost per child cured – CMAM Surge was more expensive at USD349.55 (95% CI: 324.28 – 377.30) versus USD135.56 (95% CI: 127.11 – 144.40).

There were several limitations to this study that are important when interpreting the findings. Firstly, there were additional costs directly associated with the set-up of CMAM Surge, such as trainings, that were not required for routine CMAM, given that CMAM Surge is a new approach. Secondly, because the CMAM Surge set-up and the study itself required closer supervision and data collection, Concern undertook many of the programme support activities in the Surge woreda that would usually be covered by the woreda health office. Finally, the number of children treated for SAM in the CMAM Surge woreda (891) was lower than in the routine CMAM (1,286) during the study period and caseloads were relatively low across both woredas during the study period (four admissions per month versus three admissions per month for routine and Surge CMAM respectively).

CMAM Surge cost-effectiveness study in Niger

This study was conducted across two health districts in the Tahoua Region of Niger as part of the wider evaluation of the CMAM Surge approach in Niger and Ethiopia with funding from USAID (European Civil Protection and Humanitarian Aid Operations (ECHO) supported most of the programme implementation costs). The aim of the Niger study was to assess the cost-effectiveness of the CMAM Surge approach across the two health districts in relation to the WHO benchmark – no comparison arm was planned. The analysis also sought to observe any differences in costs or cost-effectiveness measures between health facilities that had been implementing CMAM Surge longer (since 2016) versus more recent adopters (2018). Similar to the Ethiopia CEA, the Niger study found that CMAM Surge was highly cost-effective at USD26 per DALY averted – well within the WHO benchmark of three times the national per capita GDP which, for Niger, was USD378 in 2017 according to the World Bank.

Comparing CMAM Surge cost-effectiveness with CMAM cost-effectiveness

The results from the Concern CMAM Surge CEA studies in Ethiopia and Niger were quite similar to the results of the CEA studies included in the recent review of cost-efficiency and cost-effectiveness of the management of wasting in children (AAH & SCI, 2020) in terms of cost per child recovered, cost per DALY averted and cost per death averted (Table 1). This is true for both the CMAM Surge intervention and the ‘normal’ CMAM arm in Concern’s study in Ethiopia although the comparison also highlights the particularly low cost per DALY and low cost per death averted seen in the normal CMAM arm in the Ethiopia study.

Table 1. Cost-effectiveness results of CMAM Surge versus those available for CMAM

(Adapted from the Cost-efficiency and cost-effectiveness of management of wasting in children (AAH & SCI, 2020)

Challenges in measuring CMAM Surge cost-effectiveness

Several challenges have emerged during our efforts to assess the cost-effectiveness of CMAM Surge relative to other, more routine modalities of wasting service delivery. Some of these challenges stem from the nature of the CMAM Surge approach itself and others reflect broader challenges inherent in cost-effectiveness and VfM evaluation methods. While the DFID VfM framework offered a useful foundation for outlining the main comparisons and data needed to assess CMAM Surge cost-effectiveness, a number of limitations exist that DFID/FCDO has recognised and is working to overcome. These include a tendency to prioritise short-term results over longer-term change and a limited ability to include the outcomes of a programme or intervention that are important for impact but harder to quantify (DFID, 2013). Three main challenges have arisen in the process of trying to evaluate the cost-effectiveness of CMAM Surge that are important to consider when planning any future studies:

Cost-effectiveness analysis is a very general science: Given the variation in CEA methods and assumptions and the influence of contextual factors, comparisons across studies and implementation settings can be challenging. While DALY was devised to make comparisons across interventions and contexts easier by creating a more relative measure, they must still be compared with caution and with strong consideration of contextual aspects.  The WHO benchmark for a highly cost-effective intervention – that the cost per DALY averted should be less than the country’s per capita GDP – is useful for general guidance. However, given that even a low-income country like Ethiopia has a per capita GDP of USD858, many different interventions will be considered highly cost-effective and prioritising remains a challenge.

CMAM Surge is a process, not strictly an intervention, and its unique outcomes are not yet quantifiable: CMAM itself – although it has many components – can be classified as an intervention for the purposes of comparison because it has standard treatment outcomes that are quantifiable. CMAM Surge, on the other hand, is an enhanced process that helps to deliver quality wasting treatment services (CMAM) in certain contexts. However, many of its additional benefits can be difficult to discern if using only standard effect outcomes, such as the number of children treated or cured. The additional benefits of CMAM Surge often lie in less quantifiable, more qualitative improvements seen within the health system, such as the improved morale of health facility staff as they gain the skills to better manage their workload and deliver quality services throughout the year. Because CEA focuses specifically on assessing the quantifiable impact of an intervention, such positive spill-over effects are often left unaccounted for. As outlined above, this is a widely recognised challenge when using standard CEA approaches such as DFID’s VfM framework, particularly when assessing interventions with societal outcomes and impacts.⁶ Defining costs that are unique to CMAM Surge as opposed to general CMAM costs can also be challenging. A more detailed analysis of the cost categories, using a refined theory of change for CMAM Surge, will help to reflect on the cost related to the processes.

CEA generally requires a comparison scenario which is often difficult to create and capture in real-time: CMAM Surge was designed to move the nutrition community away from more traditional emergency nutrition responses that are expensive, often come late and are not adapted to existing local capacities. Such traditional emergencies are often difficult to predict and might not happen during the study period or in a comparable context. As the humanitarian and development sector are gradually shifting to more early warning and disaster preparedness strategies, comparing the cost-effectiveness of CMAM Surge with a full-blown nutrition emergency response may not be as pertinent as it was 10 years ago. It would, nonetheless, strengthen the argument for more preventative than reactive humanitarian action. The only alternative to a real-time comparison would be to build a hypothetical traditional nutrition response scenario which would require a large number of assumptions and compromise comparability. In addition, it is critical that CEA it is done over several years as one of the main advantages of the approach is its ability to help health systems cope with caseload surges over time.  

Priorities for cost analysis of CMAM Surge going forward

The main question that has emerged from the experience so far is whether more robust evidence demonstrating that the CMAM Surge approach is more cost-effective than standard CMAM is actually needed. More specifically, do we need to demonstrate that CMAM Surge is more cost-effective than the predominant delivery system for wasting treatment services via government health services or a traditional emergency response? Our emerging conclusion is – probably not. Recognising that CMAM itself appears to be broadly cost-effective (although more research is needed to strengthen this evidence) and that CMAM Surge is a variation on the original CMAM model, we feel it would be more worthwhile to invest resources to improve our understanding of how the costs and the process of costing surge activities could be improved for more effective CMAM Surge implementation. We therefore suggest that the following areas are prioritised in relation to the cost analysis of CMAM Surge:

Focus on strengthening the step of costing CMAM Surge activities within the approach rather than on broader cost-effectiveness analysis for now

Defining and costing surge actions to be triggered when a caseload threshold has been crossed is a critical step (Step 4) of the CMAM Surge approach. Understanding how this step is currently being implemented and how it could be improved will help to improve the effectiveness of the approach and the design of future CEA studies. In 2021, Concern plans to engage with partners in a review of the specific actions that are being identified in Surge Action Plans across countries, how they are being costed, how much they cost and, finally, if cost is a barrier to triggering agreed actions when caseload thresholds are crossed (Steps 6 and 7).  

Experience to date has shown that the costing process (Step 4) is often not being carried out in sufficient detail. This is likely because costing is set to take place at the health facility level in the current CMAM Surge guidance. This makes sense for smaller surge actions, as health facility teams are very able to determine the ‘cost’ of activities that only require time or minimal monetary inputs. Costing of the more significant surge actions, however, such as additional health staff or more support/cash for transporting additional RUTF can really only be done well by the District Health Management Team (DHMT) who, ultimately, will be the those who provide such support. In the revised CMAM Surge global guide (due at the end of 2021), this costing step must be more explicitly placed with the DHMT within the eight CMAM Surge steps and more extensive tools and guidance provided.⁷ This will help to ensure that the costs associated with each action are accurately calculated and, most importantly, will facilitate the inclusion of critical costs within the district health budgets or other contingency budgets at district level or higher. This, in turn, will provide health facilities with greater assurance that support will be delivered when caseloads rise, motivating them to invest in more detailed planning.  

Continue to refine the costs and more quantifiable outcomes that are unique to CMAM Surge

There are several standard cost categories specific to CMAM Surge including additional training and mentoring on the CMAM Surge steps and potentially additional staff time required to set-up the approach and monitor caseloads – these are beyond what a health facility team might do normally. These need to be captured more systematically in any cost analysis for CMAM Surge going forward. At the same time, we should continue to improve monitoring and evaluation tools to better capture the ‘softer’ impact of CMAM Surge. This will enable us to better measure the positive (or negative) effect of the approach on the capacity of the health system to deliver essential services and, in time, allow us to better measure its cost-effectiveness. This will require working closely with health system experts who routinely measure health worker and health user satisfaction, health worker capacity and health system functionality.

Develop a set of practical tools to capture CMAM Surge costs as part of implementation

For the reasons outlined above, a set of easy-to-use tools for collecting data on the costs associated with CMAM Surge is needed to improve implementation and lay the groundwork for potential cost-effectiveness studies in the future if and when we feel they would add value. This set of tools should build on those that already exist for CMAM costing and planning, such as those developed by FANTA (2012) and those developed as part of the two CMAM Surge cost-effectiveness studies in Niger and Ethiopia (Fotso and Myatt, 2019; AAH & SCI, 2020). These tools, however, must be adapted to include costs that are unique to CMAM Surge and to collect data at the critical steps in the CMAM Surge process at health facility and district level. This will require an inherent level of flexibility and iteration to capture both the cost of activities that were planned and agreed if/when thresholds were crossed and the costs that were finally incurred for the activities and support actually delivered. Such a prospective and retrospective costing process will be required as part of the annual CMAM Surge process in each setting to learn and to improve efficiency for the next Surge implementation cycle.

Conclusion

Cost-effectiveness is a critical consideration in assessing the added value of a new approach such as CMAM Surge. To produce meaningful results, we must distinguish how CMAM Surge differs from routine CMAM and how to quantify outcomes and costs that are unique to the new approach. We consider that more practically focused cost analysis, a strengthening of the costing component of the CMAM Surge process and the development of experience-informed user-friendly costing tools related to CMAM Surge are the priorities.

Such developments would allow for more robust, comprehensive and longer-term CMAM Surge cost-effectiveness studies.

For more information, please contact Kate Golden.

Subscribe freely to receive Field Exchange content to your mailbox or front door.

Action Against Hunger and Save the Children International for No Wasted Lives (2020). The cost -efficiency and cost-effectiveness of the management of wasting in children: a review of the evidence approaches and lessons.

Cost-effectiveness analysis of CMAM Surge approach in Niger. Final Report. EVIHDAF Consultants and Concern Worldwide. September 2019.

DFiD (2013) Value for Money in the business case DfiD, 2013. Available from: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/307348/Value-Money-Business-Case-Small-Arms-Survey1.pdf

Fotso and Myatt (2019). Cost-effectiveness analysis of CMAM Surge approach in Ethiopia. Final Report. EVIHDAF consultants and Concern Worldwide. September 2019.

Idris, I. (2018). Cost-effectiveness in humanitarian work: preparedness, pre-financing and early action. K4D.

Marseille et al. (2015). Thresholds for cost-effectiveness of interventions: alternative approaches. Bulletin of the World Health Organization, 2015; 93:118-124. doi http://dx.doi.org/10.2471/BLT.14.138206