Evaluation of CMAM Pakistan: UNICEF country case study

Summary of evaluation¹

A flood affected community in Khyber Pakhtunkhwa province

CMAM has been implemented in Khyber Pakhtunkhwa province of Pakistan, with scale up in 2010 associated with the emergency response to flooding. UNICEF surge capacity support in CMAM scale up has contributed to good treatment outcomes, strengthened systems and better coordination. Future scale-up needs to address limitations in joint planning, nutrition strategy and framework for integration, technical guidance, coverage and programme performance assessment, local RUTF production and capacity. A funding gap between emergency and longer term programming hampers scale up. Improved interventions around IYCF for MAM and SAM are needed.

The Government of Pakistan (GoP) and the Department of Health (DoH) initiated the Community Management of Acute Malnutrition (CMAM) programme in Khyber Pakhtunkhwa (KP) in 2008. CMAM was scaled up following the flood disaster of July 2010 to address malnutrition and promote optimal child feeding practices. It was implemented in collaboration with UNICEF, WHO and WFP, and through implementing partners (IPs) and covered northern, central and southern districts of KP. Other child nutrition programmes included Infant and Young Child Feeding (IYCF) support and prevention and control of micronutrient deficiencies. CMAM is also implemented in Sindh and Punjab provinces.

Wasting has increased over the last decade in Pakistan, with prevalence estimated at 11.8% in 2001, 13.1% in 2006, and 16.8% in 2011. UNICEF undertook a recent evaluation to assess CMAM programme performance and gather lessons to inform scaling-up of CMAM.

The four CMAM components in Pakistan are: 1) Community outreach (screening, referral, follow-up, and community mobilisation), 2) Outpatient therapeutic programmes (OTPs) for severe acute malnutrition (SAM) without complications and involving home-based administration of ready to use therapeutic food (RUTF), 3) Inpatient treatment in stabilisation centres (SCs) and (4) Management of moderate acute malnutrition (MAM) through a supplementary feeding programme (SFP).

The evaluation criteria of relevance and appropriateness, effectiveness and coverage, efficiency and sustainability were applied to CMAM components and cross cutting issues. To accommodate time and budgetary limits, data were obtained from secondary sources, health system databases, visits to sample CMAM sites and interviews with stakeholders. Quantitative data on beneficiaries were analysed to determine whether programme targets had been met. Qualitative data also supported the analysis. Programme data were analysed for sample districts from December 2010 to November 2011.

Findings

The evaluation determined that CMAM is a relevant and effective approach in KP for addressing SAM and that effectiveness, efficiency and sustainability of the programme can be improved through implementation of a number of recommendations. The implementation of CMAM in KP has been of good quality, resulting in a high rate of cure for children admitted with SAM. While the approach is evolving toward stronger integration into the national health system, CMAM’s potential is reduced by lack of government priority for nutrition and absence of a comprehensive national nutrition policy. Progress has been made in developing a Provincial Integrated Nutrition Strategy, which would be aligned to the Pakistan Integrated Nutrition Strategy (PINS) in KP. The PINS provides a strategic framework but does not have any funding attached to it.

Stronger adherence to global guidance is required for community assessment, results based planning and monitoring. There is currently no effective framework to guide integration of CMAM within the national health system. The national CMAM guidelines focus on treatment protocols and require expansion to discuss cultural adaptation, gender and equity, IYCF, and programme performance monitoring and to clarify screening, admissions and referral procedures.

UNICEF effectively provided surge capacity to expand CMAM, promoting success in meeting most Sphere standards for children admitted with SAM. UNICEF’s support resulted in the establishment of a Nutrition Information System (NIS) and strengthening of the nutrition cluster nationally and provincially. However, for scaling up, UNICEF needs greater headquarters and regional support and more staff members with nutrition expertise to promote nutrition policy and provide consistent guidance to the GoP and IPs. Longer term resource allocation to CMAM is required to retain the existing human resources required for strengthening the quality of implementation, the integration and to ensure an adequate response where and when required.

The partnerships among communities, the DoH, UNICEF, WFP, WHO and IPs effectively supported CMAM services, reaching 70,000 moderately malnourished children and 14,000 severely malnourished children in selected areas of KP. Treatment coverage for children with SAM cannot be determined in KP due to lack of coverage surveys.

Initial challenges were faced in gaining community acceptance of CMAM due to the traditional seclusion of women in their homes and lack of trust in the programme. More advance sensitisation is important to inform communities, particularly leaders, about the objectives of CMAM services and to prepare for discreet home-based screening. The Social Mobilisers (SM) through Nutrition Support Committees (NSCs) were instrumental in paving the way for Community Outreach Workers (COWs) to access households. Challenges include improving skills of community workers in using mid-upper arm circumference (MUAC), strengthening follow-up home visits of non-responders and defaulters and promoting sustainability of the NSCs. Effective follow-up of admitted children in Swat and Lower Dir can be attributed to higher incentives for COWs, well qualified IP nutrition staff, the formation of mothers’ groups and involving men in the programme. Data on screening and admissions are not structured by gender, groups, areas, and relative to population changes, making analysis of coverage and outcomes difficult.

Most admissions were children of 6-23 months of age highlighting vulnerability as complementary foods are introduced. The OTPs achieved a 91.5% cure rate for SAM cases and 7.5% default rate. The SAM treatment met Sphere standards for cured, default, and death rates, however, standards for weight gain and length of stay were not achieved in all sites. A number of CMAM sites in health facilities require upgrading to ensure adequate and well ventilated spaces and designation of play areas. Procedures for admissions, discharges and referrals to SCs are not sufficiently standardised, particularly for use of anthropometric measurements and for timely medical examinations to identify complications.

MAM management performance is effectively tracked through joint IP/WFP data collection. A 95% cure rate was achieved among registered children; the default rate was 4.2%. Sphere standards were met for cured, default and death rates, however, none of the districts achieved the average weight gain standard and some districts did not achieve the length of stay standard.

The SCs established in the paediatric wards of government-run hospitals were well managed but coordination was poor between OTP and SC both for referral to the SCs and discharge back to OTPs. The SCs require immediate support to structure programme data recording, which is currently insufficient for analysis.

The Ready to Use Therapeutic Food (RUTF) imported by UNICEF is well accepted by children but efficient usage was hampered by lack of compliance to prescribed intake, sharing of products with siblings, sale of the products, and supply shortages in some centres. Monitoring is inadequate to address these problems. There is limited national production of supplementary products but little progress has been made on promoting local production of RUTF.

Nutrition cluster support, coaching and dedicated information managers in IPs facilitate NIS effectiveness and on-site monitoring has improved. However, programme data are poorly tracked in some sites and relapse data need to be collected. To steer the programme more efficiently, more emphasis must be placed on ensuring data quality and consistently analysing performance indicators, as well as undertaking regular surveys, reviews and evaluations.

The scale-up of CMAM was facilitated by pre-existing relationships among government, UNICEF, WHO, WFP and IPs, active case finding, formation of NSCs, and established coordination mechanisms. Stronger earlier joint assessments may have enhanced implementation efficiency through the Pakistan Humanitarian Response (2010); the Flood Affected Nutrition Survey (FANS) was not conducted until four months after the disaster. Joint planning for supporting the national nutrition programme is not strong enough between the MoH, WFP, WHO, IPs and UNICEF. Stronger joint planning is also needed among provincial partners (UN, IPs and DoH) for setting and revising realistic programme targets as the situation evolves.

Multi-agency emergency response strategies effectively identified potential gender issues but no community assessments were conducted to ascertain programmatic means to address these issues. The national guidelines and assessments should more effectively guide staff to ensure coverage of groups with higher prevalence, such as girls and children aged 6-11 months. Partners are increasing their efforts to reach HIV/AIDS positive people and strengthen capacity development of women community workers and staff. CMAM integration into the health delivery system and greater involvement of civil society organisations and the private sector is important to reach malnourished children living in remote, crisisaffected and food insecure areas.

CMAM integration in KP is minimal to partial; IPs are responsible for staffing, monitoring and capacity development at higher costs and less sustainability. Stronger GoP roles are needed in accountability, financing and coordination. The time for integration planning is optimal given the strengthening of provincial authority. Progress has been made in establishing Memoranda of Understanding (MOUs) and partnerships among MoH and other relevant ministries, UN agencies and IPs, and development of a nutrition response plan.

The Nutrition Cluster effectively planned for intensive capacity development of 800 health care providers and IP staff in KP with follow-up refresher training in 2011. Greater capacity development was needed for protocols and IYCF and more in-depth training for inexperienced staff. Due to diverse qualifications and experience of staff providing CMAM services, their capacity needs have to be assessed and relevant training provided, aiming for consistency in service quality across districts.

The GoP makes a significant contribution to CMAM’s capital costs through provision of health facilities; support for heath staff and utilities comprises an estimated 7% of recurrent costs. The largest proportion (33%) of the externally funded recurrent costs is devoted to Ready to Use Supplementary Foods (RUSF) and RUTF. The costs per beneficiary differed by district and IP, the average costs were SFP ($21), OTP ($145) and SC ($230). Lower costs were associated with local NGO implementation due to lower cost of human resources.

The major issues for future scale-up are planning, capacity and funding. Mapping and prioritisation of target areas have not yet been fully undertaken. Joint planning is still weak for building institutional relationships to define nutrition policies and standards and indicators for integration, and set out tangible means to build DoH capacity. It may only be possible to achieve broader coverage through involvement of more local NGOs, private health practitioners and other civil society actors. The majority of stakeholders advocated for overcoming barriers to expanding national production of supplementary products and strongly promoting RUTF production. There is a funding gap between emergency and longer term programmes and a pooled common fund may help provide more consistent funding. Funds need to be secured in advance for coverage surveys, reviews and evaluations. Not enough resources are focused on prevention through IYCF counselling and promoting healthy local foods.

Recommendations

Key recommendations included the following: Advocate for and support joint planning for development of a national nutrition and provincial strategy which outlines the strategic priorities, assigns nutrition authority and coordination mechanisms, sets out capacity needs, and makes budget commitments for nutrition interventions. A focus on multi-sectoral and integrated longer term approaches is seriously required.

Strengthen and update national CMAM guidelines to include detailed protocols for referrals and admissions to SCs, more on IYCF, inter-sectoral coordination, and guidance on addressing cultural, gender and equity issues and monitoring programme performance.

Provide technical support to the UNICEF country office to design CMAM expansion. Strengthen monitoring and advocacy at the national and provincial levels by ensuring dedicated staff with nutrition expertise for managing CMAM.

Strengthen the NIS oversight to ensure reliable and consistent collection of gender disaggregated programme data and training of staff who are responsible for data recording.

Conduct coverage surveys in KP and track coverage as part of programme performance analysis.

Given the scale of MAM in Pakistan, seek alternative approaches to ready to use supplementary products, through researching local recipes, and strengthen IYCF through increasing numbers of COWs and Lady Health Workers.

Conduct a training needs assessment for each CMAM site and provide appropriate levels of training according to staff experience and knowledge. Evaluate training periodically.

Jointly establish indicators for progressive integration of CMAM into the national health system with the government taking an increasing role in accountability, monitoring, and implementation. IPs should develop resultsbased plans for contributing to DoH capacity development.

Expand in-country production of supplementary foods and strongly promote the development of RUTF production, for example, by providing technical assistance to food processing companies.

Advocate at high levels to secure funding commitments for scaling up with a view to supporting permanent integration of CMAM and a national nutrition strategy with coordinated projects and programmes. Seek alternatives for fund management such as a pooled fund that reduces overhead costs.

¹UNICEF (2012). Evaluation of community management of acute malnutrition (CMAM) Pakistan Country Case Study. UNICEF Evaluation Office. September 2012