CMAM Report: development of a global online reporting system for CMAM programming
By Rachel Evans, Christoph Andert and Susan Fuller
The authors would like to express their gratitude to all individuals that have contributed to the development of the CMAM Report since 2008: Carlos Navarro-Colorado, Jeremy Shoham, Vicky Sibson, Mark Yarmoshuk, Jennifer Martin, Lourdes Vazquez, Shafat Sharif, Emily Mates, Emily Keane, Geraldine Le Cuziat, Andrew Crossman, Jaime Hargreaves, Terry Reed, Vasantha Mountford, Stewart Gordon, Elena Pullara, the MSM software developer’s team around Graham Douglass and Simon Gibbs and all consultants. Further acknowledgements go the steering committee of agencies that contributed to the development of the standardised reporting categories and indicators used in CMAM Report - CDC, Concern Worldwide, FANTA2, IRC, Nutrition Information in Crisis Situations (NICS), Save the Children UK, UNHCR, UNICEF, Valid International, WFP, WHO and World Vision – and all agencies that improved the software by giving continuous feedback to the development team – Concern Worldwide, GOAL, International Medical Corps (IMC), ACF-USA, Eastern Deanery Aids Relief Programme (EDARP) and Save the Children.
The MRP/CMAM Report was funded by OFDA 2008-11, ECHO 2011-12 and HIF 2013-15. The CMAM Report online software was developed by Management Systems Modelling (MSM) Ltd, UK
Since the publication of the Humanitarian Practice Network (HPN) Network Paper Measuring the effectiveness of Supplementary Feeding Programmes (SFPs) in emergencies in 20081 highlighted inconsistencies, inadequacies and bias associated with reporting on Supplementary Feeding Programmes (SFP), the idea of standardising community based management of acute malnutrition (CMAM) reporting categories and indicators across implementing agencies and countries has gained traction. This has culminated in the development of a comprehensive reporting system for all aspects of CMAM, which features a web-based central database called the ‘CMAM Report’. This article describes certain experiences and lessons learnt from the development process of this reporting system and its use as a management tool in CMAM programming.
Background
The CMAM Report started out in 2008 as the Minimum Reporting Package (MRP), with the formulation of standardised reporting categories and indicators for SFPs coordinated by a steering committee of twelve international agencies, led by the ENN and Save the Children. These were piloted in four countries. In 2011/12 and in response to the feedback from the pilot, Outpatient Therapeutic Programme (OTP) and Stabilisation Centre (SC) reporting categories and indicators were added through a similar consultative process and a Microsoft AccessTM based reporting software was developed. Both reporting standards and software were subsequently rolled out by seven non-governmental organisations (NGOs) in 15 countries; Save the Children, Concern Worldwide, International Medical Corps (IMC) and GOAL were the main users.
A number of challenges were identified following this first phase in relation to the indicators and the software. Despite guidance on standardised indicators, these were still not fully adopted, even by agencies committed to the MRP principles. The software appeared not to be user friendly enough. Evaluations and feedback on the software also emphasised the focus on NGO implementation, the lack of capacity for coordination and the potential for parallel systems. For example, there were concerns that the MRP added another layer of reporting on top of government reporting systems (this was not the intention but rather to be a complementary initiative, whereby the standardised reporting categories and indicators of the MRP could positively influence government systems in the long run and where systems needed strengthening).
With these issues in mind, a new phase of software development and reorientation took place 2013-15. The software was redesigned into a global web-based database with a specific function for United Nations (UN) agencies and the Ministry of Health (MoH). Mid upper arm circumference (MUAC) screening and blanket supplementary feeding programme (BSFP) components were added. The system was finally renamed as CMAM Report.
Developing an online software
In order to address all the challenges identified with the initial software, Save the Children decided to modify the software to a web-based tool with offline capability. The aim was to develop a system that would not require installation and that gave access to data from several levels via synchronisation with a central server. The actual development of the software was given to a software developer but additional funding was needed to meet the cost of development, which proved several times higher than expected. When development started, it was through an iterative process between the developer, the Save the Children IT team (who reviewed system structure and general functionality) and the Save the Children Nutrition team (who reviewed content and usability). For each iteration delivered by the software developer, testing took place by Save the Children IT and nutrition experts and was reported back to the developer. The main phase of software development was completed in June 2014.
In parallel to the software development, the team at Save the Children facilitated consultations with partners and users, which resulted in a number of changes to the software and the addition of some new components:
- The structure of the software was changed from being focused around programmes funded through specific grants to the unit of focus being sites, i.e. programme feeding sites, health centres or hospitals are the main level for data entry and analysis. Grants and contextual information can be attached to this to support 1-click reporting if desired.
- INS3has been renamed within the software to ‘OTP discharges in TSFP.'3
- A ‘Country Administrator’ user access level was added. This is a view-only function that allows the MoH and UN agencies to view and analyse data from all agencies in a given country.
- Stock tracking, MUAC screening and BSFP components have been added.
- The software and all supportive material are now available in English and French.
With the wider launch to the nutrition community mid-2015, CMAM Report can now be used on computers, laptops and tablets with a common browser4 and is freely available to all agencies/countries that wish to use it. The software supports secure data collection through controlled access from the field via country to headquarters levels (five adaptable user access levels). It enables the use of standardised reporting categories and indicators (enables comparable data and unbiased reporting) and gender and age disaggregated reporting through all reporting categories. Users can analyse data for trends and create summary tables and graphs by feeding site, group of feeding sites and geographical location up to country and global levels for any required time period in one month steps. The system features export of tables and graphs to PDF, Excel and Word. Raw data can be selectively exported to Excel and in turn to statistical software. Data entry mistakes are reduced by real-time validation checks.
Lessons learned in software development
The main challenges in establishing this system have been due to the lengthy time taken to develop complex new software, the unexpected costs of this process and lack of a framework for defining what constitutes a change to the original specification for the new software.
The following lessons can be taken away from the development process:
- Software development projects require flexibility in delivery time and financial resources in order to meet needs which evolve over time.
- Clear definition on what constitutes change in specification should be defined with the software developer early on.
- Wider learning is essential as use of Information & Communication Technology (ICT) increases in our programmes. This kind of development requires a specific skill sets and level of shared understanding and common language between both IT and nutrition expertsto build a software that is fit for purpose for the end users in the field
- The potential for parallel systems and duplication of efforts remains; there are a number of ongoing related initiatives (e.g. other agencies who are developing national and global reporting systems with similar objectives). Transparency and coordination around such initiatives is essential in order to avoid duplication and maximise reporting effectiveness. Lessons should be shared between developers of different systems.
- Systems should be flexible in terms of indicators to be collected, but definitions should be standardised.
- There is much sensitivity around data storage, sharing and ownership. There may be some governments who do not wish for their data to be stored outside their country. There are questions as to whether NGO collected data belongs to the NGO or to the government (since it concerns their citizens).
- Advanced systems are not a solution to poor quality data; the system is only as good as the data.
- Software development projects should be adequately resourced, both technically and financially.
- In the world of information systems, there is no one solution that would fit everywhere; platforms which can link different systems are likely to play a vital role in facilitating more efficient ways of working in the future.
Examples of good practice and added value
Several examples of good practice were found by a recent evaluation.
Using the CMAM Report software to inform programme management
In Yemen, high defaulter rates were revealed and resulted in the management team adapting their programme delivery model. In Asia, a large number of defaulters were revealed due to a supply chain break. The CMAM Report software made this visible and results could be traced back to specific sites (Figure 1). In West Africa, the CMAM Report clearly showed a number of programming issues and therefore where to focus technical support. Stock management is one of the main challenges for CMAM programming and the CMAM Report is particularly effective and useful for estimating caseloads and supply needs, e.g. in Puntland, Somalia.
Box 1 - Yemen experience
The CMAM Report software replaced Excel sheet reporting; this was prone to errors and had limited use. The Yemen team cited the following as ways in which the CMAM Report has added value to reporting:
“With the CMAM Report, we can see indicators directly when entering data – a major advantage over the old system. We can take actions if indicators do not reach Sphere standards in single feeding sites. Data in the CMAM Report format is shared with the Ministry of Public Health and UNICEF. The CMAM Report helps to improve the quality of the programme. Before starting the CMAM Report, the defaulter rate of the programme was very high. With the CMAM Report it has been easier to monitor the data and to take corrective actions.”
Highlighting issues with data quality
CMAM Report has been useful in highlighting issues with data quality across programmes, partly through the standardisation to calculate performance indicators. Using the CMAM Report software, indicators can also be easily compared across programmes and compliance and validation reports have highlighted missing, late and erroneous data. One organisation conducted a global assessment of missing, late and erroneous data during the initial months of CMAM Report implementation and it was observed that data quality improved during that time.
Using standardised indicators to inform national reporting systems
In some contexts, agencies have also entered into discussions with partners, government and UN agencies on using the standardised indicators to inform national reporting systems:
Somalia: Save the Children Somalia was invited to present the CMAM Report as a potential reporting tool for a nutrition consortium in April 2014 that includes the CMAM implementers IMC, ACF and Oxfam. The additional utility of the tool was recognised and it was decided to use the CMAM Report as a consortium reporting tool.
Philippines: Save the Children invited partner staff for training on the CMAM Report indicators and software in Manila in May 2013. Attendees included the Department of Health at the Autonomous Region in Muslim Mindanao, WFP, UNICEF and ACF. Partners identified the CMAM Report as a strong Save the Children system. However, further efforts to de-brand the CMAM Report are required so that external agencies feel comfortable in using the system. This reflects a constraint in locating a sector-wide initiative within an operational agency; there is risk of less buy-in by others who consider it an agency-specific product.
Yemen: Save the Children and IMC presented the CMAM Report at a cluster meeting and it was well received. In addition, a joint CMAM Report training was conducted in Yemen between IMC and SCUK staff.
West Africa: Agencies are using their own internal databases due to the lack of a strong national system in many countries. There is therefore a need for a regionally standardised CMAM reporting tool which could be met by the CMAM Report software.
The added value of the software reported by users is listed in Box 2.
Box 2 - Added value of the CMAM Report software
- Data are easy to manage and share
- Reduces reporting time compared to Excel
- Error proof as false data can’t be entered
- Highlights missing data, can be used to enforce better reporting
- Compliance reports improve timeliness of reporting
- Facilitates trend monitoring to feed into donor reports/stock checks
- Easy to extract indicators for donor reporting
- Useful for programme review
- Field staff using analysis functions to produce reports and graphs
- Analysis by feeding site possible, so easier to target support in programme management Able to verify/compare with national reporting system to do a data quality assessment and inform monthly data discussions with cluster
- Graphs highlight problems assisting management
- Support from head office is easier as programme details can easily be accessed
- Good information source
- Encouraged monthly instead of quarterly HQ reporting and increasing transparency
- Remote supervision and technical support is easier
- Transparent - easier to trace back results to original sites.
The software, user manuals, standardised indicator guidelines, e-learning, demo version and more information can all be found at www.cmamreport.com.
For more information and to receive your personal login details, contact: cmamreport@savethechildren.org.uk
1http://www.ennonline.net/sfpeffectiveness
2Individual Nutrition Support: A term used for beneficiaries that move from OTP to TSFP and that are kept separate from MAM cases that enter TFSP. This allows unbiased calculation of TSFP performance.
3Targeted supplementary feeding programme (TSFP)
4 Not running on Apple devices