WFP Monitoring and Evaluation of HIV/AIDS Programming in Malawi

By Jeremy Shoham, ENN

Post distribution monitoring GOAL food distribution Chiradzulu 2

This article was written based on a WFP consultation to Malawi in February 2005.

During 2003, WFP Malawi significantly strengthened the monitoring activities for all its programmes in Malawi. The Post Distribution Monitoring (PDM) and Community Household Surveillance (CHS) programmes implemented by WFP are most relevant to targeting in Malawi and provide invaluable insights regarding the role of using proxy indicators to target people living with HIV/AIDS (PLWHA).

Community Household Surveillance (CHS)

The CHS is a regional initiative in six EMOP (Emergency Operation) countries and was initiated in early 2003. CHS is based on sentinel site monitoring and has four main objectives;

• to monitor impact of food aid
• to monitor trends in food security
• to monitor links between food security and nutrition and/or HIV/AIDS, and
• to feed into early warning system information.

The two instruments for the CHS are a household questionnaire and a monthly focus group discussion. CHS is conducted by WFP field monitors, and staff from the National Statistics Office (NSO), Ministry of Agriculture and Irrigation, and NGO implementing partners.

CHS was initially piloted in Malawi in July 2003 in 30 sentinel sites. The first round took place in October 2003, with 30 sentinel sites in six randomly selected EMOP 10290 districts. Four Final Distribution Points (FDPs) per district were selected, with one village randomly selected from serving FDPs. In each village, 22 households were selected, comprising 11 beneficiaries and 11 non-beneficiaries. A second and third round CHS were conducted in February 2004 and October 2004 respectively, with a fourth round planned for February/March 2005.

The initial pilot CHS in July 2003 mainly collected information on the food security situation, including data on coping strategies and household perceptions on need for food aid. There was also some analysis of vulnerability. The first round of surveys collected and analysed data on food security and coping strategies but also focused on targeting. The first round report claimed that social targeting was successful, e.g. 37% and 27% female-headed households amongst beneficiaries and non-beneficiaries respectively. However, it stressed that the 'asset very poor' were less represented than richer households (see figure 1).

The second report, in February 2004, concluded that social targeting, i.e. female headed households, orphan containing households, elderly and disabled, etc, was successful, e.g. 41% of beneficiary households were female headed compared to 37% non-beneficiary households, and 26% beneficiary households had chronically ill compared to 21% non-beneficiary. It also stressed that 9% of beneficiaries had no social inclusion criteria, while 31% of non-beneficiaries had three or more social inclusion criteria. Furthermore, 54% of beneficiaries were asset poor, compared to 48% for non-beneficiaries.

*excludes livestock

Post Distribution Monitoring (PDM)

The main objectives of PDM are to monitor the use of food aid, satisfaction with food aid, and access to food aid (involving targeting inclusion and exclusion). It involves a household questionnaire and focus group discussions and is conducted by NGOs and WFP field monitors. The first PDM report was compiled in December 2003, containing information for September - December 2003 (EMOP 10290 began in July 2003). There had been considerable work in the development of the PDM prior to this, as well as evaluation of its implementation. A consultant on monitoring and evaluation was taken on at country office level, and experiences of PDMs from other countries with similar programmes were drawn upon.

The first PDM narrative report was based on a total of 1196 households, with a relatively equal distribution between beneficiary and non-beneficiary. Beneficiary households were those receiving food aid through the new EMOP under Food for Work (FFW), or Vulnerable Group Feeding (VGF), i.e. those infected or affected by HIV/AIDS, children, orphans, expectant and nursing women. Social vulnerability categories under this EMOP included female headed households (FHH), elderly headed, households with disabled member or orphans or the chronically ill (CI), or households with a high dependency ratio. Social indicators were combined with food security indicators, i.e. access to land, access to livestock, access to regular income or asset ownership.

The main findings of the first report were that targeting on the basis of social vulnerability was good but weak with respect to economic wealth indicators, i.e. overall access to livestock, income or land was very similar between beneficiaries and non-beneficiaries. This was believed to be due to a high level of inclusion errors. The report recommended refining inclusion criteria through systematic discussions with local communities and traditional chiefs, giving the opportunity to work out local checklists of social and economic indicators tailored to a given local context.

The January 2004 PDM report found a clear improvement of targeting with respect to social and economic welfare indicators following a beneficiary verification exercise in late 2003¹. The February 2004 PDM report stated that the VGF programme more frequently included households keeping orphans (73%), the chronically ill (31%) and FHH (24.4%) compared to FFW activities (43%, 19% and 43% respectively). It concluded that VGF targeting remains satisfactory with a predominant inclusion of beneficiary households with at least one social vulnerability criteria.

The March 2004 PDM introduced new analytical variables - overall inclusion and exclusion errors and targeting efficiency². The executive summary of the report states that inclusion error is 16% while exclusion error is 30%, and calculates that the targeting efficiency is 78%³. The June PDM found an inclusion error of 18% and exclusion error of 54%. Thus the targeting efficiency was 64%⁴.

Informing initiatives

The PDM has had a significant impact on the targeting system in EMOP 10290 and the current Protracted Relief and Recovery Operation (PRRO). Early findings, after the first report, led to an evaluation of targeting (Nsama report in March/April 2004) and an evaluation of the HIV/AIDS programme, with a particular emphasis on targeting in April/May 2004 (Selaphera Consulting Ltd).

Findings from the PDM and CHS have also led to a number of revised monitoring initiatives, including where:

• Implementing partners were strongly encouraged to undertake systematic and re-iterated targeting verification exercises.
• In May 2004, a collaborative process was initiated between WFP and the JEFAP5 cooperating partners to review the targeting criteria. As a result, a final set of inclusion and exclusion criteria was thoroughly discussed, agreed, and later on included in the revised implementation guidelines (JEFAP III).
• Implementation of the new set of criteria and plans to appraise the new set of inclusion and exclusion criteria.

Conclusions

Although there was no rigorous monitoring of targeting during the initial EMOP (1200), partly due to the need to prioritise implementation of the general food distribution, the monitoring evolved rapidly into an extensive system which collected a variety of data needed to assess compliance with targeting. The system has compiled a large quantity of extremely useful data. The level of sophistication achieved with regard to monitoring targeting is unique and could serve as a model for other programmes.

The findings of the monitoring, in particular the weaknesses of targeting on the basis of economic criteria, were rapidly identified and reflected in JEFAP 111 guidelines.

Some of the claims regarding the success of social targeting may have been over-emphasised, i.e. the differences found between beneficiary and non-beneficiary groups were not statistically tested and almost certainly not of sufficient margin to make certain claims. Furthermore, there seems to have been an undue emphasis on inclusion criteria rather than exclusion criteria, which showed very high levels reflecting the large proportion of very poor and poor in Malawian communities. It can be argued that exclusion errors are more important than inclusion errors, because they are divisive at community level.

The monitoring has a number of methodological weaknesses:

• Comparing assets between beneficiaries and the non-beneficiary population is conceptually flawed, as one would expect nonbeneficiaries to begin disinvesting in an emergency so that asset ownership would begin to equalise during the programme.
• Inclusion and exclusion criteria are normally calculated on the basis of proportion of beneficiaries out of total target population, rather than proportion of beneficiaries who meet inclusion and exclusion criteria. The adopted method may lead to an underestimate of targeting error.
• The differences between beneficiary and non-beneficiary inclusion/exclusion percentages are not compared statistically.

Recommendations

Given the lack of success of economic targeting, there is clearly a need for more piloting and research to determine when and where such targeting is feasible and appropriate. For example, where a large proportion of the population are economically 'poor' or 'very poor', it may not be feasible. It may be that in such circumstances, social targeting is more acceptable at community level while economic based criteria will lead to conflict and disagreement. An implication of this may be that 'the first cut' of targeting should be social targeting, with economic targeting subsequently invoked once these households have been identified at community level. In order to assess efficiency of economic targeting, it would have been better to carry this out at the onset of the programme or following new registrations as, during the programme, a process of equalisation is likely to occur and non-beneficiaries are forced to disinvest. Where possible, statistical tests should be applied. It should have been possible to apply non-parametric statistical tests (e.g. pearson coefficients) to compare social inclusion /exclusion for beneficiary and non-beneficiary populations. Had this been done, the findings would have been more credible.

For further information, contact Jeremy Shoham, email: jshoham@easynet.co.uk

¹The percentage of beneficiary households with access to live stock decreased from 30% to 21%. However, there was no improvement with regard to access to land.

²The analysis considers eight major exclusion indicators proposed by the JEFAP monitoring working group. These relate to livestock, acres of food crop and cash crop land, income as medium business and income as formal wage. There are six inclusion socio-vulnerability indicators. These are single headed household, more than seven persons in a HH, elderly headed, female headed, keeping orphans and keeping chronically ill.

³In order to assess the targeting efficiency, a targeting index is derived for each household, which is composed of the difference between its number of inclusion and exclusion criteria. For the purpose of simplicity, each exclusion indicator is given a negative one (-1) and each inclusion indicator a positive one (+1), When the two are aggregated, a deserving household is one that has a positive difference, i.e. more inclusion indicators than exclusion indicators. A negative targeting index will mean that the household has more exclusion criteria than inclusion criteria and hence could be considered as food secure.

⁴It should be noted that there are different figures in the text of the report for inclusion and exclusion and efficiency (16% and 73% and 78% respectively). Joint Emergency Food Assistance Programme, a country wide consortium of 12 international and local NGOs in Malawi