Monitoring the targeting system (Special Supplement 1)

In the past, little emphasis has been placed on monitoring what happens to food aid after it has reached the distribution point. Donor reporting has been limited to the delivery of food to its intended destination (Jaspars and Young, 1995). For this reason, "good systems of monitoring and evaluation, to establish whether food aid is indeed reaching intended beneficiaries (and at reasonable cost of delivery), are disturbingly rare." (Barrett, 2002)

As noted in Section 2, targeting is never 100% accurate and there is scope for errors of inclusion and exclusion to arise at every stage of the targeting process. Table 2 highlights some of the key factors that lead to inclusion and exclusion errors at each stage. In the design stages of the targeting system, decisions are made as to which type and degree of error is more acceptable. For example, the aim may be to reach HIV/AIDS affected households by targeting households with a chronically ill household head. If this is done, it must be recognised that this will encompass households with non-HIV/AIDS chronic illness and that there will, therefore, be an unavoidable (although perhaps desirable) inclusion error. The broader and more inclusive eligibility criteria become, the smaller the exclusion error and the greater the inclusion error, and vice versa. Hence, as inclusion errors rise, exclusion errors decline and as exclusion errors rise, inclusion errors decrease.

In the implementation of targeting, i.e. once the quantity of food has been fixed and the size of the targeted population is defined, the relationship between exclusion and inclusion errors changes. That is to say, as inclusion errors increase so do exclusion errors. For example, food distributors could decide to share the food among a greater number of households, including those that are very poor but not affected by chronic illness. This may mean that the numbers of originally- targeted households who receive the food is reduced, or the quantity each household receives is reduced, resulting in a higher exclusion error.

Monitoring exclusion and inclusion assumes that the target group defined in the objectives was correctly identified as the group in greatest need. In addition to inclusion and exclusion errors, a well designed monitoring and evaluation system should determine whether a) the decision to target food within a geographical area was appropriate, b) whether the groups in greatest need were identified by the assessment, and c) whether the objectives were achievable. In practice, inclusion errors may occur because the community has a clearer understanding of need and deliberately subverts the eligibility criteria. In this instance, simply monitoring inclusion and exclusion will show failures in the system whereas in fact, because of the errors, the targeting system may be more effective in meeting people's needs. Thus, it is very important that monitoring allows review of the design of the system itself, as well as checking its implementation.

Table 3 summarises the methods that can be used for monitoring the targeting system, and Table 2 indicates at which stages the various methods can be used. These methods are described briefly. It should be noted that there is no single method for determining inclusion and exclusion errors or the appropriateness of the targeting system. Each of the methods listed here provides part of the information necessary to comprehensively monitor a system.

The information provided by monitoring should be considered immediately and the implications for the targeting system determined.

Process monitoring and evaluation

Systems should be established for ongoing monitoring of the views of the population on the targeting system. Ongoing methods for participatory monitoring is an integral part of community managed targeting. In systems using administrative targeting, appeal mechanisms can be set up to allow members of the affected population to complain if they have not been targeted or have not received food. These mechanisms can provide an immediate way of monitoring the targeting process. Where appropriate, complaints should be met with an instant response and if necessary, modification of the system.

Food basket monitoring

Food basket monitoring is conducted immediately after receipt of the ration at the distribution point. It monitors whether the ration that is supposed to be being received by a registered beneficiary is indeed received and thereby, provides a measure of exclusion error. As shown in Table 4 (see p25) planned rations often differ from those received because the number of recipients is greater than planned, and so the food is shared 'more thinly' amongst them. Table 4 shows that in Tanzania, the actual ration size was 10.3kg per capita per month, compared to a planned ration of 12 kg per capita per month. This arose because 15.4 per cent of households were underregistered, where not all household members were registered in order to allow more households to be included in the distribution.

Thus, food basket monitoring data is useful if analysed in conjunction with household profile monitoring data and with the targeting objectives. On its own, food basket monitoring does not tell you whether the beneficiary has been correctly targeted (i.e. whether they should have a ration card) nor indeed, whether they may be receiving two rations because they have two ration cards. Neither does it highlight individuals who are incorrectly excluded from the distribution. Food basket monitoring has been used predominantly in refugee situations where ration cards are widely used. (Jaspars and Young, 1995)

Household profile monitoring

Household profile monitoring is very important for monitoring systems which target households, and is a primary measure of the accuracy of the registration process. It also indicates the level of inclusion and exclusion error, assuming the size of the targeted population is known. Household profile monitoring involves determining whether the recipient households meet the eligibility criteria. Box 17 provides some results of household profile monitoring in Zimbabwe.

Post distribution monitoring

Three types of post distribution monitoring can be used for monitoring targeting:

Project village of a Concern targeted food distribution programme in Afghanistan

Food usage surveys can contribute to an understanding of inclusion errors by highlighting the extent to which food is redistributed by the recipients, either voluntarily or involuntarily post distribution. It can also help to determine whether the food aid is likely to achieve its desired objectives, e.g. if all the oil in the ration is sold in order to purchase more grain, then any nutritional objective may be harder to achieve.

Table 4 shows the data from a food usage survey in Tanzania conducted during community managed food aid targeting. The questionnaire used for the survey was pre-tested and modified when it became clear that the majority of redistribution or sharing of the ration was with unregistered community members, and occurred through the provision of cooked meals rather than dry food. A small proportion of the ration fulfilled social obligations such as kinship support, debt repayment and the cost of ceremonies. The survey showed that the food would last the average household approximately 2 months, rather than the intended 3 months (Save the Children, 1999). The survey was conducted in a sample of households covering a large geographic area, using a similar sampling frame to a 30x30 cluster anthropometric survey.

Market surveys are often conducted to monitor food aid sales. Sale of food aid should not be interpreted as a targeting failure. All households have needs for cash to meet non-food needs and may have to sell some food to obtain cash to buy other essential commodities, such as soap, clothing, fuel etc. Data from market surveys can be analysed alongside food usage surveys, to further contribute to an understanding of whether the targeting system has achieved its objectives.

Non-beneficiary monitoring is important for understanding errors of exclusion. While the scale of exclusion error can be deduced from household profile monitoring, non-beneficiary monitoring allows determination of whether a particular subgroup of the targeted population has been excluded, and the possible reasons for their exclusion. It also may help identify when the needs assessment and eligibility criteria have excluded a needy group (thereby making the population which should be targeted bigger than planned). Box 18 shows examples of some of the reasons why people are excluded from food for work.

Coverage surveys

This measure is routinely made for supplementary and therapeutic feeding programmes. Assessment of coverage, using the data required in Box 19, is usually done at the same time as an anthropometric survey. More recently, as greater attention has been paid to increasing the coverage of targeted feeding programmes (see section 4), new methods are being devised (Myatt, 2004, see Figure 6 for some results of this method). The main limitations of the method given in Box 19, are that it does not provide any measure of geographical coverage within the implementation areas (with centre based programmes one would not expect uniform coverage). In addition, the degree of reliability of this method is quite low due to sampling limitations.

Sorting distributed rice in Ethiopia

Impact monitoring and evaluation

Monitoring the impact of food aid targeting allows conclusions to be drawn as to whether the objective has been achieved (for an example, see Box 20). Impact monitoring could include periodic anthropometric and mortality surveys, if the purpose of the targeting was to prevent increases in malnutrition and mortality. It should be remembered, however, that factors other than food aid contribute to reductions in levels of malnutrition, including changes in the public health environment. Similarly, assessment of food entitlement can be conducted pre and post food aid targeting to determine whether, for example, the sale of assets or seed was prevented and to determine whether the food deficit, intended to be filled by food aid, was adequately filled.

Benchmarks?

Food basket monitoring can be carried out at distribution points

The paucity of published data on targeting systems means that it is not possible to provide a comprehensive indication of the level of error that can be expected in different types of targeting systems. Table 4 provides some evidence of data from community managed targeting and illustrates the accuracy of the registration, the extent to which those registered receive the food they are entitled to, and the ways in which the recipients use the food aid. It can be seen that an error introduced in the registration, for example, has a knock-on effect and results in errors in the distribution, e.g. people get less food because too many are registered. Sphere standards provide an indication of acceptable levels of coverage for supplementary feeding programmes, stating that coverage should be more than 50% in rural areas, 70% in urban areas and 90% in a camp situation.

*what happened to 3 months supply after 30 days
** maize only