Peer review of WFP evaluation process
Summary of report1
A mother waits for WFP food aid distribution with her child in Malawi
A recently commissioned peer review of the evaluation process and function within the World Food Programme (WFP) has been completed and posted on the WFP website2. The review assessed the central evaluation function, i.e. the Office of Evaluation (OEDE) of WFP, as its starting point, but also included analysis of decentralised evaluation in WFP. The review was based upon preparatory desk study, field visits to WFP regional bureaux and country offices, a meta-evaluation of twelve OEDE evaluations, a web-based survey of the views of WFP staff (87 responses) and peer panel interviews with selected stakeholders.
The panel concluded that the independence of WFP evaluations was adequate in comparison to similar organisations, however the credibility of products was uneven. While the process of evaluations was somewhat more credible, they were also problematic. The criteria of utility of evaluations were partially met with regard to contributing to programming, but structures and mechanisms to promote utility were weak in most other respects. In addition, OEDE is a strong unit with committed, well-trained and highly motivated staff. Over the past seven years, OEDE has invested much effort on improving evaluation. However, evaluation is of more variable quality at the level of Regional Bureaux and Country Offices, with levels of motivation and invested resources dependent upon the interest and priorities of the offices concerned.
Independence
Evaluation resources are currently safeguarded while OEDE is outside of line management whilst, at the same time, sufficiently integrated into WFP leadership structures to facilitate impact. However, accountability for the implementation of recommendations is unclear. Some OEDE staff are concerned that their careers may be affected by their evaluation role, which could lead to inappropriate risk averse behaviour in their management of sensitive evaluations. There are also insufficient safeguards to prevent partiality and conflicts of interest amongst external evaluators. The role of Regional Bureaux in both oversight and advisory support to Country Offices has problematic implications for the independence of their role in decentralised evaluation. Their evaluation function involves public critique of programming while they must also ultimately maintain collegial day-to-day relationships with Country Offices.
Credibility
Although WFP has an evaluation policy, it is a layered series of documents that detracts from clarity and applicability. Evaluation policy is not sufficiently used to guide practice. Evaluators and Regional Bureaux have been unclear regarding what is expected in terms of quality, due to a lack of specification within OEDE itself and concerns that headquarters expectations do not take into account resource and time constraints in the field. On the whole, OEDE evaluation function is impartial with the views of all stakeholders often sought. However, there appears to be an uneven emphasis on stakeholders who are more accessible and articulate, with beneficiary views, in particular, under represented. The process of preparing for evaluations, management, and advising and supporting teams in the field is handled in a highly professional manner by OEDE. Terms of reference are generally of good quality, but they are, at times, too standardised and over ambitious. The quality of evaluations is mixed but should improve with the planned establishment of new quality standards by OEDE. A failure to take into account the cost implications of recommendations, together with factors related to the nature of priority setting in WFP, has damaged credibility of evaluations among some WFP staff.
Utility
Evaluation is insufficiently integrated into many of the processes by which WFP sets, monitors and analyses policies. Evaluation is primarily focused on outputs, as opposed to outcomes and impact, which reflects the demands of many stakeholders. In a narrow sense of contributing to an understanding of how to 'do things right', evaluation makes a notable contribution to programme design and management. In a wider perspective of learning about 'doing the right thing,' performance is not so good. There have been some efforts within evaluations to present evidence that can stimulate greater reflection within WFP - over the changing role of food aid, for example. However, the corporate view of evaluation has tended to focus primarily on its utility for making modest adjustments to existing approaches. New OEDE plans, to tie evaluation closely to logical frameworks, may enhance utility through a focus on outcomes; but lack of prevailing understanding and use of logical frameworks within WFP will make it difficult. At decentralised levels there is a close link to utility, since there is a direct desire to use evaluation to inform and justify new programmes and phases.
Evaluation makes an inadequate contribution to overall knowledge building within WFP and virtually none among partners. Access to reports and findings through website, debriefings, etc, is acceptable, but promotion of the use of evaluation products is not sufficiently proactive.
Recommendations
OEDE should develop an evaluation policy as a transparent vehicle for promoting greater communication among internal and external stakeholders, regarding the aims and intended utility of evaluations.
OEDE should develop an 'accountability map' of key WFP stakeholders, both internal and external, to help in clarifying roles and responsibilities.
OEDE should look for ways of promoting, and providing incentives for staff to adopt more participatory approaches to evaluation. Engagement with partners at country, regional or global levels is primarily a responsibility of other parts of WFP.
After an evaluation has been submitted to the Executive Director, OEDE should not be involved with drafting or compilation of responses from different parts of the organisation. The management response mechanisms should include rules about the timeframe for the response and procedures for follow up, as well as for reporting to the Executive Board. A similar system for management response should be used for decentralised evaluations. Management response and follow up mechanisms should be transparent with relevant documents easily accessible for WFP and partners and routinely posted in electronic form.
The capacity of OEDE staff should be maintained over time to stimulate interest in the evaluation field and encourage professionalism.
The learning element of evaluations should be linked to a larger organisational knowledge management strategy. OEDE should continue recent efforts to systematically harvest lessons from existing evaluations, as well as external fora such as ALNAP3, the Inter-Agency Standing Committee (IASC)4 and relevant partners.
OEDE should develop a more transparent, rigorous and competitive approach to selection of team leaders. All evaluation teams should include at least one evaluation specialist, preferably the team leader.
In order to address concerns that only a small portion of the overall evaluation budget is within the direct control of OEDE, WFP's senior management should devise ways to safeguard the funding allocated to evaluations for the next biennium. The establishment of a centrally managed fund for both OEDE evaluations and decentralised evaluations should be investigated.
1Baker. J et al (2007). Peer review of the evaluation function at WFP. Stockholm, November 5th, 2007. The report is available at http://www.uneval.org/indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=2003 or on the WFP website (see footnote 2)
3Active Learning Network for Accountability and Performance in Humanitarian Action, http://www.alnap.org
Imported from FEX website