Impact evaluation of cash, food vouchers, and food transfers among Colombian refugees and poor Ecuadorians in urban Ecuador

Summary of evaluation¹

Evaluation headlines: Levels of food insecurity and associated anaemia are high amongst Columbian refugees and poor Ecuadorians in the northern provinces of Ecuador. Following an assessment in 2010, WFP established food, food voucher and cash transfer programmes through a number of implementing NGOs in two adjacent provinces in the north of the country. An evaluation by WFP and IFPR was undertaken to compare the impact of the three interventions. Key findings were that cash allows for savings that helps households smooth their food and non-food consumption. However, if the objective is to increase calories or dietary diversity, vouchers are the most cost-effective means of doing so, followed by cash although food vouchers is the modality least preferred by beneficiaries.

Despite substantial research into the impact of food assistance and the impact of conditional cash transfers (CCTs), in many context there is almost no evidence from a rigorous evaluation directly comparing the impact and cost-effectiveness of cash transfers and food transfers in the same setting. A recently published evaluation in Ecuador is one of several impact evaluations being undertaken in different countries by the World Food Programme (WFP) and the International Food Policy Research Institute (IFPRI) in which various forms of transfers will be compared to learn which modalities are most effective in different contexts. The other countries are Yemen, Niger and Uganda.

Ecuador has the largest refugee population in Latin America and the vast majority of refugees are Colombians. In 2010 there were approximately 121,000 refugees and 50,000 asylum seekers living in Ecuador. As a result of constitutional changes on immigration and refugee issues in 2008, as well as the Enhanced Registration Programme in 2009, Ecuador has provided documentation to thousands of Colombian refugees in need of international protection. However, despite programmatic action and international support, refugees remain excluded from many government services and programmes, as well as discriminated against in the job market.

In order to better understand the refugee situation in Ecuador and how to respond to it, the WFP in collaboration with UNHCR conducted a study on the food security and nutrition situation of this population. They found that 27.9% of the Colombian refugee population is food insecure and suffers from low dietary diversity. The incidence of food insecurity is higher in northern border provinces, reaching 55% in Sucumbíos and 44% in Carchi and Imbabura. In addition, 48% of children are anaemic.

Responding to the study’s findings and to a request from the Government of Ecuador (GoE) in April 2011, WFP expanded its assistance to address the food security needs of refugees and support their integration into Ecuadorian society. The new programme, partially funded by the Spanish Government, used cash, food vouchers, and food transfers to support the most vulnerable refugees as well as poor Ecuadorians in urban centres of Carchi and Sucumbíos. Both Carchi and Sucumbíos are northern border provinces that receive high influxes of Colombian refugees.

Method

The evaluation study set out to estimate the relative impact and cost-effectiveness of cash, food vouchers and food transfers on household food security indicators, as well as complementary indicators such as household expenditure and anaemia. It also investigated the gender effects of the programme and the impacts on reducing tensions and discrimination between Colombian refugees and the host Ecuadorian population. In addition to the quantitative impact evaluation, a qualitative study was conducted in order to learn more about the efficacy of nutrition trainings that accompanied the distribution of the different transfer types.

Despite sharing administrative borders, the two intervention provinces of Carchi and Sucumbíos have markedly different geographic, agro-ecological, and economic characteristics. Carchi is located in the northern highlands at high altitude, characterized by an industrial and agricultural-based economy including production of tobacco, dairy, floriculture, and staple crops such as potatoes and maize. Sucumbíos is located in the Amazonian lowlands and its economy is driven by natural resource extraction, primarily oil, making it one of the most important economic areas in Ecuador.

Carchi and Sucumbíos were selected by WFP for the transfer programme because of the high concentration of refugees and poor Ecuadorians as identified by the 2010 Emergency Food Security Assessment (EFSA). In addition, both provinces have the presence of implementing partner non-governmental organizations (NGOs) for food transfers, financial institutions including ATM branches, supermarkets and functioning central markets. The urban centres chosen for the study were Tulcán and San Gabriel in Carchi, and Lago Agrio and Shushufindi in Sucumbíos. Tulcán and Lago Agrio are the capital cities of Carchi and Sucumbíos, respectively, and represent the largest urban areas within the provinces. Both cities are in close proximity to the Colombian border (for example, Tulcán is approximately seven kilometres from the border). In addition to these four primary urban areas, three additional smaller urban areas were included in the programme target population: General Farfan, Huaca, and Pacayacu. These additional areas are suburbs close to the four primary urban areas and share the same transfer distribution centres, financial institutions, and supermarkets.

Barrios (or ‘neighbourhoods’) within these urban centres were then pre-chosen by WFP in collaboration with local partners as areas that have large numbers of Colombian refugees and relatively high levels of poverty. In November and December of 2010, the Centro de Estudios de Población y Desarrollo Social (CEPAR) implemented a household census in these barrios. Every household in the selected barrios was visited, mapped, and administered a one-page questionnaire that consisted of basic demographic and socioeconomic questions. A proxy means test was then developed to target beneficiaries based on indicators of household demographics, nationality, labour force participation, food security, and asset ownership. Based on point scores by nationality, the decision was made to automatically include all Colombian and Colombian-Ecuadorian households as "qualifying." In addition, since WFP wanted to reach households that were not already being covered by GoE grants, the decision was made to exclude all households who reported receiving the government’s social safety net transfer programme, the Bono de Desarrollo Humano (BDH). Households residing in the selected barrios with low socioeconomic status that met the criteria described above formed the original sample eligible for the intervention.

Intervention

The intervention consisted of six monthly transfers of food, food vouchers, or cash to Colombian refugees and poor Ecuadorian households in selected urban centres. The transfers were distributed in coordination with local partners and the assistance of the president of each barrio. An enrolment meeting was held in March before the first transfer was distributed to issue photo ID cards, as well as to sensitize beneficiaries to the programme objectives and logistics. The coordination of the intervention was managed by the WFP Country Office (CO) through regional sub-offices in each province and was backstopped by staff at headquarters in Quito. In the initial enrolment and sensitisation, the programme was described as a poverty and food security transfer targeted toward women and therefore, the majority of the entitlement (programme identification) cardholders were expected to be women. However, based on household demographics, men could also be entitlement holders and participate in all programme activities. Overall, approximately 79% of cardholders in Carchi and 73% of cardholders in Sucumbíos were women.

The value of the monthly transfer was standardised across all treatment arms. The total value of the cash transfer was $40 per month per household. This amount was transferred onto pre-programmed ATM cards for all cash transfer beneficiaries. Cash transfer households could retrieve their cash at any time (i.e. they could keep cash in the bank for longer than one month), however, it had to be taken out in bundles of $10. The food vouchers were also valued at $40 and were given in denominations of $20, redeemable for a list of nutritionally-approved foods at central supermarkets in each urban centre. The list of approved foods was composed of cereals, tubers, fruits, vegetables, legumes, meats, fish, milk products, and eggs. The food vouchers could be used over a series of two visits per month and had to be redeemed within 30 days of initial receipt of the voucher. The vouchers were serialised, printed centrally and were non-transferable. The food basket was valued according to regional market prices at $40 and included rice (24 kg), vegetable oil (4 litres), lentils (8 kg), and canned sardines (8 cans of 0.425 kg). The food basket was stored and distributed by local partners. The transfer size for all modalities was set to be roughly comparable to the national cash transfer scheme, the BDH, which, at the time of programme design, was $35 per household.

Nutrition sensitisation was a key component of the programme, aimed at influencing behaviour change and increasing knowledge of recipient households, especially in regards to dietary diversity. To ensure a consistent approach to knowledge transfer, a set of curricula was developed by WFP to be covered at each monthly distribution and transfers were conditional on attendance at the nutrition trainings. These topics included (1) programme sensitisation and information, (2) family nutrition, (3) food and nutrition for pregnant and lactating women, (4) nutrition for children aged 0-<12 months, and (5) nutrition for children aged 12-24 months. Nutritional recipes were also distributed throughout the six months. During the last monthly meeting, an overview and review of lessons learned was implemented, including nutritional bingo in which participants were asked questions about previous training sessions in game format. In addition to monthly meetings, posters and flyers were developed and posted at distribution sites, including supermarkets, banks, and community centres, to further expose participants to knowledge messaging. These topics included, among others, recommended consumption of food groups, daily nutritional requirements, proper sanitation and food preparation.

The strategy for estimating the impacts of the cash, food voucher, and food transfer was built into the design of the programme. Sample clusters were randomly assigned to one of four treatment arms: the cash transfer group, the food voucher transfer group, the food transfer group, and the comparison group (which received no transfers). Due to the distinct socioeconomic and geographic characteristics of Sucumbíos and Carchi, the randomisation of cluster centres was stratified at the province level.

Randomisation was conducted in two stages: first, barrios were randomised to either treatment or comparison arms and second, all treatment clusters (geographical units within barrios) were randomised to cash, food voucher, or food transfer. Before conducting the randomisation, consultations were undertaken with WFP and CEPAR to both ensure that there was sufficient number of qualifying households in each cluster and there was sufficient geographical distinction between clusters. This process led to 63 barrios and 128 clusters in the four urban areas over which to randomize. The number of clusters per barrio varied from one to six, with an average of approximately two per barrio. The barrios and clusters were randomised into the four treatment arms using percentages of 20/20 for comparison and food, and 30/30 for cash and food voucher. These percentages were established in consultation with WFP to meet both beneficiary target sizes by transfer type, as well as sample size requirements for the evaluation design.

Evaluation

In order to conduct an evaluation of the cash, voucher, and food transfer programme, baseline and follow-up surveys were conducted. The baseline survey was conducted in March-April 2011 before the first transfers were distributed. The follow-up survey was conducted approximately six months later (October-November 2011) after the last of the six distributions. The sampling for the baseline survey was conducted by CEPAR and IFPRI after receiving the beneficiary lists. Based on the distribution of clusters in the treatment arms and the required sample sizes, 27 households per comparison and food clusters and 20 households per food voucher and cash clusters were randomly selected to be interviewed in the baseline survey. In addition, since one of the main objectives of the evaluation was to compare differences across nationalities, the Colombian and Colombian-Ecuadorian households were oversampled to ensure a sufficient sample for comparative analysis. In total, 3,331 households were surveyed at baseline. However, approximately 30% were too rich and subsequently excluded from the programme, and so the evaluation focuses only on 2,357 households that were included after the retargeting period and could be matched with monitoring data. Of these households, 2,122 were re-surveyed at follow-up.

Results

Food security

The cash transfer incurs the lowest costs to participants in terms of waiting times and transportation costs. A higher percentage of participants prefer to receive all the transfer in cash as opposed to all in food and vouchers. Across all three modalities, the transfers are reported to be mainly used for consumption of food items; however, voucher recipients in comparison to cash recipients spend a larger percentage on food. Almost none of the food transfer or voucher is sold to buy other items. Besides food consumption, food recipients tend to share their transfer with friends or family, or save their transfer for later use. Cash recipients also report saving a small share of their cash for later use and spending a small portion on non-food items. Nutrition knowledge increases from baseline to follow-up with the largest knowledge gains occurring on food items that are rich sources of iron and vitamin A, and on feeding practices for infants.

Overall, programme participation leads to large and significant increases across a range of food security measures, with the value of per capita food consumption increasing by 13%, per capita caloric intake increasing by 10%, Household Dietary Diversity Score (HDDS) improving by 5.1%, Dietary Diversity Index (DDI) by 14.4%, and Food Consumption Score (FCS) by 12.6%. All three modalities significantly increase the value of food consumption, caloric intake, and dietary diversity as measured by the HDDS, DDI and the FCS. However, the increase in dietary diversity measures is significantly larger for the voucher modality and the increase in caloric intake is significantly larger for the food modality. The food modality leads to significant increases in the number of days a household consumes five out of 12 food groups: cereals, roots and tubers, meat and chicken, fish and seafood, and pulses, legumes, and nuts. The cash transfer leads to significant increases in seven food groups: roots and tubers, vegetables, meat and poultry, eggs, fish and seafood, pulses, legumes, and nuts, and milk and dairy. Finally, the voucher leads to significant increases in nine food groups: cereals, roots and tubers, vegetables, fruits, meat and poultry, eggs, fish and seafood, pulses, legumes, and nuts, and milk and dairy. The impact of vouchers on the frequency of consumption is significantly different to that of food transfers for vegetables, eggs, and milk and dairy.

Colombians in the food and cash group experience significantly greater gains in dietary diversity measures than Ecuadorians. With the exception of small increases in expenditure on toys, the combined transfer programme does not lead to increases in non-food expenditures.

Anaemia

Taken together, the results for anaemia can be summarised in the following way. First, although levels of anaemia are high, particularly for the younger age group, the intervention did not have a positive impact on anaemia reduction for either children aged 6-59 months or for adolescent girls aged 10-16 years. This result can be explained by the relatively short intervention period. In addition, the focus of the intervention was on food security and nutrition in general and not focused specifically on anaemia reduction. Based on the results, it is possible (and is corroborated by household food group caloric results) that for young children, consumption moved away from more diverse diets and towards consumption of staples that were part of the food transfer for the treatment group.

Household decision-making and behaviours

Overall, transfers lead to a significant decrease in intimate partner violence, however, there is no impact on decision-making indicators. The food treatment arm leads to a significant impact on experience of disagreements regarding child health. Otherwise, there are no significant impacts by treatment arm for women’s decision-making. While all three treatment arms lead to significant decreases in physical/sexual violence, only cash and food lead to significant decreases in controlling behaviours. However, there are no significant differences across modalities in the size of the impact for any of the intimate partner violence indicators. There are no significant differential impacts with respect to being Colombian on any of the household decision-making indicators or intimate partner violence indicators.

The food modality is the most costly and the cash and voucher less costly. In terms of cost-effectiveness across food security outcomes, vouchers are the least costly means to improve these outcomes while food is the most costly means to improve these outcomes.

Although vouchers have the largest impact in terms of food security, voucher participants report having the most difficulties with the programme, with 79% of voucher beneficiaries reporting at least one complaint with their transfer, compared to 40% in the cash group and 37% in the food group. Common difficulties experienced by the voucher recipient households include high food prices and lack of food in supermarkets, as well as problems at cashiers and understanding rules or use of vouchers. In general, cash is the preferred modality among participants. In terms of opportunity costs and transportation costs, cash is the least costly, which is consistent with participants’ revealed preferences. Despite modality-specific complaints, overall beneficiaries report overwhelming satisfaction with the programme, including nutrition trainings and interactions with programme staff.

The most costly modality to implement from the institutional perspective for WFP was the food transfer, while cash and vouchers had similar costs. In particular, as measured on a per transfer basis using modality-specific costs, it cost $3.03 to transfer $40 in cash to a beneficiary, $3.30 to provide them with a $40 voucher, and $11.50 to transfer $40 worth of food. The difference in cost between the food ration and the other modalities was primarily due to added storage, distribution, and contracting. In contrast to the other transfers, food rations require a degree of manipulation, in that they must be packaged (thus requiring labour), transported (human resource and transport cost), and stored in warehouses (rental and upkeep costs). Combining impacts with costs to compare the cost-effectiveness across modalities for food security measures, we find that food vouchers are the most cost-effective means for improving food security and food is the least cost-effective means of improving these outcomes.

Beneficiary opinions of the nutrition trainings are positive, and there seems to be some evidence of behaviour change, particularly in the testing and use of recipes as well as food purchase patterns. However, it is not clear whether this effect will persist without the added benefit of the transfer. In addition, the inclusion or involvement of spouses in the nutrition training and behaviour change process may lead to more favourable overall outcomes.

It is important to emphasize that the results from this evaluation represent a nutrition knowledge and transfer ‘bundled’ intervention and are specific to the population studied: poor urban households in Northern Ecuador. Although large overall programme impacts are found, these findings may not hold in rural areas where supermarkets are more likely to have less variety of foods and may also lack the capacity to properly implement such a programme. While these results cannot be generalised to rural areas, they do provide insight for a large, vulnerable segment of the population, the urban poor, and demonstrate how transfer programmes have a short-term positive impact on food security and social relations within households and communities. In the context considered here, choosing the ‘winner’ among the different modalities depends on the objectives of the policymakers. If the objective of these transfers is simply to improve welfare, cash is preferable. Cash is the modality that beneficiaries are most satisfied with, and it is the cheapest means of making transfers. Given the budget available to WFP for this project, shifting from food to cash could have increased beneficiary numbers by 12%.

Moreover, cash allows for savings that helps households smooth their food and non-food consumption. If the objective is to increase calories or dietary diversity, vouchers are the most cost-effective means of doing so, followed by cash. Although, vouchers are the most cost-effective means of increasing caloric availability and diet quality, it is the modality least preferred by beneficiaries. Thus policymakers are faced with the trade-off of improving overall welfare or improving specific outcomes. The former gives aid recipients’ autonomy, while the latter restricts their choices in order to achieve specific objectives.

¹WFP/IFPRI (2012). Impact Evaluation of Cash, Food Vouchers, and Food Transfers among Colombian Refugees and Poor Ecuadorians in Carchi and Sucumbíos. Final Report. International Food Policy Research Institute Poverty, Health, and Nutrition Division October 17th, 2012