Impact evaluation of a cash-transfer programme for Syrian refugees in Lebanon
By Christian Lehmann and Daniel T. R. Masterson
Daniel Masterson is a PhD student in Political Science at Yale University. Daniel worked for UNHCR in Syria in 2007 and 2008. From 2008 to 2011, he worked as the Founding Executive Director of A Plate for All, an NGO running food and nutrition programming for Iraqi refugees in Syria. He holds a Master of Public Policy from Harvard University.
Dr. Lehmann is an Assistant Professor of Economics at the University of Brasilia. His area of expertise is the study of the impact of Conditional Cash Transfer (CCT) programmes, including work with the World Bank in Nigeria and WFP in Swaziland and Mozambique. Currently, Dr. Lehmann forms part of a team of researchers that studies the impact of cash transfers to poor women in Uganda on the local economy.
The authors would like to acknowledge IRC staff who were essential to the successful implementation of this impact evaluation, Information International for their professionalism in conducting the field-based data collection and UNHCR staff who supported the research and shared key information. In addition, the authors and IRC would like to thank their colleagues from other organisations for support and guidance, including DRC, LHIF, Save the Children, World Bank, ACF and UNICEF. The project would not have been possible without the funding from the Department for International Development (DfID) of the UK Government, which supports the IRC Cash and Livelihoods Promotion interventions in Lebanon. Within DFID and ECHO, respectively, IRC is particularly grateful to Simon Little and Maureen Philippon for believing in this project. This project received funding through DFID grant agreement number 204007-111. The research project received exemption from the Yale University Human Subjects Committee under 45 CFR 46.101(b)(2) with IRB Protocol #: 1404013714.
Background
More Syrian refugees reside in Lebanon than in any other country in the region. As no refugee camps have been established in Lebanon, Syrian refugees live in over 1,000 villages and communities across the country and increasingly reside in informal settlements (ISs). The pace of the refugee flow has more than quadrupled since 2012. At the beginning of 2013, there were 130,799 Syrians registered with UNHCR in Lebanon; this has grown to more than 1.1 million registered refugees (September 2014). The magnitude of the crisis can only be understood relative to Lebanon’s population of around 4.5 million people.
The winterisation cash-transfer programme
Starting in November 2013, an inter-agency winterisation1 programme began providing cash transfers to around 60% of all refugees from Syria (including Palestinians), Lebanese returnees, and some vulnerable Lebanese families. This article details the findings of a study2 on the impacts of the winterisation cash transfer programme run by UNHCR and partners, from November 2013 to April 20143. The programme gave $575 USD via ATM cards to 87,700 registered Syrian refugees in Lebanon with the objective of keeping people warm and dry during cold winter months. The programme also provided heating fuel, tools for improving shelters and non-food items (NFI), such as blankets, children’s clothing, and stoves. About 87,700 Syrian refugee families (in Lebanon) received cash intended for the purchase of heating supplies4.
Aid was given at high altitudes to target assistance for those living in the coldest areas during the winter months. Eligibility for the programme was determined by a geographic criterion (refugees residing above 500 metres altitude were eligible, while those living below were not5) as well as demographic criteria6. UNHCR used the demographic data to calculate a ‘vulnerability score.’ Only households with a vulnerability score above a cut-off were eligible for the programme.Each eligible household was notified via SMS that they were eligible to receive an ATM card at a distribution point. The head of household could pick up the card and receive a pin number. Beneficiaries were notified by SMS message when UNHCR and implementing and operational partners transferred cash to the ATM card. Eligible households could withdraw the money at any ATM. Anyone who had the card and pin could withdraw the money. Although UNHCR and the operational partners generally told beneficiaries that the cash assistance was intended for the purchase of heating supplies, there were no restrictions on beneficiary expenditure (though the message varied across operational partners). Therefore, beneficiaries could spend received cash as they wished.
At the same time as the cash transfer programme, WFP was running an e-voucher programme, allowing recipients to buy food at specific stores. Eligibility was based on the same demographic criteria as the winterisation programme, regardless of altitude. All survey respondents (both treatment and control groups) received the e-voucher. The monthly value of the food e-voucher was $30USD per person.
Beneficiary selection
Inter-agency funding could cover transfers for 87,700 households for the winterisation cash transfer programme. To define the altitude criterion, UNHCR used the highest point within each town as the altitude for all households within that town. The vulnerability score was calculated using biometric data available in UNHCR registration records7. The initial assignment to the programme was conducted using the geographic and demographic data for each household in the UNHCR databases, as of November 2013. For the purposes of this survey, a household was defined as a group of people who spend most nights under the same roof and share in financial activities like income and spending. For instance, two ‘households’ may live under the same roof if they operate independently of each other in financial matters.
Research design
This study used a regression discontinuity design (RDD) that allowed quantification of the causal impacts of the cash transfer programme. The research measured the impacts of cash on numerous metrics of household well-being, negative coping strategies, and food and non-food consumption. It tested whether cash produces negative consequences such as local-level inflation or drawing more refugees to regions with assistance (a “pull factor”). And finally, it sought to estimate the multiplier effect of cash aid, i.e. for every dollar of cash assistance, how much would the Lebanese economy benefit.
In order to evaluate the programme’s impact the study compares outcomes of cash beneficiaries residing slightly above 500 metres (i.e. less than 550 metres) (treatment group) to non-beneficiaries residing slightly below (above 450 metres) (control group). The same demographic criteria were used at all altitudes to calculate vulnerability. Therefore the study is comparing households that are similar in their vulnerability scores and only slightly different in altitude. Households did not know beforehand that there would be an altitude eligibility cut-off. This suggests that, around the cut-off point, selection to treatment should not be related to background characteristics in expectation.
According to UNHCR refugee registration records, there were 827 households in the treatment group and 962 households in the control group, i.e. 1789 households in total within the window of analysis (450 metres to 550 metres altitude). This includes only households that had a vulnerability score above the eligibility cut-off.
Because of Lebanon’s topography, the distribution of registered Syrian households living between 450 and 550 metres covers nearly the entire country, running from the north in Akkar to the south in Bint Jbeil. Figure 1 shows the location of all towns where survey respondents lived at the time when the winterisation programme began. In November 2013, when the programme began, survey respondents lived in 15 of Lebanon’s 25 districts (aqdia)8. Due to beneficiaries who moved between the beginning and end of the programme, interviews were conducted in all 25 districts.
UNHCR’s demographic data was used to compare pre-treatment characteristics between the treatment and control groups. Among the demographic variables that were available, 21 of 24 variables were balanced9. Therefore, prior to the start of the programme, households in treatment and control group were very similar. Thus, differences measured after the programme represents causal impacts of cash assistance.
All other aid programmes were equally distributed between the two groups around the altitude cut-off.
The survey was translated and back-translated by separate parties, pre-tested in Halba, Akkar and later pilot-tested in Khirbat Daoud, Akkar and in Al Bourjein, Chouf. Enumerators were Lebanese local to the survey region. The Research Manager conducted three two-day training sessions with groups of enumerators, to enable more direct communication and understanding through smaller training groups.
Data collection
The questionnaire consisted of 226 questions. The primary respondent in each household was the person mainly responsible for how the household spends its money. An interview took about one hour. A town-level stratified random sample of households was asked 81 additional questions on local prices and market characteristics10. A full description of the data collection methodology is available in an online appendix11. The study compared households living within 40 metres of either side of the 500-metres altitude cut-off due to a drop in sample size at smaller bandwidths. For simplicity, results only for one bandwidth are included here. Other technical publications will show robustness to bandwidth specification. The survey was administered in April and May 2014, beginning about five months after the start of the programme and one day after the programme’s final cash transfer. The Research Manager spent more than 20 days in the field and sat in on more than 80 interviews12. Enumerators worked in pairs, with one conversing and reading and the second writing. Enumerators collected the data using anonymous paper-and-pencil interviewing. On average, survey teams conducted five interviews per day. Usually, other people were present in interviews including friends, family, and neighbours.
Key findings
While the use of cash has increased significantly over the past decade13, there is little rigorous evidence of the impact of cash assistance programmes in refugee crisis. The research design is a key contribution to research on Syrian refugees in Lebanon, and more generally to research on the impacts of cash aid in a humanitarian crisis. This is the first study, to the researchers’ knowledge, to rigorously compare refugees receiving cash to those not receiving cash, which makes it possible to quantify the causal impact of the assistance.
Cash assistance produces multiple positive social impacts within the household and in the community. Cash assistance increases school enrolment and reduces child labour. Findings from this research show that households receiving cash assistance were half as likely to send their children to work as households not receiving cash assistance. In the control group, about 33% of children were currently enrolled in school, compared to about 39% in the treatment group. This suggests that cash assistance increases access to education.
Cash assistance caused reductions in negative coping strategies. Households receiving cash assistance less frequently reduced the number of meals per day, reduced the portion of meal sizes and restricted consumption by adults for children to eat. Cash recipients were less likely to withdraw children from school. The programme also increased mutual support between beneficiaries and other community members, and decreased tensions within beneficiary households. The cash programme led to reduced tensions in beneficiary households and between refugees and the host community.
This study found that the cash assistance for beneficiaries at high altitude during winter months increased spending on heating supplies. However, the value of the cash assistance was too low to meet the programme’s objective of allowing all beneficiaries to keep warm constantly throughout the winter and beneficiaries only partially use it for this purpose. Almost half of beneficiaries reported that heating supplies were often not enough to keep warm. This is not because heating supplies were unavailable in the market, but because beneficiaries’ income was so low that they were forced to use the cash assistance to satisfy other basic needs, in particular food. Households spent the majority of cash assistance on food and water despite receiving food vouchers from WFP.
Markets were able to provide sufficient quantities of the goods and services that beneficiaries demanded. The programme did not have a meaningful impact on prices. Across approximately 50 consumer goods, there was no meaningful trend toward higher prices in treatment communities. The programme had significant multiplier effects on the local economy. Each dollar of cash assistance spent by a beneficiary household generated $2.13 USD of GDP for the Lebanese economy14. Also, the research shows that the grants were spent locally, meaning that local Lebanese economies benefit from the cash programme. The vast majority of beneficiaries (more than 80%) preferred cash assistance compared to in-kind assistance (e.g. food parcels).
The study confirmed the absence of a number of hypothetical negative consequences of cash assistance. For instance there was no evidence of beneficiaries spending cash assistance irresponsibly or meaningfully reducing labour supply. The research did not find that cash assistance exacerbated corruption and exploitation. There was no evidence that cash assistance is a pull factor for refugees to settle in communities where cash is distributed.
Indebtedness and asset depletion will likely continue without further assistance. The majority of households surveyed had no savings and were on average $500 USD in debt. Cash assistance helped in a very marginal way to limit further indebtedness. But the amount of cash assistance given to date is modest in comparison to the costs of the minimum expenditure basket and previously incurred debts. Even with the cash assistance, household income remained insufficient to cover refugee’s basic needs.
Limitations
This study provides a number of key findings that are relevant to policy and practice in Lebanon, and beyond. But, there are a few limitations that should also be acknowledged.
First, the results in this study are only representative of refugee households living around 500 metres altitude. Great care has to be taken to extrapolate from the findings to higher altitudes, not to mention other countries and contexts. In higher altitudes, where average temperatures can be several degrees colder, the impact of cash assistance on heating fuel purchases is likely to be stronger because the weather is colder. In lower altitudes, on the other hand, one would likely see even less spending on winter goods.
Second, this research demonstrates benefits of cash assistance and provides evidence against hypothetical negative impacts. The study does not, however, provide evidence of the positive effects of providing cash assistance in place of in-kind assistance. The comparison groups were control households that received food e-vouchers, versus treatment households that received food e-vouchers and also cash assistance.
Third, 85% of respondents were male. The results for intra-household tensions, therefore, need to be interpreted with caution, as women respondents may have answered this question differently. Any downward bias, however, will be present in both the treatment and control groups. The under-reporting will reduce the likelihood of identifying a true effect if it exists. But if one identifies a difference between the groups one could be confident that it is real and that the difference is at least as large as what the study would have found without measurement error.
Fourth, the study estimates the impacts of cash when $575 was delivered per household over the course of five months. The findings suggest, but do not prove, what would happen with a different amount or timeframe. Specifically, the absence of evidence of market distortions from the recent programme suggests that Lebanon’s market is able to adjust for increased demand. This suggests that Lebanon’s economy could adjust to larger amounts of cash aid. The study’s evidence on the multiplier effect suggests that Lebanon’s economy would benefit even more from larger cash transfer amounts and/or broader targeting.
Finally, the research design allows confident statements that the cash transfer caused the difference in outcomes between the treatment and control group and about the scale of the difference. How the cash caused a difference in a particular outcome is a different – and very challenging – research question about causal mediators and causal pathways that could require a research project to study causal pathways for each outcome.
For more information, contact: Daniel Matserson, email: daniel.masterson@yale.edu
1 By ‘winterisation’ the humanitarian community means the process of assisting beneficiaries in staying warm, dry, and healthy during winter months.
2 Winterisation cash. Aiding Syrian refugees in Jordan. From November 2013 to May 2014. IRC. August 2014.
3 UNHCR operated more than half of the cash assistance. Operational partners included: ACTED, AMURT, AVSI, CARE, Caritas, CISP, DRC, Handicap International, Humedica, IOCC, IOM, Makhzoumi, MEDAIR, Mercy Corps, NRC, Oxfam, Save the Children, SHEILD, SIF, Solidar Suisse, and World Vision. Source: UNHCR’s winterisation partner Coordination Map (December 2013).
4 All aspects of the winterisation programme assisted about 96,700 vulnerable families of various targeted groups (Syrians, Palestinian Refugees from Syria, Lebanese returnees and vulnerable hosts). Around 87,700 received cash through ATM cards, checks or Liban post, while around 9,000 received fuel vouchers. In addition, 21,000 households received one-off in-kind winterisation assistance.
5 Altitude was used to target those living in the coldest areas. 500 metres specifically was chosen, instead of 501 or 502, because it is an easy-to-remember multiple of 100. This further emphasises the as-if random nature of the altitude cutoff, which allows us to make inferences about the effect of aid by comparing recipients to non-recipients with similar vulnerability scores.
6 Additionally, families living in “inadequate shelter” were targeted regardless of altitude. The vast majority of informal settlements are located near the sea (low altitudes) or in the east (high altitudes). Since surveyed households were living between 450 and 550 metres, households living in inadequate shelter and informal settlements were rare in the sample. The demographic criteria calculated a vulnerability score based on a weighted sum of the number of: children ages 0-2y, children ages 3-4y, children ages 5-12y, children ages 13-15y, children ages 16-18y, able bodied adult males 18-59y, disabled individuals in household, adults 51-61y, adult dependents 61-70y, adult dependents 71+y, adult females 18-22y, adult males 18-22y, children at risk of not attending school. “Severe vulnerability” status was given to households (HH) that were elderly headed (HH size >= 2 and only one adult >=59y); only one non-dependent adult in household (HH size >= 2 and only one 18-59 year old in household); families with two or more disabled in the family (HH size >=2 and disabled in family >=2); elderly household with one or more disabled adult (HH size >= 2 and disabled >=1 and only 1 adult >=59y); Unaccompanied/separated minor; child-headed household (HH size >=2 and HH members are ages between 0 and 18y).
7 Syrians need to be registered to be eligible for UN winterisation cash assistance, the program studied in this paper. Some NGOs run separate small-scale cash programmes that can include or explicitly target the unregistered.
8 Chouf, El Meten , Aley, Kesrwane, El Nabatieh, Jbeil, Akkar, El Minieh-Dennie, El Koura, El Batroun, Bent Jbeil, Sour, Marjaayoun, Hasbaya, Jezzine.
9 Balance means that differences are not statistically significant (all p-values are above 0.1). The number of residents of every age group was balanced between treatment and control groups. This reveals that households in treatment and control groups had, on average, the same demographic structure prior the start of the programme. The population of the towns where respondents lived was balanced. Education levels for household members above 30 were also balanced. The Research Team subsetted on age because this was measured post-treatment, and we wanted to only consider people old enough that receiving cash would not have an effect on their education levels. The number of disabled and non-disabled individuals in households within age groups was also balanced, except for three categories. We found imbalance in “Males not disabled aged 51 – 59”, “(both genders) Disabled aged 13 – 15”, and “(both genders) Disabled aged 16 – 17”. The differences are small in absolute terms.
10 The town-level stratified sample selected up to four respondents in each village. If a village had four or fewer respondents from our full sample, then all households in that village were interviewed.
11 Online appendix material at http://www.danieltrmasterson.com/research
12 1861 individuals met the selection criteria in November 2013 at the beginning of the program. By the time the researchers sought respondents for the survey in April 2014, 62 households were no longer present in UNHCR data. So when surveying began, there were 1789 households in the sample. Given 1361 complete interviews, the study had attrition rates of 24-26 percent depending on which calculation you use.
13 Cash disbursements in humanitarian emergencies tripled between 2008 and 2012, peaking at $262 million in 2010. Global Humanitarian Assistance Report 2013 http://www.globalhumanitarianassistance.org/wp-content/uploads/2013/07/GHA-Report-2013.pdf
14 Higher-round effects can be calculated by the formula: dY=C/(1-MPC). That is, the total amount of additional Gross Domestic Product (dY) generated by one beneficiary household is calculated by dividing the amount of winterization cash that the beneficiary household spends (C) by one minus the marginal propensity to consume (MPC). The total amount of additional Gross Domestic Product (GDP) that each beneficiary households generates for the Lebanese economy is then given by dY=575/(1-0.53)=1223.40 USD. The multiplier is M=1/(1-0.53)=2.13. See the full report for more details.