Aid effectiveness and Vulnerability Assessment Framework: determining vulnerability among Syrian refugees in Jordan

By Hisham Khogali, Lynette Larsen, Kate Washington and Yara Romariz Maasri

Hisham Khogali is an independent consultant with 19 years of experience in a range of humanitarian contexts. Hisham has an MSc. in Human Nutrition from LSHTM and has worked for NGO’s, the Red Cross Movement and various UN agencies. In Jordan, he worked for the Assessment Capacities Project (ACAPS).

Lynnette Larsen is a humanitarian information management specialist who has managed Humanitarian Information Centres in Kosovo, Iraq and Liberia and similar IM efforts in the 2005 Pakistan earthquake, 2008 Cyclone Nargis (Burma) and 2010 Haiti earthquake responses. She co-led implementation of the OCHA’s IM capacity building strategy and developed an IM strategy within the Cluster approach.   

Kate Washington is the lead facilitator for the Vulnerability Assessment Framework development with UNHCR. Previously, she worked for 5 years with CARE International in Jordan and has over 13 years of regional experience across Syria, Lebanon and Jordan.  She holds a Master of Sciences degree in Development Studies from SOAS, University of London.

Yara Romariz Maasri is an Associate Coordination Officer with UNHCR Jordan and has been supporting the Vulnerability Assessment Framework since November 2013. She previously worked in refugee resettlement for over four years in Lebanon, Kenya and Cameroon, and was co-editor of the Fahamu Refugee Legal Aid Newsletter for three years. She holds a Master of Science in Forced Migration from the Refugee Studies Centre, University of Oxford. 

The current VAF development team consists of Kate Washington, Harry Brown, Marco Santacroce and Carolyn Davis.  The work of the VAF team has been overseen and supported by Alex Tyler and Volker Schimmel of UNHCR. WFP and UNICEF have also provided essential support throughout the development process.

The conflict in Syria, which began in 2011, has continued to create a worsening refugee situation. There is currently a growing population of 3,033,972¹ registered refugees in surrounding countries and the region.Realising the challenges that this number of refugees posed combined with the need to be more effective, UNHCR’s Field Information Support Servicessection launched a project entitled ‘Design and implementation of the framework for humanitarian aid effectiveness. ’The main objective of this UNHCR initiative is to improve aid effectiveness, by ensuring a needs-based and principled approach to humanitarian response. In order to achieve this objective, UNHCR and its partners needed to work together, at country level, to agree on and put in place mechanisms for:

1. Definition of vulnerable groups/households in need of assistance, and agreement on minimum sectoral data to inform this definition with partners 
2. Identification of vulnerable households
3. Development of shared tools (database and data entry form) for the tracking of assistance provided by UNHCR and partners, agreement on data consolidation, and protection, data ownership, sharing/access agreement with partners

The process was driven by two factors. First, an interest in providing the right support to vulnerable people; for example, was it enough to provide the same support to all disabled people when their vulnerabilities may actually have required some differentiation in the type of support they received? Second, it was highly likely that there would be a reduction in resources available as the crisis continued and other crises around the world emerged; a better targeting mechanism would be needed to determine eligibility for limited aid. 

The project approach

In order to test the concept of a vulnerability analysis framework, piloting was undertaken in Jordan in both the refugee camp context (specifically Zaatari camp) and with refugees in urban areas of the country. The pilot focused on health and cash assistance. The project was planned in three phases. These were:

Phase 1: Scoping and coordination:

Identification and engagement of key stakeholders, review of existing vulnerability assessment methods, set up and meeting of steering group.

Phase 2 Facilitation and design:

Support to sector leads to develop a vulnerability assessment strategy, database and data entry tool design

Phase 3 Lessons learned and recommendations:

Document a lessons learned exercise UNHCR approached ACAPS² to support the projectas a non-operational entity in the region, i.e. without any assistance programming. Furthermore, ACAPS have specific assessment expertise and experience and had enjoyed previous successful partnerships with UNHCR. Support by ACAPS included the deployment of an assessment expert to work in country with UNHCR and all relevant partners in a collaborative manner to define vulnerability based on emergency life-saving needs, and specifically to define eligibility criteria for refugees receiving assistance, in the health and cash assistance sectors (as a starting point).

Phase 1: Scoping and coordination

This phase took place in the second half of 2013. UNHCR has an established methodology for assessing vulnerability. This method, using the Specific Needs codes³, was applied in the Syrian refugee context as in other contexts where UNHCR works.  This ‘group’ approach has a number of key weaknesses including:

• Generalisations about vulnerable groups tend to exclude those that are generally not thought of as vulnerable, e.g. at a workshop in Zaatari camp, the issue of men being vulnerable particularly to violence, but also their potential to commit violent acts due to unemployment, was raised. In addition, in the context of Jordan, adolescent girls are particularly vulnerable (e.g. to becoming child brides). Yet neither unemployed men nor adolescent girls are included in the specific needs codes of UNHCR. 
• Generalisations about vulnerable groups also fail to recognise that not everyone in a vulnerable group is equally vulnerable (UNHCR addresses this through exclusion criteriain the cash programme).
• A group approach is one dimensional and cannot capture the fact that a household or individual can be in more than one disadvantaged group at a time, i.e. potentially having greater vulnerability.
• A group approach also does not explain why someone is disadvantaged; an elderly person is not vulnerable because they are old, but perhaps because they are isolated or lack resources to maintain themselves.
• The approach also does not take account of  the temporal and spatial aspects of vulnerability; people can move in and out of vulnerability, e.g. a Syrian refugee who gains employment becomes less vulnerable, or refugees with proximity to services may be less vulnerable than those further away.

Initial reviews of secondary data and meetings with partners of UNHCR, both in health and cash assistance, revealed that different systems were being used for identifying vulnerability. This was particularly true in the case of cash assistance where a number of partners had adopted a scorecard approach.However, different scorecards were used by different partners. The score card approach provided a more transparent approach to determining vulnerability and enabled a multi-dimensional approach that incorporated both vulnerable groups and potential coping strategies/vulnerabilities. However, the scoring of cards created divisions amongst partners for whom there was no standard scoring mechanism⁴. This difficulty in agreeing scoring may have been exacerbated by organisational mandates⁵. It is also important to recognise that different organisations had different objectives for their cash assistance programmes, with some adopting a one off emergency assistance approach and others (e.g. UNHCR) adopting a three month (renewable) cycle approach. Cash assistance may be conditional or unconditional.Scorecards also rarely took into account access to services as a key vulnerability determinant.

In order to ensure a common understanding of vulnerability, the following three characteristics of vulnerability were proposed and agreed in initial meetings/workshops with partners. Namely that vulnerability is:

• multi-dimensional and differential (varies across physical space and among and within social groups)
• scale dependent (with regard to time, space and units of analysis such as individual, household, region, system)
• dynamic (the characteristics and driving forces of vulnerability change over time).

These principles underpinned Phase 2: the facilitation and design of a vulnerability analysis system. 

Phase 2: Vulnerability Analysis Framework: developing an inter-agency approach

After a two-month hiatus during the development of the annual Refugee Response Plan (RRP 6), work on the Vulnerability Analysis Framework project resumed in mid-November of 2013.  Meetings with UNICEF and WFP led to a decision to broaden the scope of the project beyond the Cash Assistance working group and invite the participation of a wider range of stakeholders in its development.   Following informal presentations of a proposed approach to groups of United Nations (UN) agencies, international non-governmental organisations (INGOs) and donors, an inter-agency steering committee, (consisting of five UN agencies, five INGOs and two donors) was established to guide further development of an assessment methodology and implementation⁶. 

Throughout discussions with potential new partners, the key objective of the project remained the development of a standardised approach to assessing household vulnerability to support equitable programmatic decisions.  A standard list of approximately 10-15 household indicators of vulnerability, developed by the humanitarian community through existing sector and inter-sector coordination mechanisms, would be used by UNHCR and other agencies in determining eligibility for assistance.  However, it was stressed that while a household vulnerability ‘score’ could be used as a factor in decision-making, it should not be the sole criterion used in decision-making.  Furthermore, the final decision on allocating any assistance would always rest with the individual agency responsible for managing the particular intervention. Because vulnerability is not a static concept, the frequency with which a re-assessment would be carried out was identified as one of the critical considerations in operationalising the exercise.  Additional risk analysis would be carried out throughout the project development process to identify any potential harmful impacts of the assessment on households, for example, the potential impact on marginal populations who fail to qualify as the least vulnerable but for whom assistance may be a critical factor in preventing a deterioration in circumstances over time. 

Generating a relative household vulnerability ‘score’ would be done through a mix of sector-based and cross-cutting quantitative and qualitative indicators.  Linking to location data for each household would enable agencies to analyse the data by vulnerability level, programming sector and geographic areas. It was proposed to start with data collection through the UNHCR registration and re-registration process and home visit data, with an eventual expansion of the collection process to, for example, selected NGOs, in order to speed up the development of a significant body of information. The project would also focus initially on refugee households outside the established refugee camps because of both the larger size of the target population and high levels of assistance still being provided in the camp setting.  The data would be centrally stored and made available to participating agencies subject to normal concerns for privacy and sensitivity of data.  Linking the data to UNHCR’s Refugee Assistance Information System (RAIS) would eventually enable analysis of humanitarian assistance effectiveness in reducing or preventing vulnerability by establishing a household record of assistance to be matched against a vulnerability profile. 

Developing ‘indicators’ of vulnerability 

To increase awareness of the project and initiate the development of vulnerability indicators, a day-long workshop in February 2014 brought together representatives of sectors and sub-sectors to draft list of approximately 25 indicators from which a final (shorter) list would be culled. Although the seriousness with which all the participants approached the work was impressive, the groups achieved varying degrees of success in fleshing out indicators. Some groups were able to reach a more detailed articulation while others struggled to move beyond the discussion phase.  Although most participants felt the task was a challenging one, comments from several participants indicated appreciation for the consultative approach chosen.  The exercise, in addition to providing an important first step in the development of indicators, also helped to solidify the image of the project as an inter-agency initiative and one that would provide useful tools and information for a broad range of humanitarian actors. 

Definition of vulnerability and framework outcomes

The February workshop also established a working definition of vulnerability for the Syrian Refugee Crisis in Urban areas of Jordan: “the risk of exposure of Syrian refugee households to harm, primarily in relation to protection threats, inability to meet basic needs, limited access to basic services,  food insecurity, and the ability of the population to cope with the consequences of this harm”. Using this as a basis for defining the scope of the Vulnerability Assessment Framework (VAF) that is being developed, the VAF inter-agency steering committee is overseeing the VAF development process which will have the following outcomes:

• Data against VAF indicators are collected at the registration stage by UNHCR and during home visits by UN agencies and NGOs, and are uploaded into a central database. 
• With the data regularly updated, the database will generate a ‘vulnerability profile’ for each refugee household, based on thresholds of ‘extremely vulnerable, very vulnerable’, etc. 
• Partners are able to access the database and conduct queries, while ensuring that confidentiality and protection rules are respected, e.g. query: percentage or number of extremely vulnerable refugee households in Irbid governorate, or a district of Irbid.
• Partners will be able to conduct sector-specific queries, to help them better target their assistance by geographical area and household level, prompting further technical assessments.
• Partners who have identified beneficiaries for individual household assistance are able to check the ‘vulnerability profile’ of that household against the database, by uploading a list of unique identifiers (e.g. UNHCR or Ministry of Interior registration number). They may then be able to modify their decision of whom to assist, based on the vulnerability profile. 
• The VAF should reduce duplication of assistance. Partners are encouraged to log assistance they have provided to a refugee household in the database. If partners are systematic in this entry, other partners can then see which households have already been assisted in the database, when searching for the unique identifiers.
• Through periodic reports, the humanitarian community will be able to monitor trends in vulnerability by geographical area, informing broader strategic processes, such as the Regional Response Plan. 

Ultimately, VAF data will provide a comprehensive picture of vulnerability among refugees that may be used for advocacy purposes and for planning and prioritising of aid interventions. 

Progress to date

The VAF process is multifaceted and a number of key components have been developed, piloted and rolled out, these include:

• In March 2014, an inter-agency participatory assessment was conducted with Syrian refugees, through 70 focus groups, with responses disaggregated by age, gender and disability. The VAF indicators were included in the discussions of refugee priorities/key concerns, and perceptions of their own or their community’s vulnerabilities. 
• An assessment tool was designed using the VAF indicators identified in the February workshop.
• A World Bank team has conducted a detailed analysis of indicators used by UNHCR for Cash Assistance decisions, using proGres⁷ and Home Visit data. From a welfare perspective, this provides an objective validation of many of the VAF indicators. 
• Standard Operating Procedures (SOPs) on how the tool could be applied have been drafted by the VAF team.
• A Communication strategy for both partners and beneficiaries has been developed, and a Communications Specialist was brought on board to implement the initial phase.
• The VAF data collection tool was piloted and rolled out in June with over 15,000 households having been interviewed to date.

Throughout July-September, building on the work of the World Bank, econometric analysis of the VAF data was conducted and a VAF Welfare model that identifies the characteristics of vulnerable householdswas developed. This model uses predicted expenditure as a proxy for welfare and provides a mapping of the vulnerability spread across those households that have been interviewed. Data collection is ongoing with UNHCR, through implementing partner International Relief and Development (IRD), interviewing approximately 5000 new households a month.

In August an inter-agency appeals mechanism workshop was held and an appeal mechanism and interface designed. This appeals mechanism is now being piloted in cooperation with the WFP. Refugees can appeal for re-instatement in the WFP food assistance programme, following cuts made to the beneficiary list based on criteria developed as a result of the WFP Comprehensive Food Security Monitoring Exercise conducted in December 2013/January 2014. In October, an appeals database in RAIS was developed by UNHCR and launched in beta version⁸ to assist in the process. 

A user interface module in the RAIS is being developed by UNHCR, to allow updating of vulnerability scoring at the household level, access to interested partners to inform assistance decisions, and from which vulnerability trends analysis can be extracted. 

The VAF team also facilitated an additional participatory interagency inter-sectoral workshop in October 2014 to elaborate Sector Based Vulnerability Assessment Rules that will complement the Welfare/Vulnerability Assessment model.  The workshop built on work conducted in Lebanon to define sector level vulnerability decision trees.  Each sector was tasked with looking at the multiple data points available from the VAF questionnaire and UNHCR home visit form in order to identify and then articulate sector specific indices of vulnerability and develop weights for each, which allow a sector level calculation of vulnerability. This sector level scoring is still under review but will allow for a more nuanced picture of household vulnerability.  For example, VAF partners will be able to access information that tells them a household's overall vulnerability score but also a breakdown of relative vulnerabilities by sector. This should allow for programmatic decisions to be made on the most appropriate types of intervention and acknowledges the holistic and interlinked nature of vulnerability.

Risks and safeguards

Given the impact that the household vulnerability score could potentially have on the assistance received by a household, it is important that the nature and limitations of the data are clearly understood by all actors and that safeguards are included in the framework to minimise the risk that data are misused. Discussions with the Protection Unit in UNHCR have also taken place throughout the process. 

The assessment process needs to be carefully considered to minimise exclusion risk, i.e. the risk that households or segments of the refugee population are excluded from the process or their level of vulnerability is incorrectly categorised and they are excluded from receiving assistance. One example of a mitigating action which has been developed and is being piloted (see above) is an appeals process by which households can contest any changes in the provision of assistance based on VAF vulnerability scores.  This will continue to be articulated at a sector level as the VAF is rolled out.

Additionally, there is a risk that refugees those without support, may eventually become vulnerable so that there is a need for periodic re-assessment or other means by which to identify changing household circumstances. 

As stated above, the VAF process minimizes risk of exclusion for refugees through 

1) appeals process, or fast-tracked reassessment for border line cases 

2) periodic update of vulnerability status

3) quality assurance of data collectors and database.

It is important to highlight that the VAF will not replace the need for sector-specific detailed needs assessments, but will assist in streamlining planning of such assessments and/or programmatic interventions by, for example, identifying geographical areas where a large number of cases with a sector-specific ‘flag’ are located. 

VAF validation plan and roll out

Finally, the VAF steering committee is now articulating a validation plan that will review and validate the different components of the Welfare/Vulnerability model and the Sector Level rules before the VAF is fully rolled out to partner organisations. The validation plan will use a participatory and inter-agency/inter-sector approach.  Further consultations with refugees to review vulnerability indicators and indices will be conducted.  Additionally, multifunction teams will conduct 'blind' visits to a randomised selection of households (across the vulnerability thresholds) that have undergone VAF interviews and scoring to assess the accuracy of the models and rules.  On the basis of the results of these validation activities, the steering committee and a peer review committee of other vulnerability specialists from the region will sign off on the VAF models and the full set of VAF tools will be made available to partners.

Currently, the VAF aims to be fully operational and launched in January 2015.  During an initial six month period there will be a VAF oversight committee who will monitor the use of VAF tools and VAF data by partners.  By June 2015, Phase 3 of the process will be conducted with a full review of the process to date, revision of the models or rules as necessary and the documentation of lessons learnt and recommendations.

For more information, contact: Kate Washington, email: washingk@unhcr.org

¹ As of 17 November 2014, http://data.unhcr.org/syrianrefugees/regional.php?

²  ACAPS is the Assessment Capacities Project: it supports and strengthens humanitarian capacities to carry out coordinated assessments before, during and after crises. Through development and provision of innovative tools, know-how, training and deployment of assessment specialists, ACAPS aims to contribute towards a change in the humanitarian system’s current practice with respect to needs assessments.

³  The UNHCR Specific Needs Codes categorise refugees into groups such as unaccompanied minors, disabled etc.

⁴  More specifically the weighting of scorecards was different.

⁵ Organisations weight vulnerabilities based on the objectives or specific persons of concern that they wish to target.

⁶  Steering Committee Members: CARE, Danish Refugee Council, Handicap International/Help Age, Première Urgence - Aide Médicale Internationale, Reach, WFP, WHO, UNICEF, WHO and, UN Women.

⁷UNHCR’s registration software programme (database application) used in more than 70 countries, see: http://www.unhcr.org/pages/49c3646cf5.html

⁸ A software development phase when software features are complete.