Working with the Notre Dame Initiative for Global Development, the adelphi Consult,  and the Catholic University of Mozambique, ND-GAIN has been involved in a program evaluation funded by United Nations Development Programme in Mozambique, to study the effectiveness of a national disaster risk management program implemented at local level. USAID’s Coastal City Adaptation Program operated by Chemonics has provided network and logistic supports on ground for the work.

Due to the multiple technical difficulties, monitoring and evaluation (M&E) of climate adaptation/disaster resilience projects or programs has been considered a “wicked problem”. Even though methodologies and frameworks have been proposed to deal with these difficulties, there have not been many published reports that provide rigorous evaluation using these proposed frameworks. This project, therefore, is among the first to give a concrete example for upcoming M&E practices in the field of climate adaptation and disaster resilience.

Mozambique is a large coastal country and historically exposed to flood risks due to low elevation in its coastal areas. Climate change is likely to bring more frequent and intensified flood events. Many socioeconomic factors could exacerbate the flood risks, including low income, poor infrastructure, and weak social institutions. To respond to these challenges, the National Institute for Disaster Management (INGC) started a nation-wide implementation of community-based disaster risk management committees (CLGRCs) program in 2002, in areas identified to have higher flood risks. Committee members are elected by communities and composed of respected local leaders. They receive and communicate early warning information and coordinate emergency response locally (GFDRR, 2014). Since 2002, more than 1,200 committees have been placed all over Mozambique. Anecdotally, the CLGRC program has been considered successful and started being scaled in other regions with similar risks. However, evidence-based impact evaluation is lacking.

Coastal elevation map of Mozambique (Source: www.floodmap.com)

For the evaluation of the program, a big challenge is the measurement of the achievements made by the program, especially when the world lacks a unit measure to quantify adaptation or resilience success. We followed the result chain below, breaking down the program achievements into outputs, outcomes and impacts. Each of the three steps was carefully defined to line up with the goals specified by INGC.  

The findings of that scoping mission were considered in the evaluation design. We heard many cases that unanimously point to the importance of social capital, which could either be strengthened or weakened by a community-based adaptation/disaster resilience program like CLGRC. For example, when asked about the challenges that a committee perceives in its operation, a committee member mentioned the lack of power to mobilize community members for works such as removing trash from drainages. On one hand, the elected committee members, with certain sense of privilege, worry that labor works in front of their community peers could weaken their authority as much as it could weaken the disposition of community members to follow their leads.  On the other hand, the community members have not seen themselves unified under the committee’s leads. In fact, for multiple reasons, a majority of them are still unsure about the roles and functions of the committee in their own community. In that case, people sometimes turn their back on the committees, thinking committee is another bureaucracy, which defeats the purpose of planting committees in communities. Hence, we believed that whether and how committees have strengthened collaboration among community members, and between community members and committees, should be captured in our outcome measures. In another word, the social capacities, indicated by social capital and assets, should be used as part of the metrics to measure the program achievements.To further measure the outcomes and impacts, in Aug. 2016, we went on a scoping mission in two coastal cities pre-determined as the project sites. Several group interviews were conducted with selected committee members for us to understand the designated tasks of the committee, as well as local residents’ subjective assessments of flood risks.

Result Chain of the CLGRC program, adapted from the Result Framework by Williams et al. (2016)

We eventually built a measurement framework that seeks to explain flood risks from both the individual and the societal perspectives.

Framework that measures flood risks for the CLGRC program evaluation

Community and households jointly define the resilience to flood risks that the committees were commissioned to mitigate. The households’ typical vulnerability profile was first considered in our framework, including the demographic features, housing condition, assets, livelihood, shelter, and financial resources. In addition, we value the relationships between households and the community capitals as important risk reduction factors when the system is facing a flood shock. Elevation of the community, drainage infrastructure, early warning, community preparedness, and community capital are key in our framework.

Due to the long time horizon of typical adaptation interventions, impact measurement is particularly challenging. However, the program that we are evaluating started 15 years ago so the team was required to measure its impacts on flood risk reduction, even though the latest committee was built in 2015 in our study areas. In the rainy season 2016/2017, both cities had intensive rainfall and a couple of flood events materialized, causing certain damages and losses in some areas. Tracking the reoccurring flood events, the team decided to use the flood impacts as the inverted proxy of the program impacts, assuming that communities that have benefited more from the program should perceive less impact from flood. Such evaluation design also makes sense because of the timeframe of the rainy season, which is at least one year after the latest committee took some immediate actions for risk reduction. To measure the “outcome” of the flood risks, we define the flood impacts from four perspectives: lives, livelihoods, housing, and assets. Getting these statistics at household level is challenging, even with the household survey. So we focused on households’ experience to measure “whether” instead of “how much” certain impacts were observed in the past rainy season.

In May 2017, the team completed the data collection in the study area, having interviewed 541 households in 9 control communities (i.e. communities that do not have a committee established) and 9 treatment communities (i.e. communities with the committees). Qualitative information was also collected through focus group discussions, where committee members (for the treatment communities) were consulted for the team to understand how each interviewed committee had been operating, seeking to find reasons why some seem work better than others and whether there is operational experience to share.  The quasi-experiment approach was designed for data collection and analysis. We are hoping to share with you the evaluation findings soon.