Impact Measurement Can also Work in Fragile Contexts

Often these are contexts of lawless violence, which can embolden non-state actors to use force for their own ends. There is then a far higher risk that development cooperation measures will either be completely ineffective or even make the situation worse. Ineffective or weakly effective measures give rise to high opportunity costs. They tie up funding that could have been
invested in effective projects in other locations. These costs are ultimately borne by another population elsewhere that misses out on a water supply project, for example.

Negative effects

However, it is negative effects that are especially problematic. For instance, resources from donors might entrench or ignite distributional struggles between ethnic
groups. The legitimacy of the state can be further undermined if the population gains the impression that state officials are enriching themselves from development
projects, or if cooperation takes place only with partners from civil society so that parallel structures are created (see pages 58–59, DEval´s Focus Report on "Fragility"). In addition, local rulers can “tax” development cooperation measures, meaning that, directly or indirectly, development organisations have to pay protection money to be able to implement their projects at all.

Minimising risks

To minimise these risks, even in fragile states it is imperative to evaluate the effects of interventions as well as possible. The following four measures may help.

1. Despite all the impediments, in almost all cases it is possible to carry out rigorous evaluations. For example, a meta-analysis of international evaluation activities in Afghanistan shows that development organisations conducted 32 rigorous evaluations in the country between 2008 and 2018. Of course this is still far too few relative to the total funding flowing into Afghanistan. At the same time, the experience they represent demonstrates that methodologically demanding impact evaluations are indeed possible inhighly fragile contexts. The bulk of this impact measurement was carried out in cooperation with research institutions, which illustrates the high – and, unfortunately, still often unutilised – potential of cooperation between development organisations and research in the field of rigorous impact evaluation (see pages 84–86, DEval´s Focus Report on "Fragility").
2. Rather than undertaking complex impact measurement, it is always possible to conduct less demanding performance audits. These examine how a project was managed, whether the intended outputs were delivered, and whether it is at least plausible that impacts could have been achieved. For example, US performance audits established that many US cooperation projects were ineffective because they did not even achieve the intended objectives at output level.
3. There are innovative approaches to data collection (see pages 60–61, DEval´s Focus Report on "Fragility") such as a database of German development cooperation in Afghanistan that records all the outputs of German projects in conjunction with georeferencing data. Also operated by German development cooperation, the Risk Management Office issued local conflict analyses, which were prepared by international and local experts on the ground and regularly updated. Meanwhile, representative surveys continued to be conducted at district level. By synthesising data from these three data sources, it is possible to draw conclusions about trends and effects of development cooperation.
4. Another instrument that is very helpful but, unfortunately, still far too seldom used is the country-specific systematic review, which is a cross-donor summary of the available evaluation evidence. Systematic reviews identify all the evaluations in a fragile country that satisfy certain criteria, such as language, year of publication and methodological quality, and evaluate them in relation to certain content-based criteria. Systematic country reviews of this kind provide an objective, transparent and replicable summary of everything that is known about the effectiveness of development cooperation in a fragile country. They are an effective and necessary corrective to anecdotal findings and dogmatic wishful thinking, and support coordination and the division of work between different donors, which is often especially demanding in fragile contexts.

Example: Meta-analysis on Afghanistan

A prominent example of such a country review is the meta-analysis on Afghanistan commissioned by the BMZ in 2020, which takes 148 studies into consideration. The analysis found that interventions in the fields of health, education, water and livelihoods were somewhat effective. Measures on good governance, women’s rights and stabilisation very seldom showed any effect, however. On that evidence, development cooperation organisations should only be able to implement projects in such difficult fields if they can substantiate why their projects
would be better and more effective than all the other ineffective projects to date. Moreover, they should only implement them as pilots initially, flanked with a solid monitoring and evaluation system. Because country-based systematic reviews summarise the experience of all donors in a given context, they can also strengthen multi-donor accountability and learning processes. It would therefore be desirable if the international donor community undertook to produce systematic reviews for all fragile contexts on a regular basis. Especially in the light of recent developments in Afghanistan and Mali, this pragmatic and low-cost instrument should  be used considerably more often and at an earlier stage, especially in fragile states.