Update: Measuring Political Settlements

On September 3, 2015, the PRSP team held a round table discussion on “Conceptualising and Measuring the Success and Failure of Political Settlements: Constitutions and Peace Agreements.” A small but fascinating group of scholars from various disciplines gathered – all of them researching a similar question in their respective fields: how do you measure failure and success in relation to institutional settlements (whether a constitution or peace agreements, or both). Chaired by Dr Cormac Mac Amhlaigh, the panel included presentations by Professor Denis Galligan (University of Oxford), Professor Roger Mac Ginty (University of Manchester and PSPR International Board), Professor Christine Bell (University of Edinburgh) and closing remarks from Dr Alex Schwartz (Queens University Belfast, and PSRP Associate). The panellists are all involved in research projects that deal with mixed quantitative and qualitative methods to understand the relationship of institutional and legal frameworks with society, with a particular emphasis on conflict affected areas. The round table aimed to bridge different themes of the PRSP – peace processes and measurement, as part of our on-going thinking in this area.

As part of ‘The Science of Constitutions’ project, Denis Galligan explained how his project aims to determine the main variables of the relative success or failure in constitutional orders. This project is located in a broader socio-legal approach to understanding the correlation between different constitutions and the societies they apply to. The project asks: from within a sociological approach that links conceptualisation of constitutions with reference to their function in society: what does success and failure mean with regards to constitutions?

The research project involves three key steps: first, Identifying the variables in order to establish a spectrum of successful and failed constitutions (e.g. France vs. Egypt); second, identifying comparative weight of the identified variables – i.e. which are more important than others, and three, clarifying the overall success or failure of a constitution (according to criteria such as: the quality of the text itself, the effectiveness of applications, and the ‘acquiescence of the people, i.e. how the population react to the text and the durability of the constitution itself). The discussion addressed a number of conceptual challenges aimed at clarifying the appropriate criteria by which constitutional success or failure could be research structure of this endeavour, which was useful as it framed later discussions held at the round table.

The second presentation was by Roger Mac Ginty who presented two of his research projects which aim at measuring the concept of peace from very different perspectives. On the one hand, the bottom-up ‘Everyday Peace Indicators’ projects measures how local community members define their own perception of peace. Based on anthropological methods, these alternative indicators have been identified through the use of focus groups in fragile and conflict-affected states, and include dimensions relevant for in their everyday lives. One example, was the barking of dogs during night-time hours, which in conflict zones could indicate heightened levels of movement by soldiers or rebel fighters. Another example was an unwillingness to use the outhouse latrine during the night for fear of attackers.

Such factors, despite their very real correlation with insecurity, Roger explained, perplex some policy makers, one of whom told him for instance that, “we cannot silence dog!” Such comment illustrate the extent to which peacebuilding efforts are framed according to the perceptions of policy makers. The ‘Everyday Peace Indicator’ project underlines crucial aspects of peacebuilding. Among them, ‘politics’ was rarely mentioned by focus group participants; the perceptions of peace strongly correlated to gender; and crime rates are key to levels of insecurity. In addition, local communities are creative at finding everyday solutions as a means of coping with the consequences of conflict.

Mac Ginty contrasted this with another more ‘top down’ measuring project that he was involved in; the ‘Peace Accords Matrix’, which measures peacebuilding provisions within Peace Accords. The project tracks the implementation of 51 different types of provisions in 34 comprehensive peace agreements. The findings of this project provide greater insight into the global peacebuilding project and indicate a proliferation of provisions focused on security. Furthermore, the longitudinal approach underlines that things are changing, as increasing numbers of socio-economic aspects are also being included. From this research, the key observation by Mac Ginty is that more liberal peace agreements are also more durable. The Matrix also encouraged other various research outcomes and we invite you to see their list of publications.

Lastly, Christine Bell presented the portfolio approach used in the PSRP PA-X data base project. Part of the Political Settlements Research Programme is developing a ‘Peace Agreement Access Tool’ (PA-X) which codes and maps all the peace agreements implemented since 1990, which so far includes roughly 1,200 in 102 different countries. In the following months, this data will provide a large-N statistical picture of how peace agreements deal with issues of inclusion, poverty, development, and conflict resolution on a global scale. This project analyses peace agreements through the exploration of trade-offs among elite bargains and other forms of inclusion and it also consider the difficulties of implementing peace agreements. This project uses peace agreement analysis as a ‘mid-level’ methodology capable of showing both broad global trends and providing qualitative detail for each country. The data will enable a set of correlations to be explored quantitatively and qualitatively.

Rounding off the discussion, Alex Schwartz highlighted three key observations emanating from the three presentations: (1) the importance of pluralism of approach: process tracing, quantitative and qualitative research methods; (2) the need to disseminate quantitative results in a clear manner – i.e. with writing and graphs that also make sense to non-quantitative researchers, and (3), the necessity to address the issues of reliability and the validity of the research material in the analysis – for example by not pretending to have perfect data.

These points are also touched on in Mac Ginty’s recent blog post – ‘A Summer of Rejections’ – on the difficult dissemination of mixed methods research in academic journals:

Mixed methods research is often messy and contradictory. Mine attempts to mix my long-running critical qualitative interest in international intervention (influenced by ethnography and post-colonialism) with empirical studies from the Peace Accords Matrix and the Everyday Peace Indicators project. You can see how it is neither fish nor fowl. Yet, if only “safe” material can be published, where is the room for innovation, risk-taking and creativity? Instead we are limited to the realm of replication, discipleship and … well … dullness.

These different projects are crucial as a mean for peacebuilding to move on from a reliance on assumptions towards evidence-based practice. Overall, the discussions included fruitful brainstorming about possibilities and limits of measurement. More importantly, it produced a lot of food of thought for ongoing analysis and the future outcome of our RPSP measurement projects.

Similar discussions continued among our team members and the programme associates Sahla Aroussi (University of Coventry), Cindy Daase (Konstanz University) and Alex Schwartz on September 7, which focused on the work of PSRP and discussed the coding process, challenges within it, and strategies to overcome them and intelligibly categorise provisions, while recognizing the limitations of the database. We then scheduled and team up for writing on different matters to be analysed, such as women provisions, TJ provisions, the global trends and unusual provisions emerging from very specific contexts.

See also our working paper on measurement, by PSRP consortium member ISS.

Stay tuned for forthcoming research!

By Astrid Jamar, PSRP Research Assistant

Photo: Martina Bacigalupo