People Power Under Attack 2021 - methodology
Methodology
The CIVICUS Monitor aims to provide a comprehensive assessment of the conditions for civil society within countries and over time. It analyses civic space, defined as the respect in policy, law and practice for the freedoms of association, peaceful assembly and expression and the extent to which the state protects these fundamental rights.
In an attempt to capture these dynamics on a global scale, over 20 organisations with research expertise from around the world joined forces on the CIVICUS Monitor with the aim to provide an evidence base for action to improve civic space.
Civic space updates from CIVICUS Monitor research partners contain qualitative, narrative information related to the situation for civil society in a country. This qualitative information is directed by a set of guiding questions and the resulting data is gathered from a variety of primary and secondary sources. In many cases, country-specific updates come directly from national civil society themselves. In countries where there is no research partner, the CIVICUS Monitor relies on a variety of other sources produced at the national, regional and international levels to produce analysis on civic space.
These civic space updates are then triangulated, verified and tagged by the CIVICUS team. The tagging system helps us classify the information included in the updates and allows us to analyse the most common violations and restrictions faced by civil society groups, activists and journalists when defending their rights.
The research partners posted a total of 568 civic space updates from 1 November 2020 to 31 October 2021, which form the basis for the analysis presented in this report. For the period assessed, these civic space updates cover 156 countries and territories.
In addition to the qualitative information presented in the civic space updates, in order to draw comparisons at the global level and track trends over time, the CIVICUS Monitor produces civic space ratings for 197 countries and territories.[1] Each country’s civic space is rated in one of five categories – open, narrowed, obstructed, repressed or closed – based on a methodology that combines several sources of data on the freedoms of association, peaceful assembly and expression.
This year we introduced a new quantitative indicator to evaluate the degree of respect for the freedom of peaceful assembly in each country: the Peaceful Assembly indicator developed by Varieties of Democracy (V-Dem). This indicator replaced the Political Terror Scale (PTS) indicator, which measures the level of political terror within a country. At the time the Monitor’s methodology was developed, this was selected as the best source of data available to evaluate the freedom of peaceful assembly. Now a more precise indicator that measures the ability to assemble publicly in practice is available.
Because the CIVICUS Monitor surveys a variety of metrics before arriving at a country rating for civic space, this change did not cause any substantial shifts in our ratings. The PTS was one of three data points making up the Key Scores, which is one of the five data streams that lead to final ratings. Because a multiplicity of sources is used to arrive at the final ratings, overreliance on any particular data source is avoided and the stability and the sustainability of the CIVICUS Monitor ratings in the face of methodological changes are enhanced.
Frequently asked Questions
At the heart of the CIVICUS Monitor’s methodology is the combination of several independent data sources, comprising both quantitative and qualitative data. These sources include updated ratings from civil society organisations and reports from national, regional, and international civil society organisations with relevant information on the four indicators of civic space (Freedom of association, freedom of peaceful assembly, freedom of expression and the state duty to protect). These external analyses are then paired with CIVICUS’ own analysis to arrive at country ratings for all assessed countries and territories, ranging from open to narrowed, obstructed, repressed or closed.
| Key Analysis | Key scores | CIVICUS Analysis | Research partners input | Civil Society voices |
|---|---|---|---|---|
| Reports produced on civic space by civil society | International indices on civic space indicators by international CSOs and academic institutions | Country specific reports on civic space produced by CIVICUS | Periodic narrative and quantitative reports by CIVICUS’ network of research partners | Structured interviews with national level civil society groups |
The goal of CIVICUS Monitor is to provide a comprehensive assessment of the conditions for civil society within countries and over time. Civic space is defined as the respect in policy and practice for the freedoms of association, peaceful assembly and expression. The state has a fundamental duty to protect these rights. As the diagram illustrates, the CIVICUS Monitor conceptualises the conditions for civil society as the respect for these four indicators.
The CIVICUS Monitor combines qualitative and quantitative data inputs generated by primary and secondary research. These inputs are split into two distinct categories: 1) the base score and 2) the live adjustment score.
Recognising that indicators used to conceptualise civic space cannot be directly observed or assessed by a single measure, our methodology constructs a composite indicator to provide an overall view of conditions in each country. To do so, we created a fixed scale which establishes the range of the CIVICUS Monitor, i.e. we impose a minimum and a maximum theoretically possible value. This range encompasses a range of completely unrestricted (100) to completely restricted (1) civic spaces. Given that the state of civic space across all countries and territories vary from very restricted to more open, we can be reasonably confident that the range we set up is broad enough to encompass the variety of civic experiences around the world.
The CIVICUS Monitor uses information from its research network to constantly update a country’s score. In order to capture current events, one of our sub-regionally based research partners evaluates whether a country's current trend arches downward, upward, or is stable. Based on this input, a country rating could change at any moment.
In addition, once a year the CIVICUS team updates the indicators of the Base Score (composed of three aggregate components: key analysis, key score, and CIVICUS analysis). Based on this input, we undertake an annual analysis of countries ratings, resulting in changes that we release in our annual report: People Power Under Attack.
Underpinning all ratings is a numerical score which determines a country’s rating. Depending on what a country score is it could move up or down depending on the civic space situation on the ground, and for a rating to change it would need to move from the above numeral boundaries.
Open: (81-100), Narrowed:(61-80), Obstructed (41-60), Repressed (21-40) Closed (1-20)
We have constructed the CIVICUS Monitor so that all the base score indicators are implicitly weighted equally. With the Key Analysis indicator, we use explicit weights, to give greater weight to national sources than to sources produced by regional or international organisations. The advantage of our approach is that we give greater voice to national level civil society organisations producing data on civic space violations. We contend that actors closer to the source are best able to contextualise information and that potential incentives for over-stating restrictions are offset by local organisations' interest in remaining credible.
The Monitor top violations for each region can be further disaggregated.
On a periodic basis and in response to developments on the ground, CIVICUS researchers also conduct structured interviews with country-based civil society representatives. Based on these interviews, the CIVICUS researcher assesses the current trend in a country and assigns an additional “country consultation” score. In addition, anyone could submit verified information to our researchers and CIVICUS team will channel that information to our Research Partners.
Ratings are on a five-category scale defined as: open, narrowed, obstructed, repressed, and closed. These ratings are conceptualised as broad bands, where a variety of civic experience can exist within any given rating category. The goal of ratings is to offer comparisons between countries and over time which are robust but responsive to current events.
Methodology Update
The CIVICUS Monitor relies on a variety of sources before arriving at a rating for the quality of civic space in each country. The Key scores (International indices on civic space indicators by international CSOs and academic institutions) are one of those sources and it includes data from Freedom House’s Freedom in the World Index, The Political Terror Scale (PTS) and Reporters Without Borders’ World Press Freedom Index.
This year we are replacing one of those components: The Political Terror Scale is being replaced by the Peaceful Assembly indicator produced by V-Dem.
When the CIVICUS Monitor was launched in 2016, the most accurate indicator available to measure Peaceful Assembly was The Political Terror Scale. This indicator measures the level of political violence and terror that a country experienced. Recently, an indicator that strictly measures the degree of respect for freedom of peaceful assembly was developed by V-Dem. We are confident that these adjustments now mean that our ratings more precisely reflect realities on the ground in each country and will thus be more reliable over time.
The PTS measures the level of political violence and terror experienced by a country, conceptualised as state-sanctioned killings, torture, disappearances and political imprisonment. V-Dem measures the degree of respect for peaceful assembly using information gathered from experts that answer the following question: To what extent do state authorities respect and protect the right of peaceful assembly?
The CIVICUS Monitor methodology was designed to include a multiplicity of sources that are used to get to the final ratings. This was intentionally done to reduce over-reliance on any particular data source and to enhance the stability and sustainability of the CIVICUS Monitor ratings in the face of changes like this one. We are not changing the calculation or aggregation method of the data underlying the ratings; we only substitute one input in the quantitative component of our methodology, therefore the impact on the current and previous ratings is not significant.
In summary, the ratings assigned to each country capture the civic space situation by triangulating:
- qualitative information from reports covering the four core civic space freedoms and the role of the state in their protection.
- quantitative data reflecting the situation of each country in relation to their freedom of association, peaceful assembly and expression.
- inputs from research partners that provide information on the important developments regarding the civic space situation of the countries they cover.
Please note that the information contained in PTS (and now in V-Dem) is only one third of the ii) component.
The incorporation of V-Dem produces variations in the country scores even though their civic space situation may remain unvaried, however in most cases these changes are not significant enough to cause rating changes. Behind all downgrades and upgrades presented in the report there are events that have changed the civil society situation since last year.
No, there was a previous methodological change in 2018. These adjustments were part of the phase of development of the CIVICUS Monitor. CIVICUS will always reflect on the data being generated by the Monitor and will ensure that its methodology includes the latest thinking and data sources on civic space. In this vein we are now substituting the Political Terror Scale indicator for another closer to the freedom we are assessing, the V-Dem indicator.
[1] In 2021, we added Hong Kong to our rating analysis, distinct from China.