Rewriting history II: The (un)predictable past of ESG ratings

Importance of ESG ratings

Academic research on environmental, social, and corporate governance (ESG) topics has exploded in recent years. This surge mirrors that of the investment management industry, where the importance of ESG principles has also seen a massive rise. For example, funds that invest according to ESG principles attracted net inflows of US$71.1bn globally between April and June 2020, despite the Covid-19 crisis, pushing their assets under management to an all-time high of over US$1trn.

But measuring a firm’s ESG quality – quantifying how well it performs with respect to ESG criteria – remains a key challenge for researchers and investment professionals alike. To address this, most empirical ESG analyses use ESG scores (or ratings) constructed by professional data providers, raising questions by policy makers, investors, researchers, and firms about their reliability, consistency, and overall quality.

Refinitiv ESG downloads

In a new paper, we document widespread changes to the historical scores that Thomson Reuters Refinitiv ESG assigned to companies and show that this has important implications for analyses linking ESG scores to outcomes such as firm performance or stock returns.

The ESG scores constructed by Refinitiv ESG, formerly known as ASSET4, are influential, having been used (or referenced) in more than 1,000 academic articles over the past 15 years(see Figure 1). Moreover, Refinitiv ESG data is used by many major asset managers to control ESG investment risks.

To document the rewriting of the ESG scores, we downloaded two versions of the same Refinitiv ESG data for the same set of firm-years at different points in time. We downloaded the first version of the data in September 2018, and the second version in September 2020. The data included overall ESG scores, as well as E, S, and G sub-scores. The sample contained 29,828 firm-year observations between 2011 and 2017 from 72 countries.

Divergences between data downloads

We observed that the ESG scores for identical firm-years differed between the two data versions – in some cases dramatically. In fact, not a single ESG score was the same across the two versions. Thirteen percent of the sample observations were subject to a score upgrade, that is, the rewritten ESG score was higher than the initial ESG score, while 87% of the observations received a score downgrade.

The differences between the two data versions raise the question of why and how the scores were changed by Refinitiv ESG. According to the vendor, the score deviations originate from adjustments in its scoring methodology, which came into effect on 6 April 2020 – between our two data downloads. Importantly, Refinitiv ESG applied the methodology change to newly created and historical scores in its database.

As we do not have access to Refinitiv ESG’s methodology to understand and verify these changes, we use statistical methods to determine how economic variables can explain the score deviations. We demonstrate that the ex-post score changes are systematic and partially driven by reassessments of industry- and country-level drivers of ESG performance (or risks). Substantial parts of the score rewriting also play out at the individual firm level. These firm-level effects can partially be explained by time-varying firm characteristics and past stock returns. Overall, we show that much of the score deviation originates from the reassessment of firms’ ESG performance in specific years following Refinitiv ESG’s methodology change. Further, we demonstrate that the score changes might have been data-mined, such that firms that performed better in a given year experienced an ex-post upgrade in their environmental and social (E&S) scores through the rewriting of the data (e.g. firms that had higher stock return in 2013 experienced an upgrade in their E&S scores for 2014).

Impact on the relationship between ESG ratings and performance

We then examine whether the ESG score deviations impact how the relationship between ESG scores and outcome variables are estimated and interpreted, focusing on S&P 1500 firms. We find that the deviations strongly affect the ESG-based ranking of S&P 1500 firms, which in turn impacts their classification into different ESG quintiles. For the overall ESG score, only 68.5% of firm-year observations are classified in the top decile (top 10%) in the initial and rewritten data versions; the numbers are similar for the bottom decile. We find similar patterns for the classification of firms based on their environmental, social and governance sub-scores. Hence, the retrospective score rewriting leads to large changes in what are deemed to be high or low-scoring ESG firms. This insight is important as the classification of firms based on these scores (or sub-scores) is widely used in ESG research and the investment industry.

We use the recent Covid-19 crisis to explore the effects of these classification changes, building on prior work showing that firms with higher pre-crisis E&S ratings exhibited better stock market performance during the pandemic. High-E&S firms are classified as such if they are ranked in the top quartile of the S&P 1500 sample based on the average value of their E&S scores. Our tests then compare daily abnormal returns of high- and low-E&S firms before and after a Covid-19 event date (24 February 2020).

When classifying firms based on the initial E&S scores, we find no evidence that high-E&S firms performed better during the Covid-19 pandemic. This picture looks entirely different if we run regressions using a classification of firms based on the rewritten data. We now find strong evidence that high-E&S firms exhibited better performance during the pandemic.

Implications for researchers and investors

The large differences in results that we document have economic implications. Retrospectively, we would attribute a positive performance effect to high-E&S firms when they are classified based on the rewritten data. However, this would not have been achievable with the data available to investors before or at the onset of the pandemic. At that point, investors would have classified firms differently into high- and low-E&S groups, and the performance variations between these two sets of firms would not have been economically and statistically significant. Hence, the benefits of being invested in a high-E&S firm during the crisis would have been exaggerated.

The implications of this observation extend beyond our setting. They apply more broadly to the back-testing of ESG strategies, as it is critical to verify that the original, not the rewritten, scores are being used for such tests.

The consequences of the documented data rewrite may be limited if it reflects a one-time event. However, this does not seem to be the case – we show that Refinitiv ESG continues to rewrite its historical ESG data, albeit without announcing these changes to the public. We demonstrate this more gradual rewriting by comparing ESG data downloads from September 2020 and November 2020. The two downloads also reflect a rewriting of the ratings data for the years 2011 to 2017, which once more changes how firms are classified into ESG quantiles.

Our insights are critical for future ESG research using Refinitiv ESG data – we recommend that researchers using this data verify whether the initial, originally available data is needed to test their hypotheses. This consideration is important, given that ESG research is expected to continue growing.

 

 

This blog is written by academic guest contributors. Our goal is to contribute to the broader debate around topical issues and to help showcase research in support of our signatories and the wider community.

Please note that although you can expect to find some posts here that broadly accord with the PRI’s official views, the blog authors write in their individual capacity and there is no “house view”. Nor do the views and opinions expressed on this blog constitute financial or other professional advice.

If you have any questions, please contact us at [email protected]