What will remain of the Impact Factor?

As has been the case for many years, at about this time of the year the world of research is full of expectancy. This year, even more so. Clarivate, as customary, has shared the 2022 release of the Journal Citation Report (JCR), which includes the Impact Factor scores relative to citations received in 2021 to articles published in 2019–2020. For the first time ever, JCR has introduced a major change and after years of debates on the value, calculation methods, use and relevance of the Impact Factor, considered the king of bibliometry for the evaluation of researchers and institutions, for the first time we jolted.

While the 2019 performance was in line with that in previous years, 2020 was heavily influenced by two events, which had an incredible—an in part unprecedented—weight: the inclusion of early access content in the JCR for the first time, and the emergence of COVID-19, a novel and prioritarian field of research. These events made the 2020 data particularly unusual. As Figure 1 shows, larger number of journals increased their impact factor from 2019 to 2020, as opposed to any other year-to-year interval in the past decade (Fig. 1) (1).

Analyzing the elements that created these remarkable changes showed that early access content contributed to Journal Impact Factor (JIF) citations for about 10%. The remaining remarkable change was due to a generalized increase in the volume of content published in 2020, the pandemic year. This most recent release contained over 25% more articles and citations compared to the previous year. It was not only articles on COVID-19: research efforts to face this global health crisis were extremely intensive across various disciplines (2). The pandemic impacted every aspect of life, from economy to tourism, political affairs, music, arts and sports, generating a global effort to find ways to understand it and deal with it.

In this whirlwind of novelty, there are still some open questions. First, how is early access content counted? For Web of Science early access is defined as “the version of the record” posted online prior to final assignment in a set volume/issue (3). The pandemic has given a great impetus to share (not just publish) new research outputs as quickly as possible (think of the use of preprints now also available in PubMed). But ahead of print or early access, the phenomenon existed for many years; hence, the need to incorporate early access in JCR to accurately represent how researchers use literature in their daily work. This approach made early access a new source of citations in JIF numerators last year, contributing to the widespread increase in JIF across the entire JCR. For the first time, early access content has been incorporated into the count of citable items in the denominator (total of articles published in the previous two years) of the JIF (3,4). In addition, the continued expansion of early access content throughout 2021 provides another increase in the number of JIF numerator citations (calculated on citable articles, now including early access, published during the previous two years). COVID-19 did the rest.

First the first time, with an important leap in the category “Medicine General and Internal—SCIE,” Lancet’s IF surpassed that of the New England Journal of Medicine (NEJM). The Lancet’s IF more than doubled compared to the previous year, increasing from 79.32 in 2020 to 202.73 in 2021 (5). The IF of NEJM also nearly doubled, from 91.25 in 2020 to 176.07 in 2021 (5). Lancet can thank the COVID-19 pandemic for its surge, as three of the ten most cited scientific articles of 2021 appeared in the Lancet, all of them related to COVID-19 (Fig. 2).

Five other journals also had an impact factor higher than 100 for the first time and published a large amount of COVID-19 research: the Journal of the American Medical Association, Lancet Respiratory Medicine, Nature Reviews Drug Discovery, Nature Reviews Immunology, and Nature Reviews Molecular Cell Biology. But many more registered an increase in their IF. Simply scrolling tweets on Twitter shows a flourishing of journals sharing their new IF, generally with a more or less substantial increase. With no doubt, we have no memory of other events that have come close to the effects that COVID-19 research documents had on citation records in 2021. However, as the impact factor is so sensitive to highly cited articles, there is a serious possibility that some journal scores will reach stratospheric heights this year, for collapsing the following year. Clarivate’s editorial integrity team is extremely efficient. It eliminates journals from its report, that is, does not attribute them an IF, if it identifies an anomalous citation behavior that could distort it, such as excessive self-citation or citation stacking, sometimes called participation in “citation cartels” or “citation rings.” In the 2022 release of JCR, Clarivate removed 3 journals for self-citation; fewer than the 10 were removed in 2021 and the 33 removed in 2020.

In addition, this year Clarivate’s editorial integrity team identified a new type of citation behavior that could distort IF: we are talking about “self-stacking,” in which “a journal contains one or more documents with highly concentrated citations in the JIF numerator of the title itself.” (6). As this was the first year they defined the term, they issued warnings to six journals identified as “self-stacking citations,” instead of suppressing them, and the warning certainly deserves attention.

All those directly involved with citations and their metrics share a great question. Will we see a “return to normal” after the unprecedented changes reported above? Or will we continue to see new effects from the influx of COVID-19 research and the inclusion of early access content in JCR metrics calculations? Unfortunately, this superloaded boom-and-bust cycle will only fuel a greater sense of skepticism on the meaning and interpretation of the IF. And this brings me back to the title of this editorial: what will remain of the Impact Factor? Can it still be considered a cardinal bibliometric parameter for evaluating research? Will early access, which will increasingly count for the IF calculation, push magazines to rely more and more on these types of articles? Will this change the publisher models and the peer review, with a strive to publish quickly? What will happen to the H-index, which is based on citations received in articles incorporated in a volume/issue, as the IF itself was before this major upheaval? I am sure these questions are not just mine.

Disclosures

Conflict of interest: The authors declare no conflict of interest.

Financial support: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.