Introduction

Background

The journal impact factor (JIF), a representative indicator for measuring the influence of journals, provides insights into whether a journal is influential in the relevant academic subject area and can be used to compare and analyze the importance of journals by subject area. However, in situations where a large number of papers are published in a short period, such as during the COVID-19 pandemic, the JIF value may change significantly and be overestimated due to self-citation. Several indicators have been proposed to overcome the JIF’s limitations. JIF without self-citations is an indicator used to solve the problem of exaggerating the JIF value through self-citation [1]. The 5-year JIF is an indicator that calculates the paper citation period as 5 years, evaluates the impact of the journal in the long term, and provides information on a journal’s impact in a field where the paper publication period is long [2].

Because the JIF varies greatly by discipline and can be affected substantially by events occurring during the measurement period, a new parameter, the Journal Citation indicator (JCI), has been used since 2020. The JCI focuses on the category-normalized citation impact, making it suitable for comparing the impacts of journals in different fields. While the JIF evaluates the impact of papers using citations from the past 2 years, the JCI takes a longer period (3 years) for the evaluation. The JIF can be significantly affected by special circumstances in which a large number of papers in the same field are published in a short period, but the JCI is relatively free from this risk [3].

The outbreak of COVID-19, a new infectious disease necessitated the rapid dissemination of new information, including clinical characteristics, treatment, prognosis, and impact on the community. This has led to an increase in the volume of academic publications and the speed of publication [4]. The impact of this on journals’ quantitative indicators has been previously reported. For instance, the JIF of pediatric journals surged significantly in 2020 after the COVID-19 outbreak, which was related to the sudden increase in total publication volume at that time [5]. A rising JIF trend was also observed for journals in other fields such as pulmonology, where both open access (OA) and subscription journals significantly increased their publication volumes in 2020; however, OA journals showed much higher growth than subscription journals [5].

Similarly, in six highly influential medical journals (Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, Lancet, Nature Medicine, and New England Journal of Medicine), the JIFs increased significantly compared with the pre–COVID-19 era; from 2019 to 2020, increases of 283%, 199%, 208%, 392%, 111%, and 196% were observed, and in 2021 the JIFs further increased by 41%, 90%, 6%, 22%, 53%, and 72%, respectively. The number of publications increased annually during the study period, and JIFs tended to increase. Among them, papers not related to COVID-19 showed a gradual increase, like in previous years, whereas the number of papers related to COVID-19 increased relatively steeply [4]. Therefore, the JIF should be used carefully when evaluating journals.

According to the 2021 Journal Citation Reports (JCR), clinical medicine is the academic area in Korea with the most publications, accounting for 18% of the total. The number of clinical medicine papers published in Korea was 15,137, ranking 11th globally (3.68%). The number of citations per paper in clinical medicine area was 1.65 worldwide and 1.51 in Korea [6].

The JIF is an indicator that evaluates the scientific impact of a journal. Owing to the COVID-19 pandemic, the number of publications has increased rapidly, and quantitative evaluation indices of journals, such as the JIF, have also increased. In Korea, COVID-19 broke out in January 2020 and several outbreaks have since occurred. The quantity of publications and citations of medical papers in Korea indicates that it is a country with high publication influence.

Objectives

This study compared the changes in the quantitative indicators of Korean medical journals, including JIF, during the 2020 COVID-19 epidemic and thereafter.

Methods

Ethics statement

This study did not involve human subjects, so neither approval by the Institutional Review Board nor obtainment of informed consent was required.

Study design

This was a bibliometric study based on a literature metrics database.

Study setting

The JCR database was searched on July 10, 2023.

Data sources and measurement

JCR analyzes the citation information of the world’s most influential journals, including the Science Citation Index Expanded (SCIE), Social Sciences Citation Index, Arts and Humanities Citation Index, and Emerging Sources Citation Index. The JCR provides data needed to understand the components used to index each journal’s value and impact. The key metrics provided include the JIF and JCI [7].

We selected journals classified by country/region to include South Korea, from the JCR database. Among them, journals listed in SCIE and classified as “clinical medicine” in categories such as “allergy,” “health care sciences & services,” and “rehabilitation” were selected. To identify changes during the COVID-19 pandemic, we extracted data from 2018 to 2022 (i.e., for 5 years), and found 58 journals (52 in 2018, 57 in 2019, 61 in 2020, 67 in 2021, and 69 in 2022). After excluding overlapping journals, 43 journals with JIF reports from 2018 to 2022 were selected for the final analysis.

All included journals and three areas with a large number of journals in the corresponding academic fields were selected, and major indicators, such as the JIF for the 5-year periods, 2018, 2019, 2020, 2021, 2022, were presented as means and standard deviations. Among the included journals, the key indicators of the five journals with the highest JIF as of 2021 are presented.

Variables

We included the journal name, JCR year, category, JIF, JIF without self-citations, 5-year JIF, JCI, Eigenfactor score, Article Influence Score, Immediacy Index, and the percentage of gold OA journals [7]. Definition of each metric is described in Suppl. 1.

To identify changes in indicators before and after COVID-19, the ratio and difference of each indicator were calculated, i.e., “(2020–2021 average) / (2018–2019 average)” and “(2020–2021 average) – (2018–2019 average),” respectively.

Bias

There was no bias in searching and selecting the target literature.

Study size

It was not necessary to estimate the sample size. All target journals were included.

Statistical analysis

Descriptive statistics were applied. All analyses were performed using SAS ver. 9.4 (SAS Institute Inc).

Results

Trends in the impact of journals in Korea before and during the COVID-19 pandemic

The average JIF of the 43 journals increased from 2.33 in 2018 and 2.50 in 2019 to 3.45 in 2020 and 3.86 in 2021. The 5-year JIF and JCI showed similar patterns, and the percentage of gold OA journals increased from 60.0% in 2018 to 79.8% in 2021. However, the JCI and Eigenfactor scores remained consistent or showed relatively small increases. The number of journals in each of the analyzed disciplines ranged from 1 to 5, the three areas with the highest number of included journals were clinical neurology (n= 5), oncology (n= 4), and pharmacology and pharmacy (n= 4). The JIF ratio increased as follows: clinical neurology, 1.45; oncology, 1.51, and pharmacology and pharmacy, 1.43; and the 5-year JIF, JCI, and percentage of gold OA journals all increased. However, impact measures declined in 2022, including a JIF decrease to 3.55 in 2022 (Table 1 and Fig. 1). Suppl. 2 provides the full list of the included journals.

Trends in the impact of the top five journals in Korea before and during the COVID-19 pandemic

In 2021, all the top five JIF journals (Experimental and Molecular Medicine, Journal of Stroke, Korean Journal of Radiology, World Journal of Men’s Health, and Archives of Pharmacal Research) showed increases in the JIF, and World Journal of Men’s Health had the highest JIF ratio at 2.47. The two journals with the highest JIF in 2021 (Experimental and Molecular Medicine, Journal of Stroke) and World Journal of Men’s Health, with the highest JIF increase, were gold OA journals. However, in 2022, JIF decreased compared to the previous year or did not increase to the same degree as in the previous 2 years (Table 2).

Discussion

Key results

This study compared the quantitative impact indicators of Korean international journals over 5 years before and during the COVID-19 pandemic. The JIF, 5-year JIF, and JCI of the 43 included journals increased steadily, and the percentage of gold OA journals also increased significantly through 2021. However, the JCI and Eigenfactor scores remained steady or showed relatively small increases, and, impact measures (including the JIF) declined in 2022.

Interpretation

Our study has the following strengths: to the best of our knowledge, this is the first study to address changes in the JIF and related impact indicators before and during the COVID-19 pandemic in Korean international medical journals. Therefore, our study provides time data on changes in impact indicators before and during the COVID-19 pandemic.

This study included not only the JIF but also various indicators, such as JIF without self-citations, 5-year JIF, Eigenfactor score, Article Influence Score, and Immediacy Index, that compensate for the shortcomings of the JIF, and the magnitude of change was presented through both ratios and differences.

The JIF differs greatly by academic field because the number of published papers varies from field to field. For example, among Korean journals in 2021, the JIF of the Korean Journal of Radiology was 7.11, approximately 4.14 times higher than that of the Korean Journal of Physiology and Pharmacology (1.72). To address this issue, this study included the JCI, an indicator that compensates for differences by field.

Comparison with previous studies

This study found an increase in the JIF in 2020 and 2021 compared with that in 2018 and 2019. The JIF showed a steady increase over the 4 years, and the increase was smaller in 2020–2021 than in 2019–2020. This is similar to the findings in a previous study in which the JIF in influential medical journals tended to increase during the COVID-19 pandemic [4]. However, the previous study selected only six representative medical journals to compare the JIF, whereas this study included all international journals in Korea. In addition, comprehensive trends were presented for quantitative indicators of journals, including the 5-year JIF, Eigenfactor score, Immediacy Index, and JIF.

In a study of differences by publication type, both OA and subscription journals continued to show increasing numbers of citations and publications from 2018 to 2020, and the number of citations and publications in 2020 increased significantly compared to 2019. The increase in the number of citations and publications was greater in OA journals than in subscription journals [5]. In our study, the percentage of gold OA journals among 43 Korean journals in the medical field increased significantly, from 60.0% in 2018 to 79.8% in 2021.

In a study evaluating the use of MeSH (Medical Subject Headings) terms to assess the impact of COVID-19 on scientific research production in the life sciences, the use of COVID-19–related MeSH terms, such as SARS-CoV-2, increased 6.5-fold, whereas that of unrelated MeSH decreased by 10% to 20%. This may have been due to the journal’s editorial policy on the rapid review and publication process of COVID-19–related research; however, it has raised concerns about a decline in biomedical research that is not related to COVID-19. The increase in OA publications may have been influenced by the editorial policies of journals that prioritized providing COVID-19–related information to the public [8]. However, this study investigated the quantitative indicators of all medical journals and did not evaluate the relevance to COVID-19 (e.g., MeSH terms or the topics of papers). Therefore, we could not directly evaluate whether the increase in the quantitative indices of journals, such as the JIF, was due to an increase in COVID-19–related publications. In future research, it will be necessary to add COVID-19–related characteristics, such as the journal’s COVID-19–related publication policy, study topics, study population, and key variables.

Implications for future studies

Quantitative evaluation indices of journals, such as the JIF, can be used to select influential journals and spread research. However, with the emergence and spread of COVID-19, the volume of publications has soared in a short period, raising concerns about the appropriateness of quantitative indicators for evaluating journals. The JIF surge serves as an example of a phenomenon of concern. This study presents various quantitative indicators during the COVID-19 pandemic in international journals based in Korea.

Our study targeted all journals classified in the field of clinical medicine within the JCR and could broadly visualize changes in various indicators, including the JIF, before and during the COVID-19 pandemic. Future research investigate how journal impact indicators are impacted by factors that can affect publication speed and citations, such as the journal publication policy and peer review processes.

Quantitative journal indicators, such as JIF, have been used as evaluation indicators by journals and researchers for research productivity, academic impact for research proposal approval, and research funding support in the field of education or research. A useful indicator requires a system that reflects the characteristics of each academic field, is rigorous and transparent, and can produce stable long-term values. Many studies, including ours, have reported that special circumstances, such as the COVID-19 pandemic, can make a major contribution to changes in journals’ quantitative indicators. Therefore, academic and research communities should consider this when evaluating journals and researchers. Academic journals should continue to operate rigorous peer review and publication processes that control the rate and volume of publications and maintain and improve research quality.

Limitations

This study did not include all Korean journals because it only examined Korean journals listed in the JCR. However, because we focused on international journals listed in SCIE, we suggest that our findings are representative of the major influential journals published in Korea. Moreover, factors that may affect journal impact measures, such as the publication process of journals and papers with a high frequency of citations among the papers published in journals, were not considered.

Conclusions

The COVID-19 pandemic has resulted in a surge in research and scientific production in all areas of knowledge worldwide, especially in COVID-19–related research. The rise of COVID19 publications has met the demand for scientific advancement, improved coping capabilities, and enhanced information-sharing on new emerging infectious diseases; however, the reliability of indicators for assessing academic impact has been questioned. Although the JIF has functioned as a useful indicator in the era of library-centered journal access in the past, its reliance on JIF has raised many concerns in modern society regarding various access and distribution methods for journals. We presented trends of quantitative indicators in journals, such as the JIF, in Korean international medical journals, and found an overall increase in these indicators. Because it may be difficult for quantitative indicators such as the JIF to reflect the quality of journals in special circumstances, such as the COVID-19 pandemic, journals should maintain strict publication procedures to improve the quality of research. Additionally, the research community should exercise caution when using quantitative evaluation indicators for journals. Future research should examine journals’ publication policy, research content, and long-term results.

Notes

Conflict of Interest

No potential conflict of interest relevant to this article was reported.

Funding

The authors received no financial support for this article.

Data Availability

Data analyzed in this article are available from the corresponding author upon reasonable request as Clarivate does not allow researchers to share the data retrieved from their product.

Supplementary Material

Fig. 1.

Trends in the journal impact factor of Korean medical journals according to year.

References

• 1. Górski A, Zimecki M, Krotkiewski H. Journal impact factor and self-citations. Arch Immunol Ther Exp (Warsz) 2021;69:2. https://doi.org/10.1007/s00005-021-00621-w. ArticlePubMedPMC
• 2. Jacsó P. Five‐year impact factor data in the Journal Citation Reports. Online Inf Rev 2009;33:603-14.https://doi.org/10.1108/14684520910969989. Article
• 3. Huh S. The Journal Citation Indicator has arrived for Emerging Sources Citation Index journals, including the Journal of Educational Evaluation for Health Professions, in June 2021. J Educ Eval Health Prof 2021;18:20. https://doi.org/10.3352/jeehp.2021.18.20. ArticlePubMedPMC
• 4. Delardas O, Giannos P. How COVID-19 affected the journal impact factor of high impact medical journals: bibliometric analysis. J Med Internet Res 2022;24:e43089. https://doi.org/10.2196/43089. ArticlePubMedPMC
• 5. Mondal P, Mazur L, Su L, Gope S, Dell E. The upsurge of impact factors in pediatric journals post COVID-19 outbreak: a cross-sectional study. Front Res Metr Anal 2022;7:862537. https://doi.org/10.3389/frma.2022.862537. ArticlePubMedPMC
• 6. Han H. 2021 Paper publication and citation status of scientific research in Korea. Korea Institute of S&T Evaluation and Planning (KISTEP); 2023.
• 7. Clarivate. About Journal Citation Reports [Internet]. Clarivate; c2021 [cited 2023 Jun 12]. Available from: https://jcr.help.clarivate.com/Content/home.htm.
• 8. Riccaboni M, Verginer L. The impact of the COVID-19 pandemic on scientific research in the life sciences. PLoS One 2022;17:e0263001. https://doi.org/10.1371/journal.pone.0263001. ArticlePubMedPMC

Figure & Data

References

Citations

Citations to this article as recorded by  

• Scientific Publication Speed of Korean Medical Journals during the COVID-19 Era
  Hyeonseok Seo, Yaechan Kim, Dongryeong Kim, Hanul Kang, Chansu Park, Sejin Park, Junha Kang, Janghyeog Oh, Hyunsung Kang, Mi Ah Han
  Healthcare Informatics Research.2024; 30(3): 277.     CrossRef