Correlation among the citation indices of Korean scientific journals listed in international databases
Article information
Abstract
We would like to verify the correlation among various citation indicators of 62 Korean scientific journals listed in the Web of Science (WoS) and Scopus. From a total of 85 Korean journals listed in both WoS as of January 2013, and 132 journals listed in Scopus as of 2011, 62 Korean journals listed in both citation indices were selected for analysis. Citation index indicators selected for analysis include impact factor (IF), 5-year impact factor (5yrIF), Eigenfactor score (EF), article influence score (AIS) (list of WoS indicators), SCImago journal rank (SJR), h-index, and impact index (ImIndex) (list of Scopus indicators). It took an average of eight years for a newly founded journal to be listed in Science Citation Index Expanded (SCIE). Since the IF, ImIndex, and AIS values failed to exceed 1.0, Korean journals’ popularity and prestige were confirmed to be minimal. Analyzed journals that were written in English exhibited higher SJR and h-index values than ones written in Korean. WoS’ IF exhibited a correlation with WoS’ 5yrIF, EF, AIS, and Scopus’ SJR, h-index, and ImIndex. Since the ‘popularity and prestige of Korean journals’ have been confirmed to be minimal, steps must be taken to improve this status. Popularity-based indicators have been shown to strongly correlate with prestige-based indicators in Korean science journals. Therefore, there must be a strategic approach taken to improve IF values.
Introduction
In South Korea, the listing of an academic journal in prestigious international citation indices such as Science Citation Index Expanded (SCIE), Social Science Citation Index (SSCI), Arts & Humanities Citation Index (A&HCI), and Scopus not only validates the scientific quality and influence of the journal but also positions the journal advantageously in terms of journal reviews and competition for research funding [1]. Furthermore, published members of the journal enjoy additional advantages in hiring, rehiring, performance evaluation, promotion, and awards. SCIE indexes science and technology journals, SSCI indexes the social sciences, and A&HCI indexes the arts and humanities. All of the above indices are managed by Thomson Reuters. On the other hand, Scopus is managed by Elsevier [1].
Journals indexed in Science Citation Index (SCI) and Scopus cover a very large range of scientific disciplines and exhibit a significant overlap. In principle, however, the two indices are diverse and differentiated, with Scopus larger in size than SCI [2,3]. In the case of South Korea, the number of journals listed in prestigious citation indices such as SCIE, SSCI, A&HCI, and Scopus is woefully small in comparison with major developed countries [4,5]. Even indexed Korean journals do not exhibit significant scientific quality and influence in their respective disciplines.
There exists a myriad of indices that seek to quantitatively reflect academic journals’ levels of scientific quality and influence [6]. Over the past few decades, many indices were developed to gauge individual, academic group, and national contributions to scientific progress through journal publications [7]. Originally suggested as a tool for academic research, citation indices present access points to a given field’s articles, authors, and references to help analyze the research trends of the given field. The aforementioned access points can also be used to evaluate the research achievements of individual scholars [8]. The value of the journals and the articles within the journals is determined by the citation information. For instance, the value of an article in a journal is determined by the number of times it is cited by others and the pertinent journal’s impact factor (IF) [9]. This method relies on the premise that the number of citations received reflects both the quality of the article and the pertinent journal [2]. IF has emerged as a superior alternative to the long, drawn-out process of comprehensive research evaluation [7].
In the case of foreign academic journals, the improvement of the IF value receives significant attention as the IF value of a journal is viewed as the barometer of the journal’s prestige [10]. Thomson Reuters utilizes IF, a standardized value that represents the number of times an article has been cited by others, as the primary standard of selecting outstanding journals since they believe that the IF value indirectly demonstrates an academic journal’s quality [5]. IF is also an important index to gauge the scholastic importance of a given academic journal [11]. However, IF has received some criticism due to the fact that the quality of the citations cannot be measured and an author of multiple articles can cross cite him/herself to increase the IF value. Furthermore, IF has also received criticism for an English language bias [11-13]. As a result, numerous alternative indices using different methodologies have been developed to gauge the quality of academic journals.
Although the number of Korean scientific journals listed in international citation indices has increased significantly, the influence and quality of these journals have been deemed minimal. Therefore, the present research intends to examine the influence of Korean scientific journals and find solutions for future growth by performing a correlation analysis of the indicators of citation indices for Korean scientific journals cross-listed in the Web of Science (WoS) and Scopus.
Methods
Sample citation indices
First, a total of 85 Korean journals listed in WoS as of January 2013, and 132 Korean journals listed in Scopus as of 2011 were extracted. Finally, a total of 62 Korean journals crosslisted in the two indices was selected for analysis.
Citation analysis service refers to the service that uses search words to perform a bibliometric analysis on the journals and citation report refers to a periodically published report that presents the bibliometric information of notable journals, institutions, and countries [8]. From this perspective, WoS and Scopus fall under the category of citation analysis services while Journal Citation Report (JCR) and SCImago belong to the citation report category [8].
Web of Science and Journal Citation Report
Provided by Thomson Reuters, WoS is the most widely used fee-based citation index [14]. WoS analyzes 9,300 academic journals listed in SCIE, SSCI, and A&HCI [15], constructing a citation database and presenting the number of citations received by individual articles [16]. The majority of existing citation services rely on WoS as a model to select a service category, gather the necessary resources, and develop a journal index that applies to the service category [8]. However, new functions and indices have been recently introduced as a result of external researchers’ ideas and experimentations [8].
Scopus and SCImago
Scopus is a citation database managed by Elsevier that indexes 20,500 literary resources (as of 2012), including 19,400 academic journals [17]. While JCR is a fee-based citation report created from WoS data, SCImago is a free-to-use citation report created from Scopus data [8]. SCImago is a group within the University of Grenada’s Consejo Superior de Investigaciones Cientificas (CSIC) that researches information analysis and visualization. This research group uses the Scopus’ database to operate SCImagojr.com and provide scientific indicator values to individual journals and countries [8]. SCImago permits the search and analysis of journals’ scientific indicator values, countries’ scientific indicator values, comparisons among countries, and visualization of the data [8]. It can be said that SCImago provides a similar service to WoS’ JCR. However, SCImago differentiates itself from JCR in that it permits the data to be easily exported to an excel file, and provides a country-level analysis and comparison [8].
Sample indicators of citation indices (citation index indicators)
Citation index indicators that were analyzed include IF, 5-year impact factor (5yrIF), Eigenfactor score (EF), and article influence Score (AIS), all of which are provided by WoS. The indicators analyzed also include SCImago journal rank (SJR), h-index, impact index (ImIndex), all of which are provided by SCImago, for a total of seven indicators.
Impact factor
The representative citation evaluation indicator, IF, is computed by dividing the number of times a journal has been cited by others over the past two years, by the total number of published articles during the same time [6,14].
In the above formula, the published items in the numerator include letters, news items, book reviews, and even errata, but the cited items in the denominator only include the original article and review articles [11].
Five-year impact factor
In contrast to the regular IF, which measures its value based on an interval of 2 years, the 5yrIF is computed using an interval of 5 years [6,14].
Eigenfactor score
If IF is a popularity indicator that computes how many times an article has been cited, EF is an indicator of scientific prestige. The concept not only accounts for the quantity of the citations but the quality by giving more weight to citations from more commonly cited journals than from those less cited [18,19]. In other words, the indicator computes the number of citations an article receives from well-respected journals. The score effectively gauges the relative influence of a given journal in the scientific community compared to that of another journal. EF was developed to supplement the shortcomings of IF and its value is computed in a similar manner to the Google Search Engine’s PageRank algorithm [20,21]. The computation is based on the citation data and number of articles in the past five years and excludes self-citations [22]. The cumulative EF score of all of the articles within the database is 100 and is not standardized by individual journal publishers. As such, ceteris paribus, journals that publish more articles tend to exhibit higher EF scores and these scores allow for interdisciplinary prestige comparisons of academic journals through the normalization of citation patterns across scientific disciplines [6].
Article influence score
AIS is an indicator that measures the relative average influence of a given article within the first five years of its publication. The indicator was combined with EF in 2009 by JCR [23]. The AIS gauges the prestige of a given journal on the basis of citations received by individual articles within the journal. The value of the AIS can be computed by dividing EF by the number of citable items in the journal [24]. AIS measures the average influence of an article within the first five years of its publication [23] and shares a similar concept with IF in that it bases its score on the number of citations received [24]. However, the AIS is fundamentally different from IF in that AIS seeks to gauge the influence of individual articles and weights the score based on the quality of the journal that cites a given article [18,23]. Consequently, receiving citations from commonly cited journals will result in an AIS value that is many times higher than one receiving citations from uncommon or rarely cited journals [18]. Furthermore, by using 5-year intervals and excluding self-citations the analysis will be similar to the EF method [24].
The influence of all of the articles included in the JCR average ought to be 1.0 [23], which indicates that articles with values greater than 1 represent an influence that is greater than the average of all of the articles, while values less than 1 represent an influence less than the average of all of the articles in JCR.
SCImago journal rank
Due to the limitations of IF and limited access to the JCR indicator, efforts have been made to create alternative indicators that are more inclusive and accessible [7]. One of the main results of this effort is the creation of SJR, which is a prestige indicator according to SCImago [23]. SJR is the citation index indicator suggested by SCImago, computing a value that reflects the topic, quality, and reputation of the journal citing a particular literature. Consequently, a citation from a renowned journal is regarded highly and each citation yields a different factor of influence.
SJR uses the citation data from the Scopus database over the past three years and computes the value utilizing Google’s PageRank algorithm, similar to WoS’ EF [7]. However, SJR is considered to be a more inclusive indicator than EF because Scopus contains more journals than WoS and non-English publications such as those found in SCI [25]. In addition, SJR excludes self-citations unlike IF and includes both citable and non-citable items in the denominator of its formula. As a result, SJR reflects the quality of the academic journals better [2] and provides more objective and inclusive information [12]. However, it also draws reliability criticism [6] since citations from top-ranked journals can be weighted far too heavily compared with those from mid- and bottom-ranked journals [26].
H-index
H-index was originally suggested as a means to evaluate the productivity of individual researchers [27], but it can also be used to evaluate the impact of a particular journal. The indicator lists the articles of individual researchers or journals in the order of most to least cited. The h-index of a researcher or a journal is derived from the numerical rank of the article published equal to the number of citations the article receives or the lowest numerical rank that is lower than the number of citations received [27].
Since h-index measures the accumulation of citations received per article, its value increases as time passes and researchers that have published few very exceptional articles are unfairly assigned a low h-index value [6]. In addition, like the other citation index indicators, a journal’s h-index should only be used for comparisons within its scientific discipline; it is inappropriate to make interdisciplinary comparisons with the indicator [28]. H-index also faces such concerns as self-citation, a diminished role of non-citable items, and the intentional abuse created by including review articles. The indicator is disadvantageous for nascent journals [28] and cannot account for numerous citations received that play an important role in determining the scientific prestige of a journal [7].
Impact index
ImIndex is a citation index indicator provided by SCImago that follows the same computational methods as WoS’ IF but instead uses the citation data listed in the Scopus database [29].
Data analysis method
The data collected from the two citation indices (WoS and Scopus) was analyzed using the IBM SPSS Statistics ver. 21.0 (IBM Co., Armonk, NY, USA). The academic discipline, language, founding year, and initial SCIE listing year of the analyzed journals were summarized using descriptive statistics such as frequency, percent, average, and standard deviation. The difference in citation index indicators based on academic discipline, language, and founding year was compared. The correlation among journals’ citation index indicators was confirmed through bivariate Pearson correlation coefficient.
Results
General characteristics of the analyzed journals
The general characteristics of the analyzed journals are presented in Table 1. After classifying the journals’ academic disciplines based on the Korean Citation Index (KCI), it was shown that engineering journals were the most represented at 43.5% and medical journals followed at 30.6%. Engineering and medical journals represented 74.1% of the total journals. Ninety point three percent of the journals were written in English, which was significantly higher than the percentage of journals written in Korean. Among the total analyzed journals, 45 journals had their founding year confirmed through resources such as their homepages. Forty journals had their SCIE initiation year confirmed. Among the 45 journals with confirmed founding years, 40% of the journals were founded between 2001 and 2010, and 35.6% of the journals were founded between 1991 and 2000. Seventy-five point six percent of the journals were founded between 1991 and 2010. With regard to SCIE initiation years, 2008 and 2009 represented 25% and 20% respectively, for a total of 45% of total selected journals. The age of the journals, or the number of years since their founding, was 17.1±12.3 years (range, 2 to 53 years). The average number of years it took for a nascent journal to become listed on SCIE was 8 years, but it deviated widely from 2 to 38 years.
General characteristics of the analyzed journals (n=62)
Descriptive statistics of the journals’ citation index indicators
The descriptive statistics results of the analyzed journal’ citation index indicators are presented in Table 2. Among WoS’ indicators, IF did not exceed the value of 1.0 by exhibiting a value of 0.85±0.56. The AIS was shown to exhibit values of 0.30±0.18. Among SCImago’s indicators, SJR exhibited 0.40±0.17 and h-index exhibited 14.13±10.45.
Descriptive statistics of the journals' citation index indicators (n=62)
Difference in citation index indicators based on journals’ academic discipline and language
After analyzing for differences in citation index indicators based on the journals’ academic discipline and language, it was shown that natural sciences journals has the highest value in IF (0.92±0.60), EF (0.004±0.003), AIS (0.344±0.182), SJR (0.45±0.20), and h-index (21.42±12.873). Medical journals exhibited the highest mean score in IF (0.92±0.48) and 5yrIF (1.24±0.46). Whereas agriculture and fisheries journals exhibited the lowest value in all 7 citation indicators (Table 3). With regard to the language of the journals, among the five citation index indicators where at least two journals written in Korean had reported value of the citation indicator, SJR and h-index of the journals written in English exhibited a higher value than those of the journals written in Korean while EF of the journals written in Korean exhibited a higher value than those of the journals written in English (Table 4).
Citation index indicators based on the academic disciplines of analyzed journals (n=62)
Citation index indicators based on the language of the journals (n=62)
Correlation of citation index indicators with respect to the age of the journals and the journals’ initial SCIE listing year
After analyzing for correlation among the citation index indicators with respect to the age of the journals and initial SCIE listing year, it was shown that the age of the journals exhibited a weak positive correlation with EF (r=0.310) and h-index (r=0.346) (Table 5). SCIE initial listing year exhibited a strong negative correlation with EF (r=-0.665) and h-index (r=-0.617) (Table 5).
Correlation of citation index indicators with respect to the age of the journals and the ’journals’ initial SCIE listing year
Correlation coefficient among indicators of different citation indices
The test results for the correlation among indicators of different citation indices are shown in Table 6. When the correlation between WoS’ indicators and SCImago’ indicators was examined, WoS’ IF exhibited a medium positive correlation with SCImago’s SJR (r=0.429) and h-index (r=0.435). In addition, IF exhibited a strong correlation with ImIndex (r=0.678). WoS’ 5yrIF exhibited a strong positive correlation with SCImago’s ImIndex (r=0.643) and EF exhibited a medium positive correlation with ImIndex (r=0.455). The AIS exhibited a weak positive correlation with SJR (r=0.345) and a medium positive correlation with ImIndex (r=0.597).
Correlation coefficient among indicators of different citation indices (n=62)
Discussion
With the rise of interest in citation indices and citation analysis, much research comparing WoS and Scopus has been published [3]. Furthermore, research comparing different citation indices based on the two databases has also been published [2,3,11,12,23].
Among the journals analyzed in the present research, 85 journals were listed in WoS, 132 journals were listed in Scopus, and 62 journals were cross-listed in both databases. Although the number of Korean journals listed in Scopus and WoS has been steadily increasing, the number still pales in comparison with that of major developed countries [14].
After classifying the journals’ academic disciplines based on the KCI, it was shown that engineering journals were the most represented at 43.5% followed by medical journals at 30.6%. Engineering and medical journals represented 74.1% of the total analyzed journals. In addition, natural science journals comprised 19.4% of the total number of journals. These results varied from the results of Choi [14], who concluded that natural science journals were the most represented (32.9%) among 82 Korean journals listed in SCIE in 2010. The reason for this difference is assumed to be the result of different journal selections. While Choi’s research only analyzed Korean journals listed in SCIE, the present research included journals cross-listed in Scopus and WoS.
Ninety point three percent of the journals were written in English, which demonstrated that despite their Korean origin, journals written in English have a decisive advantage in being listed in WoS or Scopus. The inclination to globalize research findings has already compelled many Korean journals to publish in English. If the IF values are considered to be important, it is critical that Korean journals are published in English so that foreign researchers are able to cite these journals [30]. To publish a journal in English is not a phenomenon observed solely in South Korea; other countries where English is not the primary language have also been pushing to publish scientific journals in English [31]. The English language has benefited significantly from the new era, in gradually penetrating into other cultures in many fields such as music, arts, education, and more recently science [32].
Many scientists in Africa, Asia, Latin America and Europe still publish their work in national journals, often in their mother tongue, which creates the risk that worthwhile insights and results might be ignored, simply because they are not readily accessible to the international scientific community [33]. To overcome this dilemma, several initiatives now aim to strengthen the impact and quality of national journals with the goal of gaining greater international visibility for articles published in a language other than English [33]. Non-English language articles are rarely cited, resulting in a decline of the journal’s IF [34,35]. In fact, the journal IF is more associated with journal language (i.e., English versus non-English), rather than journal country of origin [35]. To publish a journal in English is considered a barometer of globalization and progress of research institutions [36].
Although the age of the analyzed journals varied widely from 2 to 53 years, it was shown that 40% of them were founded between 2001 and 2010. This indicates that just because a journal has been prestigious locally for a long period of time does not give it an advantage in being listed in an international citation index. Instead, a new journal that makes significant strides to conform to the international standards of research publications will be at an advantage in being listed in an international citation index. As such, a journal must not only be published in English to increase its impact and global circulation, but it must also fulfill various quantitative indicators including timely publication, geographically diverse contributors and reviewers, and readability required to be listed in an international citation index.
Although the average length of time it took from the founding of a journal to be listed on SCIE was 8 years, the time deviated widely from 2 to 38 years. With regard to SCIE initiation year, 2008 and 2009 comprised of 25% and 20% of the SCIE listed journals, respectively, for a total of 45%. Such results reflects Thomson Reuters’ implementation of its Local Journal Selection Policy established in 2006, which chooses outstanding journals from an underrepresented region despite the journals’ low IF values. It appears as though Korean journals benefited from this policy in 2008 and 2009 [37].
The IF and SJR values of the analyzed journals did not exceed 1.0. Of course, one of the problems with using indicators that are based on the number of citations received is that citation practices vary greatly among different academic disciplines, leading to a systematic bias in the number of citations received [38]. Yet, even as the present research overlooked the above problem when it computed the IF and SJR values, the results revealed that the IF for Korean journals is far too low. Bollen et al. [39] concluded that usage-based indicators are more powerful prestige-based indicators than the current indicators that are in use. Consequently, IF and SJR are not useful prestige-based indicators, but popularity-based indicators, while EF, AIS, and h-index are prestige-based indicators. However, even the values of prestige-based indicators were not high for the analyzed journals.
In the present research, journals that were published in English exhibited a higher popularity-based SJR and prestige-based h-index value than the journals published in Korean. Such results once again confirmed that publications in English have a significant advantage in the global circulation rate [37,40,41].
After analyzing the correlation among various citation index indicators, the results showed that popularity-based indicators such as IF and 5yrIF were strongly correlated with the prestige-based indicators EF and AIS. It should be noted that there are some claims that prestige does not correspond to popularity-based indicators such as IF and SJR [39]. On the other hand, plenty of research has been done to confirm the results of the present research that popularity-based indicators and prestige-based indicators are correlated. Rousseau and The Stimulate 8 Group [23] used JCR to test the 2004 and 2006 indicator correlations of 77 journals in the following fields: allergy, analytical chemistry, artificial intelligence, automation, business administration, cell biology, civil engineering, ecology, environmental science, immunology, information systems, medicine, neuroscience, ophthalmology, and physics. In the results, IF and EF exhibited a strong correlation of 0.806 in 2004 and 0.827 in 2006. IF and AIS exhibited a strong correlation of 0.895 in 2004 and 0.918 in 2006. Between SJR and EF there was a strong correlation of 0.673 in 2004 and 0.731 in 2006. Between SJR and AIS there was a strong correlation of 0.760 in 2004 and 0.813 in 2006. In all cases, the correlation coefficient increased from 2004 to 2006, demonstrating that there is a strong correlation between popularity and prestige that increases over time. Notably, IF exhibited a stronger correlation with the prestige-based EF and AIS than SJR, leading to the conclusion that IF can also be used as a good predictor for the prestige-based indicators despite its popularity-based characteristic. In summary, the present research supports the claims of Rocha-e-Silva [2] that issues of self-citation and citations from reputable sources cannot alter the conclusions reached by mathematical computations.
Furthermore, WoS’ IF and Scopus’ ImIndex, which have similar computational formulas, have exhibited cross-correlation with all of the indicators that are not part of IF and ImIndex’ citation indices. Such results support previous research efforts [23,42] that state that WoS’ IF, EF, and AIS and Scopus’ SJR and h-index exhibit strong correlations with each other despite their differences in computational formula and citation databases.
Since bibliometric indicators have different computation methods and significance, it is inappropriate to consider only a few particular indicators to evaluate or enhance the quality of a journal. In the case of IF, however, because a strong correlation has been established between IF and prestige-based indicators despite the problems and limitations of the indicator, the fundamental strategy that Korean journals should employ to raise their international prestige is to increase the number of citations received. In the results of the present research, the number of Korean journals listed in international citation indices has been shown to be far too small. Additionally, Korean journals have been shown to exhibit relatively low values in both popularity and prestige-based indicators. Therefore, future articles published in Korean journals must consider raising the quality of the content and pique the academic interest and demands of international readers.
Notes
No potential conflict of interest relevant to this article was reported.