| Home | KCSE | Sitemap | Contact Us |  
Science Editing > Volume 6(2); 2019 > Article
Kim, Yi, Cho, and Huh: How many retracted articles indexed in KoreaMed were cited 1 year after retraction notification



It aimed to investigate how many retracted articles indexed in KoreaMed were cited in both the Scopus and the Korea Medical Citation Index (KoMCI) databases and to investigate whether the frequency of post-retraction citations was different according to the presence of a retraction mark.


Retracted articles from the KoreaMed database were collected on January 28, 2016. Scopus and KoMCI were searched for post-retraction citations, which were defined as citations 1 year after the retraction, excluding retraction-related citations.


The 114 retracted articles were found in KoreaMed. The proportion of retracted articles in KoreaMed, the Korean medical journal database, through January 2016 was 0.04% (114/256,000). On the journal homepage, a retraction mark was present for 49 of the 114 retracted articles. Of the 114 retracted articles, 45 were cited in Scopus 176 times. Of the 176 citations, 109 (of 36 retracted articles) were post-retraction citations. The number of citations in KoMCI, except for citations of retraction notices, was 33 (of 14 retracted articles). Of those citations, the number of post-retraction citations in KoMCI was 14 (of 8 retracted articles). The presence of a retraction mark did not influence post-retraction citations (P>0.05). Post-retraction citations were frequent in the range of 1 to 3 years.


Post-retraction citations that were found in both Scopus and the KoMCI occurred frequently for retracted articles in KoreaMed. Adoption of Crossmark is recommended as one choice to prevent post-retraction citations.


Retraction can be defined as “a mechanism for correcting the literature and alerting readers of publications that contain such seriously flawed or erroneous data [1].” The COPE (Committee on Publication Ethics) retraction guidelines state that “the purpose of retractions is to correct the literature rather than to punish the authors. Nevertheless, most authors take a negative view of retractions and may fear that they will harm their reputation.” Retractions due to research misconduct and honest errors should be distinguished [2]. In Korea, the abstracts of all local (i.e., Korea-based) medical journals are indexed in the KoreaMed database. KoreaMed is an abstract database of medical journals published from Korea launched in 1997. Out of 217,839 articles in KoreaMed published from 1990 to January 2016, the publication type of 111 articles was retraction (0.051%). The reasons for retractions included duplicate publication (57.0%), plagiarism (8.8%), scientific error (4.4%), author dispute (3.5%), and other (5.3%); the reasons were unstated or unclear in 20.2% of cases [3]. In KoreaMed retractions due to honest error accounted for a small proportion (4.4%) of cases [4]. To avoid quoting retracted articles, a notice of retraction should be issued, and a retraction mark must be provided in the HTML or PDF version of the document to notify readers [5]. It has been suggested that post-retraction citations can be divided into two types: citations before announcement of the retraction and citations after the retraction announcement [6].
A previous study investigated how many retracted articles were cited in Medline-listed journals. There were 235 retracted Medline articles from 1996 to 1997, which were cited 2,034 times after the retraction announcement. In an examination of 299 of those citations, 280 citations treated the retracted article positively [7]. For articles published in radiology journals listed in PubMed, 48 retracted original research articles were identified from 1983 to 2015. The mean number of citations of retracted articles was 10.9±17.1 (range, 0 to 94 citations) [5]. However, no previous research has investigated whether this trend differs across countries with different research cultures and levels of information technology. The Korea Medical Citation Index (KoMCI) is a reference citation database that includes references published in Korea of articles in KoreaMed. Therefore, if any KoreaMed articles are cited by journals published in Korea, those citations can be traced [8].
This study aimed to investigate how many retracted articles indexed in KoreaMed were cited in both the Scopus and KoMCI databases and to investigate whether the frequency of post-retraction citations was different according to the presence of a retraction mark. Furthermore, the chronological occurrence of post-retraction citations was counted.


Ethics statement: No informed consent was required because this study was based on a literature database. It does not deal with human or human-originated materials.
Study design: This was a descriptive literature database analysis.
Setting: The research consisted of selecting retracted articles, analyzing their characteristics, and analyzing the frequency of their citations. The selection method for retracted articles was as follows. Retracted articles in KoreaMed were searched by a medical librarian (HMC) on January 28, 2016 (search query: “retraction of publication” (PT) OR “retraction” (TI)). Three reviewers (SH, SYK, and HMC) each examined all of the retraction notices independently. Furthermore, if opinions differed, the decision for inclusion was made based on consensus. The presence of a retraction announcement on the journal homepage or a notice in the retracted article was checked in the PDF and HTML files. In PDF, the presence of a watermark was checked. In HTML, a watermark or notification was checked.
We searched both Scopus and KoMCI for the retracted articles in the source articles’ references through the end of 2017. The citing articles of the retracted articles were searched, limited to “References” in the “Documents” search screen in Scopus. Citations of retracted articles in KoMCI were searched by inputting the bibliographic information of the retracted articles. The count was made only when an explicit citation was identified through hand-searching after the initial search for citations. The total number of citations was counted. Scopus was selected as a citation database since it comprises the greatest number of source titles with reference databases out of all reference- indexing databases at the time of the search by a librarian (HMC). Furthermore, KoMCI was selected because it is the only well-formatted citation database based on KoreaMed journals. Of the 251 KoreaMed source journal titles, 81 (32.3%) were indexed in Scopus at the time of the search. Although the other 170 KoreaMed journal titles were not indexed in Scopus as source titles, they could be searched from the reference documents that Scopus source journals cited. Post-retraction citations were defined in this study as those occurring at least 1 year after the retraction, considering the index time for the notice of retraction and the time to publication after submission. A diagram of the study design and methodology is presented in Fig. 1.
Variables, data source, and bias: The variables were the characteristics of retracted articles in KoreaMed and the frequency of post-retraction citations in Scopus. The data sources were Scopus, KoreaMed, and KoMCI. Retracted articles were counted and distributed according to presence of a notification of retraction. The frequency of post-retraction citations was counted in Scopus and KoMCI chronologically. No reportable bias was present since this was a literature-based study.
Statistical methods: Observational statistical descriptions were employed. The Mann-Whitney U-test (Wilcoxon rank sum) was used to investigate differences according to the presence of a retraction mark. The statistical software package used was DBSTAT ver. 5.0 (DBSTAT Co., Chuncheon, Korea).


At the time of the search, there were 254,000 articles in KoreaMed. Of all articles in KoreaMed, 114 were retracted (0.04%). The results for the characteristics of retracted articles in KoreaMed are presented in Table 1.
Forty-five of the retracted articles were cited in Scopus 176 times (mean±standard deviation [SD] of number of citations of the 45 retracted articles: 3.9±3.7; median number of citations: 3 [range, 1 to 18]). If notices of retraction were removed, the number of citations was 168 from 42 retracted articles (mean±SD of number of citations of the 42 retracted articles: 3.9±3.8; median number of citations: 2 [range, 1 to 18]), including 109 post-retraction citations from 36 articles (mean±SD of citations of the 36 retracted articles: 2.9±3.6; median number of citations: 2 [range, 1 to 10]) (Table 2, Dataset 1). Of the 36 articles with post-retraction citations, a retraction mark was present in 13 articles. There was no difference in the number of post-retraction citations between the articles with and without a retraction mark (P=0.9865) based on the Mann-Whitney U-test. Parametric statistics was not possible because the data did not show a normal distribution, although the average number of post-retraction citations of articles with a retraction mark was 3.23 and that of those without a retraction mark was 2.91.
In KoreaMed, there were 43 citations of 22 retracted articles. If notices of retraction were removed, the number of citations was 33 (of 14 retracted articles), including 14 post-retraction citations of 8 retracted articles (Table 2, Dataset 1).
The chronological occurrence of post-retraction citations is presented in Fig. 2 (Dataset 2). Post-retraction citations were frequent 1 to 3 years after the 1-year period subsequent to the retraction notification. We found that post-retraction citations could occur 10 years after the retraction announcement.
Out of 114 retracted articles, 76 articles (66.7%) were published before 2009. Out of 76 retracted articles, the retraction announcements of 38 articles (50.0%) were published before 2009. Thirty-eight retracted articles were published from 2009.


Key results: Citations of retracted articles indexed in KoreaMed were frequent, and post-retraction citations were also frequent. Additionally, the presence of a retraction mark did not influence post-retraction citations. Out of 254,000 articles in KoreaMed, 114 were retracted (0.05%).
Interpretation: This rate is comparable to what has been reported in previous studies: 0.07% for BioMed Central [9] and 0.03% for Journal of Bone and Joint Surgery (American volume) [10]. In present study, the number of post-retraction citations for the 36 retracted articles in Scopus ranged from 1 to 10, with an average of 3.0 and a median of 2. Furthermore, the number of post-retraction citations for the 8 retracted articles in KoMCI ranged from 1 to 3, with an average of 1.5 and a median of 2.
In Scopus, of the 176 citations, 109 (61.9%) were post-retraction citations. To date, few analyses in the literature have presented post-retraction citation rates. Steen noted that, of the 5,503 citations of 180 retracted articles from the PubMed database between 2000 and 2010, 1,917 were post-retraction citations, with a total rate of 34.8% [11].
The lack of a retraction mark on the HTML version or PDF available from the journal homepage may explain the frequency of post-retraction citations. Of the 114 retracted articles, a retraction mark was not present for 66 (57.9%) articles in the HTML version and for 108 articles (94.7%) in the PDF version. There was no mark of retraction in 65 (57.0%) articles. In other studies, the percentage of retraction marks in PDF or HTML versions was likewise not high. In the radiology field, 37.9% (14/37) of retracted articles had a retraction mark in the PDF version, while 62.2% (23/37) did not. Furthermore, 2 of 13 (15.4%) retracted articles had a retraction mark in the HTML version, while 11 out of 13 (84.6%) of the PDF versions did not [5], and the percentage of articles with a retraction mark was 30% (14/37) in the orthopedic field [10]. In this study, there was no significant difference in post-retraction citations according to the presence of a retraction mark in the PDF/HTML versions.
As shown in Fig. 2, we found that post-retraction citations could persist for more than 10 years after the retraction announcement. The graph does not show the decrease by year because the publication date was not considered in the graph. Bar-Ilan and Halevi [6] demonstrated that post-retraction citations may occur up to 16 years after the announcement.
Decrease of retracted articles after 2009 may be results of enforcement policy which was supported by the publication of the Good publication practice guidelines for medical journals in 2008 by the Korean Association of Medical Journal Editors.
Why are retracted articles frequently cited even after the retraction announcement?: Bar-Ilan and Halevi [6] suggested four reasons: First, the full texts of retracted articles may remain freely available on the web. Second, public or media attention can cause an increased frequency of negative citations. An example was a retracted article that “evoked a public debate regarding the safety of genetically modified (GM) foods”. In this case, researchers cited retracted articles to warn the public about the incorrect information contained in the retracted article [6]. Third, in some cases, data and image manipulation was not considered to be as serious as other violations which suggest that articles retracted for such reasons can continue to be cited. Fourth, if the reason for retraction was text-recycling or duplicate publication, the data themselves are not a problem from an editor’s perspective; therefore, they may continue to be cited after the retraction announcement.
There are two possibilities for the status of post-retraction citations in the Korean medical literature. First, it is possible that the authors did not know that the documents in question had in fact been retracted. Moylan and Kowalczu [9] suggested that readers can be unaware of articles being retracted because retraction notices are seldom cited. As mentioned above, the PDF or HTML versions of retracted articles did not contain retraction marks in a fairly high percentage of cases; therefore, an author can inadvertently cite retracted articles. Authors can also obtain articles through non-publisher websites that provide a substantial number of articles. PubMed Central, educational websites, commercial websites, advocacy websites, institutional repositories, and personal Mendeley libraries are examples of such sites [2,12]. Secondary citations, in which the author does not look at the article itself but cites only the references of other articles, is also a situation in which the author may not know the retraction status of a paper. The second possibility is that the authors knew that the articles had been retracted but cited them for other reasons. In the Korean medical literature, the most common reason for retraction is duplicate publication [3]; therefore, an author may cite a retracted article without any doubt about the scientific validity of the article, which results in a positive citation.
How can we efficiently reduce post-retraction citations?: One method for editors or publishers is the adoption of Crossmark (the “Check for updates” button), which was launched in April 2012 and sends a signal to researchers that publishers are committed to maintaining their scholarly content. It gives scholars the information they need to verify that they are using the most recent and reliable versions of a document. Readers simply click on the Crossmark logos on PDF or HTML documents, and a status box tells them if the document is current or if updates are available [13]. This system was adopted by a journal in Korea in 2013 [14]. In 2016, 105 (43.4%) of the 242 KoreaMed journals had adopted it [15]. However, it might be difficult for researchers to confirm the up-to-date status of articles published before April 2012. If this system is implemented for the HTML and PDF files of all scholarly articles, there may be a drastic decrease in post-retraction citations. Of course, no clear evidence yet exists of the effect of Crossmark in preventing citations of retracted articles. Six years have passed since the Crossmark service was launched for scholarly journals in Korea. Therefore, more concrete findings should be collected on the effects of Crossmark. At present, this system is one of the best solutions for elucidating the present status of articles.
Another solution is to visit the Retraction Watch Database (http://retractiondatabase.org/), on which it is possible to search for retracted articles by inputting relevant information. Another method that researchers can use is to visit the journal homepage to check for any notice of retraction or errata, because PDFs of articles can be distributed through a variety of sources, databases, and repositories.
Why did authors cite retracted articles?: The reason why authors cited retracted articles is difficult to elucidate, because there were no data on this topic. If authors are aware of the present status of retracted articles, they are reluctant to cite them. Furthermore, the editors or manuscript editors usually did not check the status of references during the editorial process, although they always check the references for adherence to the style and format of the journal. Therefore, manuscript editors are recommended to check the references to identify whether they have been retracted. All retracted articles are not scientifically flawed, and there are a variety of reasons for retraction. One reason is duplicate publication, in which case, authors are recommended to cite the first-published article. Although it is up to authors to cite or not to cite retracted articles, caution should be used when citing retracted articles. If a retracted article is cited positively, authors should be notified by the editorial office.
Limitations: There was no recordable limitation to this study.
Conclusion: In this study, we found that the citation frequency of retracted articles from medical journals in Korea was considerable. Citations occurred from not only Scopus journals, but also from KoreaMed journals. If the number of medical journals in Scopus (11,734) and KoreaMed (251) is considered, the number of post-retraction citations from KoreaMed journals (14) is not relatively smaller in comparison with the 109 post-retraction citations from Scopus journals. Retraction announcements did not affect the frequency of post-retraction citations for articles published in Korea; therefore, another measure should be introduced to clarify retraction status. Furthermore, this study provides unique data on post-retraction citations among medical journals from a single country, which was possible owing to the presence of the abstract database (KoreaMed) and the citation database (KoMCI).

Conflict of Interest

Hye-Min Cho serves as an editorial board member of the Science Editing, but has no role in the decision to publish this article. Except for that, no potential conflict of interest relevant to this article was reported.


Data Availability

Dataset is available from the Harvard Dataverse at: https://doi.org/10.7910/DVN/KR55YV.

Dataset 1. Raw data of the retracted articles.

Dataset 2. Raw data of the chronological citation frequency of post-retraction citations of retracted articles in KoreaMed.

Diagram of the research process. KoMCI, Korea Medical Citation Index.

Fig. 1.

Chronological trends of the frequency of post-citation retractions of retracted articles listed in KoreaMed. KoMCI, Korea Medical Citation Index.

Fig. 2.

Table 1.

Characteristics of retracted articles in KoreaMed cited January 28, 2016 (n=114)
Characteristics Number (%)
Retraction mark in PDF
 Yes 6 (5.3)
 No 108 (94.7)
Retraction mark in HTML
 Yes 48 (42.1)
 No 66 (57.9)
Retraction mark total
 No 65 (57.0)
 PDF or HTML 44 (38.6)
 Both 5 (4.4)
No. of articles cited in Scopus
 Yes 45 (39.5)
 No 69 (60.5)
No. of articles cited in Scopus except notice
 Yes 42 (36.8)
 No 72 (63.2)
No. of articles of post-retraction citations in Scopus
 Yes 36 (31.6)
 No 78 (68.4)
No. of articles cited in KoreaMed
 Yes 22 (19.3)
 No 92 (20.7)
No. of articles cited in KoreaMed except notice
 Yes 14 (12.3)
 No 100 (87.7)
No. of articles cited as post-retraction citations in KoreaMed
 Yes 7 (6.1)
 No 107 (93.9)

Table 2.

Frequency of citations of retracted articles in KoreaMed obtained from the Scopus and KoreaMed databases 1 year after the retraction, excluding retraction-related citations
Retracted article Scopus
No. of articles No. of citations No. of articles No. of citations
Cited articles Total 45 176 22 43
Post-retraction citationsa) 36 109 8 14
Non-post-retraction citations 6 59 6 19
Retraction-related citations 3 8 8 10
Non-cited articles 69 0 92 0
Total 114 114

a) One year after the retraction, excluding retraction-related citations.


1. Committee on Publication Ethics. Guidelines for retracting articles [Internet]. London: Committee of Publication Ethics; [cited 2019 Jun 15]. Available from: https://publicationethics.org/resources/guidelines-new/guidelines-retracting-articles

2. Wager E. Why are retractions so difficult? Sci Ed 2015;2:32–4. https://doi.org/10.6087/kcse.34
crossref pdf

3. Huh S, Kim SY, Cho HM. Characteristics of retractions from Korean medical journals in the KoreaMed database: a bibliometric analysis. PLoS One 2016;11:e0163588. https://doi.org/10.1371/journal.pone.0163588
crossref pmid pmc

4. Kim SY, Bae CW, Cho HM, Huh S. Comparison of the patterns of duplicate articles between KoreaMed and PubMed journals published from 2004 to 2009 according to the categories of duplicate publications. Sci Ed 2018;5:44–8. https://doi.org/10.6087/kcse.117
crossref pdf

5. Rosenkrantz AB. Retracted publications within radiology journals. AJR Am J Roentgenol 2016;206:231–5. https://doi.org/10.2214/ajr.115163
crossref pmid

6. Bar-Ilan J, Halevi G. Post retraction citations in context: a case study. Scientometrics 2017;113:547–65. https://doi.org/10.1007/s11192-017-2242-0
crossref pmid pmc pdf

7. Budd JM, Sievert M, Schultz TR. Phenomena of retraction: reasons for retraction and citations to the publications. JAMA 1998;280:296–7. https://doi.org/10.1001/jama.280.3.296
crossref pmid

8. Huh S. Citation analysis of the Korean Journal of Internal Medicine from KoMCI, Web of Science, and Scopus. Korean J Intern Med 2011;26:1–7. https://doi.org/10.3904/kjim.2011.26.1.1
crossref pmid pmc pdf

9. Moylan EC, Kowalczuk MK. Why articles are retracted: a retrospective cross-sectional study of retraction notices at BioMed Central. BMJ Open 2016;6:e012047. https://doi.org/10.1136/bmjopen-2016-012047
crossref pmid pmc

10. Rai R, Sabharwal S. Retracted publications in orthopaedics: prevalence, characteristics, and trends. J Bone Joint Surg Am 2017;99:e44. https://doi.org/2106/JBJS.16.01116
crossref pmid

11. Steen RG. Retractions in the medical literature: how can patients be protected from risk? J Med Ethics 2012;38:228–32. https://doi.org/10.1136/medethics-2011-100184
crossref pmid

12. Davis PM. The persistence of error: a study of retracted articles on the Internet and in personal libraries. J Med Libr Assoc 2012;100:184–9. https://doi.org/10.3163/1536-5050.100.3.008
crossref pmid pmc

13. Lammey R. How to apply CrossMark and FundRef via CrossRef extensible markup language. Sci Ed 2014;1:84–90. https://doi.org/10.6087/kcse.2014.1.84
crossref pdf

14. Huh S. Revision of the instructions to authors to require a structured abstract, digital object identifier of each reference, and author’s voice recording may increase journal access. J Educ Eval Health Prof 2013;10:3. https://doi.org/10.3352/jeehp.2013.10.3
crossref pmid pmc pdf

15. Jeong GH, Huh S. Status of digital standards in Korean medical journals in 2016. Sci Ed 2016;3:100–4. https://doi.org/10.6087/kcse.73
crossref pdf

Editorial Office
The Korea Science & Technology Center 2nd floor,
22 Teheran-ro 7-gil, Gangnam-gu, Seoul 06130, Korea
TEL : +82-2-3420-1390   FAX : +82-2-563-4931   E-mail : kcse@kcse.org
Copyright © Korean Council of Science Editors.           Developed in M2PI