Introduction

Peer review (PR) is the critical appraisal of manuscripts submitted to scholarly journals by independent experts who have sufficient knowledge and expertise on the subject [1]. PR acts as a quality metric that aims to ensure that only studies fulfilling adequate quality standards are published. Despite concerns about its effectiveness, impartiality, and reliability, PR is still accepted as one of the best methods to ensure that published studies are high-quality. Almost all researchers agree that PR is a necessary tool for vetting the scientific literature; nonetheless, many researchers find that the current PR system has several shortcomings.

Through PR, studies are filtered and screened based on the novelty of the idea, rigor of methods, clarity and reproducibility of results, and adequacy of conclusions. PR is applied to assess submitted manuscripts in terms of various aspects. Firstly, PR is conducted to ascertain whether the submitted manuscript adds any new knowledge to the existing literature. New knowledge does not necessarily have to be entirely novel, but a study should at least address a known gap in the literature. Secondly, PR is used to assess the methods used in the study and determine if they are scientifically sound, rigorous, and reproducible. Thirdly, PR is conducted to evaluate the study results and determine if they are appropriate based on the stated methods. Finally, the study conclusions should be derived from the study findings and should be reported according to the level of evidence of the study [2-6]. All reviewers should strive to address all these facets of review. However, in addition to critiquing the study design and content, a statistician may sometimes need to focus on design, methodological rigor, and statistics.

The outcome of PR is used as a basis for rejecting a submitted manuscript, allowing the authors to revise, or accepting a submitted manuscript. The reviewers’ comments can help the authors revise their manuscript to improve its quality. Constructive PR is imperative for improving the manuscript quality, language, and flow, and reviewers may also suggest adding new analyses that augment the scientific value of the study. Hence, the ideal PR system should be not only a quality metric and guarantor of quality of the literature, but also a constructive platform that provides better research output to the scientific community.

Objectives: The aim of this review was to assess the current PR approaches and offer some alternatives to enhance PR.

Methods

Ethics statement: Ethics approval was obtained from the institutional review board of the Faculty of Medicine, Mansoura University (R.21.10.1474). The study was exempted from the requirement to obtain informed consent.

Study design and strategy: This collaborative study was conducted via the #OpenSourceResearch collaborative platform (https://osrc.network/). #OpenSourceResearch is an international independent research organization with a special focus on implementing information technologies and artificial intelligence (AI) in medical research. It aims to encourage innovation in medical research at different stages of the production of scientific knowledge, including the review process.

The study combined a comprehensive literature review on the current PR system with a social media survey of the opinions of various parties on the current PR system. Six independent researchers conducted a literature search on the types of PR, its shortcomings, and methods to improve PR. Electronic databases including PubMed/MEDLINE and Scopus, in addition to Google Scholar, were queried for studies and websites that discussed different aspects of PR of the scientific literature. In addition, a cross-sectional survey of the opinions of readers, authors, reviewers, and editors of journals about the current PR system and how to potentially improve PR was conducted through electronic polls organized via the social media platform Twitter in March 2021.

Outcome of Literature Review

Types of PR

PR is classified according to authors’, reviewers’, and editors’ awareness of each other as single-blind, double-blind, triple-blind, and open PR [7,8]. In single-blind review, the reviewers are aware of the identity of the authors, but they remain anonymous to the submitting authors. The concept of this type of review is to allow reviewers to detect any conflict of interests with the authors that preclude assigning them for review. However, this method can also potentially include inherent reviewer bias towards or against certain institutions and or researchers [9]. It has been stated that while revealing the authors’ identities may be beneficial for renowned authors and high ranked institutions, it might also be disadvantageous for junior researchers, particularly those based in low- and middle-income countries, who may be affected by geographic bias [10].

Double-blind review aims to ensure that the identities of the authors and reviewers are anonymous to each other. It has been suggested that double-blind reviews are less likely to be positive for papers submitted by renowned authors and institutions as compared to single-blind reviews [8,11,12]. However, double-blind reviews may not be totally double-blind, as Haffar et al. [3] found that in 25% to 50% of cases the reviewers can successfully identify the authors of the manuscript when they unintentionally revealed their identities by quoting their previous work in the manuscript. When the handling editor is also unaware of the identity of the authors, this is called triple-blind review, which is less frequently used than single-blind and double-blind review [13].

In contrast to previous models, open PR allows the identity of the authors and the reviewers to be revealed to all participants during and/or after the manuscript review process [14,15]. Open PR is intended to be a more transparent and collaborative approach that enhances communication among the reviewers and authors and potentially minimizes prejudices. As compared to the classic models, proponents of open PR suggest it can be more meticulous and effective [16]. However, practice may be different from theory. van Rooyen et al. [17] randomized consecutive papers that were sent to two reviewers to have the reviewer’s identity either revealed to the authors or remain anonymous, whereas editors and authors were unaware of the intervention. The study documented no impact of open PR on the quality or duration of the review process and found that the reviewers who had their identities revealed still had lower acceptance rates than reviewers with anonymous identities. Moreover, it has been noted that having the authors and reviewers aware of each other’s identity heightened tension and resulted in strong hostility and retaliation among parties [3]. Another concern about open PR is that junior reviewers might be fearful of criticizing or rejecting the work of senior and highly accomplished and renowned authors. Similarly, friends might not appropriately critically judge the work of friends. Furthermore, bias may exist against authors based upon many other factors including reputation, prior work, and their locations.

New PR models

The above classic PR models have been claimed by their critics to have several potential alleged shortcomings such as inconsistency, lack of transparency, bias, inadequate reviews, and conflicts of interest [3]. These possible problems have motivated the search for better quality review models [7]. Other PR models have been introduced to overcome the flaws identified in the classic assessment methods [18].

Pre-PR commenting involves making a formal pre-release comment or discussion on a study submitted for review on a public platform. The advantages of this approach include being fast, transparent, and low-cost; however, a lack of editorial review is its main drawback. Pre-publication PR is PR by invited experts in the relevant field after formal acceptance of the article and before its publication. It has the disadvantage of being a non-transparent process with regards to the basis on which the experts are invited, which may be associated with potential bias [19].

In contrast, post-publication PR entails review of the published articles by expert reviewers in the field who are formally or voluntarily invited. Post-publication review consists of a discussion of published research, independent of any formal PR. This platform allows anyone to publicly contribute without any filtration. Similarly, post-publication commenting (community review) entails evaluation of the articles after publication by experts in the field, by invited officials, and by the common readers. One major flaw of post-publication PR is its openness to every reader, regardless of their qualification or expertise on the topic of the published article. Having unqualified and sometimes biased individuals able to comment freely on published science may have negative consequences. Spam and trolling can be additional threats. Since post-publication PR is still considered PR, it requires time and effort, which many busy scientists may not afford.

Transparent PR involves publishing the reviewers’ reports along with the accepted articles, with or without specifying the identity of the reviewers. Transparency in PR can highlight possible changes made to the manuscript based upon the reviewers’ suggestions [20]. Interactive PR is a dynamic PR platform where the identities of the authors and reviewers are kept anonymous yet there is more direct communication. In this system, responses to the reviewers are quick, and waiting time and misinterpretation of the reviewers’ comments may be eliminated [21]. Hybrid PR combines open public and classic PR to enhance the impact and value of open PR while ensuring compliance to academic standards through single-blind PR per evaluation [22].

Collaborative PR is novel model for PR in which reviewers, editors, and readers agree on a set of revisions through a discussion process. Through this interactive environment, a set of well revised feedback is shared with the authors. Another novel, yet controversial model is portable review [23], in which PR is done in an independent manner without involvement of the journals. In this model, the reviewers’ reports are detached by third-party companies and are transferred along with the manuscript to suitable journals. An example of this model is the article transfer service offered by some journals when a manuscript is rejected and its authors receive an offer to transfer the submission, along with the review reports, to a sister, usually open-access, journal published by the same publisher. This application may help save time in terms of uploading the article to the journal and reduce duplications; however, it can be expensive. Another problem with this model is that many journals and authors do not accept this evaluation, which takes place with a service procurement model. Table 1 summarizes the types of PR systems.

Criticisms Levelled at the Current PR System

Although the current PR system has helped ensure the quality of published articles, it may also have some potential limitations.

Time commitment

Conducting PR in a timely manner without compromising the focus on the integrity of the scientific work and ethics is critical for the effectiveness of the PR system. Several key indicators have been used to evaluate the duration of the multiphase review process. These factors include the immediate rejection time, the reviewer invitation time or ‘invitation phase,’ the duration of the first review round or ‘first response time,’ the revision time taken by authors, the number of review rounds, and the total review duration [24,25]. A summary of the time phases of the PR process is illustrated in Fig 1.

Previous have reports suggested that the PR process is time-consuming and its duration may have even increased in recent decades [26,27]. Delays in the review process may adversely impact the transition of scientific evidence into practice [28]. Although there are considerable variations among different scientific fields, lengthy PR durations can potentially be ascribed to three main factors: inefficient editorial processes, delay in manuscript revision by authors, and importantly the timely availability of appropriate reviewers [24]. In a recent comprehensive study by Huisman and Smits [25], the time until desk rejection was more than 2 weeks for almost one-third of journals; such a long time until immediate rejection may point to an inefficient editorial process. Consistent with this finding, the journal’s impact factor was found to be inversely correlated with the immediate rejection time, the first response time, and the total review duration [25]. It is perceived that high-ranked journals generally have more resources at their disposal, and thus have more efficient and quicker manuscript handling at the editorial office.

If the manuscript passes the initial editorial evaluation, the role of reviewers remains the most important determinant of the PR duration, and may significantly influence different phases of the review process and outcome. It has been noted that reviewers who are personally known to the editor tend to be more likely to accept review invitations than other reviewers with no previous personal connection (89% vs. 33%) [28]. Reviewers may also prolong the first response time if they are overwhelmed by other professional commitments or have no spare time to invest in PR [28,29]. Moreover, they may extend the author’s revision time by providing contradictory or unclear comments or by focusing on trivia coercing authors into unnecessary revisions, which may in turn increase the number of review rounds. On the contrary, papers receiving more consistent and favorable reviews tend to be associated with a shorter author’s revision time and total review duration [30]. However, it is important to emphasize that quicker reviews do not necessarily entail higher-quality, more detailed, or better-focused review reports that ensure publication of good-quality articles.

The impact of the type of PR on review time has also been evaluated. Open PR is associated with a comparable review duration to the classic PR system [17]. Transparent PR was associated with a longer time when the reviewers were informed that their signed reports would be available to readers; however, it did not seem to have a comparably discernible effect on the total review duration [31]. Thus, unfortunately, none of the alternative proposed PR methods seem to offer any improvements in the duration of the review process.

Inconsistency

Perhaps the most frequent criticism of the current PR system is the poor inter-reviewer agreement on the same manuscript, also referred to as inconsistency [32]. Inconsistency has been theorized to have at times been associated with unethical practices, especially in the widely adopted single-blind review system [33]. According to recent reports, inconsistent reviews were provided by independent reviewers in 25% to 55% of manuscript submissions [34-36]. These figures were shown to be higher for recommendations concerning rejection than those for acceptance [36,37]. A study examined the reviewers’ recommendation of ‘reject’ versus ‘accept/revise’ in an internal medicine journal and found that level of the inter-rater agreement on recommendation was barely beyond chance, yet the editors nonetheless placed considerable weight on reviewers’ recommendations. This reinforces the fact that recommendations rendered by reviewers, particularly those favoring rejection, markedly influence publication decisions [38,39].

One major reason for such inconsistency could potentially be variation in expertise among reviewers. Indeed, it was shown that similar research productivity and experience of reviewers in a particular subject area, as measured by the number of indexed peer-reviewed publications, significantly increased the level of inter-reviewer agreement [36]. One of the jobs of journal editors and editors-in-chief is to ensure that reviewers with sufficient expertise are requested to review submissions. It is also the job of these journal leaders to regularly evaluate the submitted reviews to actively optimize reviewer selection and assignments.

Further, reviewers are often provided with limited written instructions and are not required to undergo training before submitting reviews [40], resulting in a lack of uniformity in assessing scientific articles and hence poor reviewer agreement regardless of the level of expertise. In particular, different reviewers may value some article traits such as timeliness, the quality of writing, and the originality of the material higher than others. To mitigate against this limitation, some journals have created reviewer training sessions, which have been very popular with reviewers and authors. In addition, the Publons organization has offered a PR training course for early-career researchers [41].

Potential conflict of interest with authors is another theorized source of inconsistency that could possibly bias judgement made by reviewers in favor of or against a manuscript that is not receiving adequate attention. Currently, fewer than half of biomedical journals require reviewers and editors to disclose conflicts of interest [42]. Potential conflicts of interest for reviewers may arise from financial interests related to the research under review and personal or professional relationships with authors. Other reviewer biases that may contribute to inconsistency include biases towards positive research results, theoretical support or paradigm confirmation, and personal attributes of authors such as gender and prestige of institutional affiliation [43,44].

Biases

Since PR is conducted by humans, by design it cannot be totally free of bias. The single-blind PR system, which accounts for about 85% of all annually conducted PRs [45], is particularly notorious for its alleged association with implicit or explicit bias. Single-blind PR has the benefit of enabling reviewers to give an honest assessment since their identity is anonymous and thus can make better independent decisions. However, this type of PR has come under increasing criticism, mainly for a theoretical lack of accountability [3,46]. Indeed, this system enables unscrupulous reviewers to act on biases based on the authors’ information and/or institutional affiliation and either underrate the ideas with which they disagree or promote the work of like-minded researchers in an inappropriate manner [47,48]. Moreover, there may exist an implicit conflict of interest between the reviewer or editor and the authors. Driven by inter-group competition or intention to plagiarize the authors’ work, a peer reviewer who is working in the same field could provide unjustified negative recommendations, in order to block or slow down the publication, and the authors are often left without recourse [49]. Concerns regarding biases due to reviewers’ and editors’ financial interests have also been raised [42]; nevertheless, the impact of this type of bias on PR outcomes has not been studied yet. Eventually, biases associated with single-blind PR may result in fewer recommendations for publication [50], and may undermine the authors’ trust in the PR process. Again, editors and editors-in-chief must be vigilant in their search for and elimination of any and all such possible behavior.

Lack of transparency

Another hypothesized limitation of the current PR system, particularly single-blind review, is a lack of transparency. Some reviewers may recommend rejection of an article with unjustified comments, and sometimes with no comments at all. This behavior could be due to personal grudges between the reviewer and author, or a reviewer may just decide to be biased or overly critical because his/her identity is not known [51]. In these cases it is incumbent upon the editor or editor-in-chief to find alternative reviewers and/or make decisions themselves based upon the other submitted reviews. In open PR, reviewers may find it difficult to reject a manuscript for undeclared reasons as they may be prompted to explain why. However, unjustified rejection by reviewers can still occur with open PR; therefore, the editor should require the reviewers to justify their recommendations and decisions for both open and blind PR. Similarly, in an open review, the reviewers cannot be overly critical as they will be prompted to explain the merit of their remarks. Conversely, in single-blind PR the reviewers tend to focus more on the negative aspects of the work rather than providing constructive comments [52]. In contrast, open review is more transparent, and the reviewer is more focused on the positive aspects of the work and improving it. This renders the review process more useful to the author.

One of the prominent issues with the current PR system is that reviewers are usually assigned to assess the scientific merit and quality of the manuscript; however, they should also assess the possibility of research misconduct such as plagiarism, fabrication, and falsification. It can be argued that specialized software will detect most cases of plagiarism; however, while such software is available to editors and publishers, it may fail to detect some cases of misconduct. Unfortunately, specialized software may not find every breach of ethics or research conduct, particularly mirror-copied images and tables and plagiarism by translation, which reviewers may be able to detect. Furthermore, public PR, whether pre- or post-publication, may also serve as a gatekeeper against this problem because with a larger PR audience such problems with research integrity may be easier to detect and annotate.

Possible Alternatives to Enhance the Current PR System

In order to optimize the quality and rigor of PR, several suggestions have been proposed regarding the PR method and platform, reviewer selection and recruitment, managing PR reports, and ensuring the integrity and impartiality of the PR process.

Double-blind PR is potentially one of the best methods to eliminate or at least minimize bias towards or against the authors. Knowing the identity of the authors may positively or negatively impact the reviewer’s impression based on the authors’ reputation, gender, and institutional affiliation. This behavior was demonstrated in a study showing that double-blind PR, instead of single-blind PR, improved the representation of female authors, with a significant increase in female first-authored papers [53]. Furthermore, there is accumulating evidence that in single-blind PR the manuscript acceptance rates may be higher for authors familiar to the reviewers, prominent authors in their fields, and authors from prestigious universities and institutions [54]. A strong geographic bias has also been observed, as native English-speaking authors from the United Kingdom, North America, Australia, and New Zealand were more likely to have their work accepted for publication than non-native authors from other countries. This apparent bias may stem from the disadvantage of the suboptimal linguistic quality of manuscripts submitted by non-native English speakers or lack of access to the relevant literature [55]. The major critique of double-blind PR is that in small research sectors the reviewer can make an educated guess regarding the authors’ identity when they try to make a point in their research and reference their own previous research [3]. While this may be an exception in most cases, in order to ensure totally blind PR, it would be helpful to require anonymous self-citations by authors and declarations of conflicts of interest by reviewers.

In addition to the double-blind PR system, having an interactive platform may expedite the process and eliminate time wasted in the routine submission/revision/resubmission process [19]. Having an interactive platform where the authors and reviewers can mutually discuss the reviewers’ comments and their applicability and usefulness may subsequently yield better research output by clarifying any vague points and misinterpretations. This platform has been applied by some journals [56] that have an interactive review forum in which the authors read the review reports and take part in an online discussion with reviewers while the identity of both parties is kept anonymous. Although the idea does seem compelling, there is not enough data to support its routine application. Moreover, it has been assumed that many traditional reviewers would not welcome the idea of interacting directly with the authors, either because they do not want extensive discussions or simply because of a lack of time.

Although, the problems of PR are usually associated with reviewers, editors and publishers may also contribute to these issues. Editors may find it particularly difficult to recruit expert reviewers on the subject in a timely manner, and may inadvertently assign unqualified reviewers, thereby increasing the overall PR time and decreasing the quality of PR. A novel solution for this issue is the use of AI for reviewer selection. For example, the UNSILO (https://unsilo.ai) AI platform uses a concept engine that recognizes hundreds of concepts which are key phrases that distinguish articles from each other, ranks these concepts in order of relevance to the manuscript submitted, and matches the results with millions of articles and abstracts in the PubMed/MEDLINE database. The platform then identifies the authors of the matching publications and ranks them by relevance, presenting the top 20 for the editorial office to review [57]. The use of AI to recruit reviewers has been tried by some publishers/journals; although the experience remains preliminary and limited, it may be an option for finding reviewers.

Another possible way to improve the process of reviewers’ recruitment is providing some sort of incentive or compensation for their efforts and time. These incentives can stake many forms, including honoraria, waivers of article processing charges, recognition in dedicated services such as Publons, issuing certificates of recognition, and providing continuing medical education points. While providing financial compensation to the reviewers may seem suitable, an experimental study found that offering material incentives to the reviewers may undermine the moral motives that guide the reviewer’s behavior [58]. Conversely, it has been suggested to avoid overwhelming reviewers by employing a strict editorial filtration policy to avoid sending reviewers manuscripts that are less likely to be published. Rejections without PR can be decisions made by editors-in-chief and or editors, which can lessen the burden placed upon reviewers. Similarly, it has been suggested that upon minor revisions the editors can check the authors’ compliance with the review comments and render a final decision, rather than sending the revised manuscript back to the reviewers.

One method to minimize inconsistency among different reviewers is to provide pre-set templates to comment on each section of the manuscript and ensure objectivity and clarity of their remarks. Typical PR templates ask the reviewers to rate the manuscripts in terms of novelty, scientific rigor, relevance to the field, ethical compliance, use of appropriate statistics, adequacy of table and figure numbers, and the validity of the conclusions. Equally, offering PR training courses and workshops helps peer reviewers to understand the principles and functions of PR and empower them to write solid, actionable PR reports [59].

Finally, publishing the PR reports along with the accepted articles can play an important role in improving the process. Knowing that their reports would be published along with the article, reviewers may choose to avoid personal and inappropriate comments, proofread their reports to ensure clarity, and provide a professional, step-wise appraisal of the manuscripts. On another note, publishing PR reports can benefit the readers and other authors by drawing attention to the main points of critique and weakness in each published article, which may enable them to address and avoid them in their future work. Indeed, there is a growing trend among major journals to publish the PR reports together with the published articles [60]. The suggestions to improve the current PR system are summarized in Table 2.

Opinions of Social Media Users

The #OpenSourceResearch collaboration uses the ‘wisdom of the crowd’ in research, as demonstrated in recently published papers [61,62]. The platform enabled the conduction of a cross-sectional survey of the opinions of the surgical community on the current PR system. Four polls regarding to the preferred type of PR and possible alternative approaches were introduced to the surgical community on the social media platform Twitter. Polls were anonymized to ensure the confidentiality of participants’ data. The #OpenSourceResearch account distributed the Twitter polls among members of the surgical community that not only included authors and readers, but also included reviewers and editors of surgical journals. The main purpose of the polls was to investigate authors’ and reviewers’ opinions about the current system of PR. The Twitter polls were not restricted to authors of papers; instead, the whole surgical community was invited and many of the authors who participated and engaged in conversations about the polls were also peer reviewers. Therefore, the opinions presented may reflect the combined perspectives of both authors and reviewers.

The first poll asked the participants about their personal preferences for the PR system; two-thirds of the 158 participants preferred either the double or triple-blind system. Interestingly, the single-blind system, which is by far the most commonly used one, ranked last, as fewer than 10% of participants chose this option.

The second poll asked the participants to select one of the new PR models that they think might “improve” PR. More than half of 60 participants chose the interactive PR system, perhaps because they felt it would be more practical and time-saving. One-quarter of users chose transparent PR, presumably because they thought that publishing PR reports with the articles would make reviewers keener to produce clearer and more consistent and organized reports.

The third poll asked about the most prominent limitations of the current PR system. Out of 102 participants, 41% chose the inconsistency of review reports, whereas 25% considered the time-consuming nature of PR as its major potential flaw. The final poll asked the participants what they would suggest to enhance the PR system. Of 54 participants, 33% chose providing the reviewers with incentives of any form, whereas 31% and 24% suggested that interactive and double-blind systems would be good means for improving PR. Fig. 2 illustrates screenshots of the four Twitter polls used to generate these data.

Limitations: The present study is limited by being mainly oriented towards surgery, as most authors were surgeons who are active on Twitter. We also cannot assume that the results would be identical among surgeons who are not active on Twitter.

Conclusion

PR is an integral component of scientific publishing and plays an essential role in vetting of the scientific literature. The current PR system has several postulated limitations including time commitment, lack of consistency, possible conflicts of interest, potential biases, and lack of transparency. Possible options to improve the PR system include applying doubleblind, interactive PR platforms, expediting reviewers’ recruitment by using AI or providing incentives to reviewers, eliminating inconsistency by offering PR training courses and preset review templates, and publishing the PR reports along with each accepted article. In the interim it is the job of editors to ensure the consistent, timely delivery to authors of fair, unbiased, balanced, constructive, and detailed reviews. At the present time, editors and editors-in-chief can best serve authors, reviewers, and readers by providing the authors with detailed, constructive comments in addition to the individual points and critiques raised by the reviewers.

Notes

Conflict of Interest

Steven D. Wexner is an editor-in-chief of Surgery (eISSN 1532-7361) and an editor of Colorectal Disease (eISSN 1463-1318) and Techniques in Coloproctology (eISSN 1128-045X). Sameh Hany Emile and Hossam Elfeki are editors of BMC Surgery (eISSN 1471-2482) and members of the editorial advisory board of Colorectal Disease. Otherwise, no potential conflict of interest relevant to this article was reported.

Funding

The authors received no financial support for this article.

Acknowledgements

We would like to thank Mr Neil Smart, Consultant Colorectal Surgeon and editor-in-chief of Colorectal Disease, for his comments and remarks on the final draft of this manuscript.

Fig. 1.

Time phases of peer review.

Fig. 2.

Survey results via Twitter on the peer review system. (A) Preferred type of peer review system and suggested model to improve the process. (B) Most prominent limitation of the current system and suggestions to improve.

References

• 1. Gerwing TG, Allen Gerwing AM, Avery-Gomm S, Choi CY, Clements JC, Rash JA. Quantifying professionalism in peer review. Res Integr Peer Rev 2020;5:9. https://doi.org/10.1186/s41073-020-00096-x. ArticlePubMedPMC
• 2. Kelly J, Sadeghieh T, Adeli K. Peer review in scientific publications: benefits, critiques, & a survival guide. EJIFCC 2014;25:227-43.PubMedPMC
• 3. Haffar S, Bazerbachi F, Murad MH. Peer review bias: a critical review. Mayo Clin Proc 2019;94:670-6.https://doi.org/10.1016/j.mayocp.2018.09.004. ArticlePubMed
• 4. Patel J, Pierce M, Boughton SL, Baldeweg SE. Do peer review models affect clinicians’ trust in journals? A survey of junior doctors. Res Integr Peer Rev 2017;2:11. https://doi.org/10.1186/s41073-017-0029-8. ArticlePubMedPMC
• 5. Ross-Hellauer T. What is open peer review? A systematic review. F1000Res 2017;6:588. https://doi.org/10.12688/f1000research.11369.2. ArticlePubMedPMC
• 6. Stroebe W, Postmes T, Spears R. Scientific misconduct and the myth of self-correction in science. Perspect Psychol Sci 2012;7:670-88.https://doi.org/10.1177/1745691612460687. ArticlePubMed
• 7. Barroga E. Innovative strategies for peer review. J Korean Med Sci 2020;35:e138. https://doi.org/10.3346/jkms.2020.35.e138. ArticlePubMedPMC
• 8. O’Connor EE, Cousar M, Lentini JA, Castillo M, Halm K, Zeffiro TA. Efficacy of double-blind peer review in an imaging subspecialty journal. AJNR Am J Neuroradiol 2017;38:230-5.https://doi.org/10.3174/ajnr.A5017. ArticlePubMedPMC
• 9. Krinsky G. How to avoid “unblinding” the peer review process. AJR Am J Roentgenol 1999;172:1474. https://doi.org/10.2214/ajr.172.6.10350273. ArticlePubMed
• 10. Skopec M, Issa H, Reed J, Harris M. The role of geographic bias in knowledge diffusion: a systematic review and narrative synthesis. Res Integr Peer Rev 2020;5:2. https://doi.org/1186/s41073-019-0088-0. ArticlePubMedPMC
• 11. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci USA 2017;114:12708-13.https://doi.org/10.1073/pnas.1707323114. ArticlePubMedPMC
• 12. O’Connor EE, Cousar M, Lentini JA, Castillo M, Halm K, Zeffiro TA. Efficacy of double-blind peer review in an imaging subspecialty journal. AJNR Am J Neuroradiol 2017;38:230-5.https://doi.org/10.3174/ajnr.A5017. ArticlePubMedPMC
• 13. Kwee TC, Adams HJ, Kwee RM. Peer review practices by medical imaging journals. Insights Imaging 2020;11:125. https://doi.org/10.1186/s13244-020-00921-3. ArticlePubMedPMC
• 14. DePellegrin TA, Johnston M. Opening up peer review. Genetics 2020;216:619-20.https://doi.org/10.1534/genetics.120.303727. ArticlePubMedPMC
• 15. Ware M. Publishing Research Consortium peer review survey 2015. Bristol: Mark Ware Consulting; 2016.
• 16. Pros and cons of open peer review. Nat Neurosci 1999;2:197-8.https://doi.org/10.1038/6295. ArticlePubMedPDF
• 17. van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ 1999;318:23-7.https://doi.org/10.1136/bmj.318.7175.23. ArticlePubMedPMC
• 18. Tennant JP, Dugan JM, Graziotin D, et al. A multi-disciplinary perspective on emergent and future innovations in peer review. F1000 Res 2017;6:1151. https://doi.org/10.12688/f1000research.12037.3. ArticlePubMedPMC
• 19. Mahmic-Kaknjo M, Utrobicic A, Marusic A. Motivations for performing scholarly prepublication peer review: a scoping review. Account Res 2021;28:297-329.https://doi.org/10.1080/08989621.2020.1822170. ArticlePubMed
• 20. Klebel T, Reichmann S, Polka J, et al. Peer review and preprint policies are unclear at most major journals. PLoS One 2020;15:e0239518. https://doi.org/10.1371/journal.pone.0239518. ArticlePubMedPMC
• 21. Emile SH. Interactive platform for peer review: a proposal to improve the current peer review system. World J Clin Cases 2021;9:1247-50.https://doi.org/10.12998/wjcc.v9.i6.1247. ArticlePubMedPMC
• 22. Mandernach BJ, Holbeck R, Cross T. Hybrid review: taking SoTL beyond traditional peer review for journal publication. J Electron Publ 2015;18:https://doi.org/10.3998/3336451.0018.202. Article
• 23. Bell GP, Kvajo M. Tackling waste in publishing through portable peer review. BMC Biol 2018;16:146. https://doi.org/10.1186/s12915-018-0619-z. ArticlePubMedPMC
• 24. Mrowinski MJ, Fronczak A, Fronczak P, Nedic O, Ausloos M. Review time in peer review: quantitative analysis and modelling of editorial workflows. Scientometrics 2016;107:271-86.https://doi.org/10.1007/s11192-016-1871-z. ArticlePubMedPMC
• 25. Huisman J, Smits J. Duration and quality of the peer review process: the author’s perspective. Scientometrics 2017;113:633-50.https://doi.org/10.1007/s11192-017-2310-5. ArticlePubMedPMC
• 26. Mulligan A, Hall L, Raphael E. Peer review in a changing world: an international study measuring the attitudes of researchers. J Am Soc Inf Sci Technol 2013;64:132-61.https://doi.org/10.1002/asi.22798. Article
• 27. Azar OH. The slowdown in first-response times of economics journals: can it be beneficial? Econ Inq 2007;45:179-87.https://doi.org/10.1111/j.1465-7295.2006.00032.x. Article
• 28. Nguyen VM, Haddaway NR, Gutowsky LF, et al. How long is too long in contemporary peer review? Perspectives from authors publishing in conservation biology journals. PLoS One 2015;10:e0132557. https://doi.org/10.1371/journal.pone.0132557. ArticlePubMedPMC
• 29. Kumar MN. Review of the ethics and etiquettes of time management of manuscript peer review. J Acad Ethics 2014;12:333-46.https://doi.org/10.1007/s10805-014-9220-4. Article
• 30. Luft HS. From the editors: perspectives on turnaround time. Health Serv Res 2004;39:1-6.https://doi.org/10.1111/j.1475-6773.2004.00211.x. ArticlePubMedPMC
• 31. van Rooyen S, Delamothe T, Evans SJ. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ; 2010 341:c5729. https://doi.org/10.1136/bmj.c5729. ArticlePubMedPMC
• 32. Marsh HW, Bond NW, Jayasinghe UW. Peer review process: assessments by applicant-nominated referees are biased, inflated, unreliable and invalid. Aust Psychol 2007;42:33-8.https://doi.org/10.1080/00050060600823275. Article
• 33. Resnik DB, Gutierrez-Ford C, Peddada S. Perceptions of ethical problems with scientific journal peer review: an exploratory study. Sci Eng Ethics 2008;14:305-10.https://doi.org/10.1007/s11948-008-9059-4. ArticlePubMedPMC
• 34. Weiner BK, Weiner JP, Smith HE. Spine journals: is reviewer agreement on publication recommendations greater than would be expected by chance? Spine J 2010;10:209-11.https://doi.org/10.1016/j.spinee.2009.12.003. ArticlePubMed
• 35. Kwon JY, Gonzalez T, Miller C, et al. Assessment of variables that influence agreement between reviewers for Foot & Ankle International. Foot Ankle Surg 2020;26:573-9.https://doi.org/10.1016/j.fas.2019.07.007. ArticlePubMed
• 36. Baethge C, Franklin J, Mertens S. Substantial agreement of referee recommendations at a general medical journal: a peer review evaluation at Deutsches Arzteblatt International. PLoS One 2013;8:e61401. https://doi.org/10.1371/journal.pone.0061401. ArticlePubMedPMC
• 37. Rothwell PM, Martyn CN. Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain 2000;123:1964-9.https://doi.org/10.1093/brain/123.9.1964. ArticlePubMed
• 38. Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial peer reviewers’ recommendations at a general medical journal: are they reliable and do editors care? PLoS One 2010;5:e10072. https://doi.org/10.1371/journal.pone.0010072. ArticlePubMedPMC
• 39. Vintzileos AM, Ananth CV, Odibo AO, Chauhan SP, Smulian JC, Oyelese Y. The relationship between a reviewer’s recommendation and editorial decision of manuscripts submitted for publication in obstetrics. Am J Obstet Gynecol 2014;211:703.e1-5.https://doi.org/10.1016/j.ajog.2014.06.053. ArticlePubMed
• 40. Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS One 2012;7:e35621. https://doi.org/10.1371/journal.pone.0035621. ArticlePubMedPMC
• 41. Publons academy course overview [Internet]. London: Clarivate; [cited 2021 Apr 16]. Available from: https://publons.com/community/academy/course-overview. Article
• 42. Resnik DB, Elmore SA. Conflict of interest in journal peer review. Toxicol Pathol 2018;46:112-4.https://doi.org/10.1177/0192623318754792. ArticlePubMedPMC
• 43. Walker R, Barros B, Conejo R, Neumann K, Telefont M. Personal attributes of authors and reviewers, social bias and the outcomes of peer review: a case study. F1000Res 2015;4:21. https://doi.org/10.12688/f1000research.6012.2. ArticlePubMedPMC
• 44. Atkinson DR, Furlong MJ, Wampold BE. Statistical significance, reviewer evaluations, and the scientific process: is there a (statistically) significant relationship? J Couns Psychol 1982;29:189-94.https://doi.org/10.1037/0022-0167.29.2.189. Article
• 45. Ware M. Peer review: benefits, perceptions, and alternatives [Internet]. London: Publishing Research Consortium; 2008 [cited 2021 Apr 16]. Available from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.214.9676&rep=rep1&type=pdf. Article
• 46. Shaw DM. Blinded by the light: anonymization should be used in peer review to prevent bias, not protect referees. EMBO Rep 2015;16:894-7.https://doi.org/10.15252/embr.201540943. ArticlePubMedPMC
• 47. Severin A, Martins J, Heyard R, Delavy F, Jorstad A, Egger M. Gender and other potential biases in peer review: cross-sectional analysis of 38,250 external peer review reports. BMJ Open 2020;10:e035058. https://doi.org/10.1136/bmjopen-2019-035058. ArticlePubMedPMC
• 48. John LK, Loewenstein G, Marder A, Callaham ML. Effect of revealing authors’ conflicts of interests in peer review: randomized controlled trial. BMJ; 2019 367:l5896. https://doi.org/10.1136/bmj.l5896. ArticlePubMedPMC
• 49. Al-Khatib A, Teixeira da Silva JA. Is biomedical research protected from predatory reviewers? Sci Eng Ethics 2019;25:293-321.https://doi.org/10.1007/s11948-017-9964-5. ArticlePubMedPMC
• 50. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single-versus double-blind peer review. Proc Natl Acad Sci USA 2017;114:12708-13.https://doi.org/10.1073/pnas.1707323114. ArticlePubMedPMC
• 51. Manchikanti L, Kaye AD, Boswell MV, Hirsch JA. Medical journal peer review: process and bias. Pain Physician 2015;18:E1-14.ArticlePubMed
• 52. Oleinik A. Conflict(s) of interest in peer review: its origins and possible solutions. Sci Eng Ethics 2014;20:55-75.https://doi.org/10.1007/s11948-012-9426-z. ArticlePubMed
• 53. Budden AE, Tregenza T, Aarssen LW, Koricheva J, Leimu R, Lortie CJ. Double-blind review favours increased representation of female authors. Trends Ecol Evol 2008;23:4-6.https://doi.org/10.1016/j.tree.2007.07.008. ArticlePubMed
• 54. Cox AR, Montgomerie R. The cases for and against double-blind reviews. PeerJ 2019;7:e6702. https://doi.org/10.7717/peerj.6702. ArticlePubMedPMC
• 55. Tregenza T. Gender bias in the refereeing process? Trends Ecol Evol 2002;17:349-50.https://doi.org/10.1016/S0169-5347(02)02545-4. Article
• 56. Pain E. How interactive peer review works [Internet]. Washington, DC: American Association for the Advancement of Science; 2013 [cited 2021 Mar 7]. Available from: https://www.sciencemag.org/careers/2013/04/how-interactive-peer-review-works.
• 57. Upshall M. Using artificial intelligence (AI) in peer review [Internet]. Mumbai: Cactus Communications; 2020 [cited 2021 Mar 7]. Available from: https://www.editage.com/insights/using-artificial-intelligence-ai-in-peer-review. Article
• 58. Squazzoni F, Bravo G, Takacs K. Does incentive provision increase the quality of peer review? An experimental study. Res Policy 2013;42:287-94.https://doi.org/10.1016/j.respol.2012.04.014. Article
• 59. Wilkinson J. How to write a peer review: practical templates, expert examples, and free training courses [Internet]. London: Clarivate; [cited 2021 Mar 7]. Available from: https://clarivate.com/blog/how-to-write-a-peer-review-practical-templates-expert-examples-and-free-training-courses/.
• 60. Nature will publish peer review reports as a trial. Nature; 2020 578:8. https://doi.org/10.1038/d41586-020-00309-9. ArticlePubMedPDF
• 61. Yalcinkaya A, El-Hussuna A, OpenSourceResearch Collaborating Group. #OpenSourceResearch: a novel medical research technique/style in terms of accessibility and way of work. Colorectal Dis 2021;23:321-2.https://doi.org/10.1111/codi.15392. ArticlePubMed
• 62. El-Hussuna A, Rubio-Perez I, Millan M, et al. Patient-reported outcome measures in colorectal surgery: construction of core measures using open-source research method. Surg Innov 2021;28:560-6.https://doi.org/10.1177/1553350621998871. ArticlePubMed

Figure & Data

References

Citations

Citations to this article as recorded by