Introduction

Background

Some strategies that journals employ to attract quality reviewers include requesting authors to suggest potential candidates for their manuscripts, inviting colleagues to participate in the review process, searching for reviewers via academic platforms and social media, and reaching out to users already registered in the journal’s system. These approaches are considered best practices that editors should adopt to improve the quality of peer review. Frequently, editors must tailor these strategies to fit the specific platform used by the journal. For instance, within the Open Journal Systems (OJS), editors might need to informally contact potential reviewers (e.g., through academic social media) and encourage them to register as users in OJS.

Our proposed approach builds upon these advancements by utilizing Scopus data to identify potential reviewers among existing users in the OJS. This database is accessible via the application programming interface (API), which offers details on authors and their publications. By utilizing this API, researchers can access information about an author’s work, track their research output, and explore opportunities for collaboration within the scientific community.

Objectives

This study proposes utilizing existing user data within OJS to identify potential reviewers. It aims to maintain the quality of publications by offering recommendations for the most suitable reviewers.

Methods

Ethics statement

This study did not involve human subjects; therefore, neither institutional review board approval nor informed consent was necessary. We required all developers involved in this system to agree to a data confidentiality statement to ensure the maintenance of data confidentiality.

Study design

This research was conducted using the design science research methodology (DSRM) paradigm [1,2]. We chose DSRM for its rigorous process of designing software programs to address observed problems.

Study setting

We employed term frequency-inverse document frequency (TF-IDF) and cosine similarity to generate relevant reviewer recommendations. The evaluation assessed precision, mean average precision (MAP), and mean reciprocal rank (MRR). The journal editorial team participated in validating the recommendation results, demonstrating the effectiveness of the proposed reviewer recommendation system.

Dataset

The data utilized for this research comprised user data registered within a journal, specifically targeting potential reviewers. For our case study, we selected data from the Journal of Information System Engineering and Business Intelligence (JISEBI). We accessed 1,079 records of reviewers, encompassing 19 variables such as given name, family name, affiliation, email, country, ORCID (Open Researcher and Contributor ID), username, date registered, date of last login, inline help, phone, and mailing address. In the JISEBI dataset, the reviewer data encompassed the given name, family name, Scopus ID, and email. Subsequently, the Scopus search API was used to retrieve the reviewer’s research data.

System flow

A process for matching research expertise to recommend reviewers is illustrated in Fig. 1. User data were exported from OJS, specifically focusing on usernames. These data were then uploaded to the journal reviewers database system. Utilizing the Scopus search API, the system searched for “Scopus ID” data corresponding to the uploaded names. These data, derived from publication titles and abstracts, were used to map the reviewers’ research subject areas. The text data underwent preprocessing, which involved combining the title and abstract, converting the text to lowercase, removing stopwords, and tokenizing and lemmatizing words. Subsequently, a TF-IDF algorithm calculated the importance of each word within the text associated with each reviewer. To perform research expertise matching, editors are required to input the title and abstract of the article needing review into the system. The system then compares this information against the reviewers’ research titles and abstracts. Using TF-IDF and cosine similarity, the system identifies similarities in the subject areas. The top 10 manuscript authors with the most similar research interests and expertise are recommended as reviewers. These recommendations ensure that the selected reviewers have research interests and expertise that align closely with the topic of the manuscript, thereby enhancing the peer review process.

Upload user data from OJS

The system utilized data from a Microsoft Excel file (Microsoft Corp) that was uploaded by the user, specifically targeting information on OJS usage. This Excel file was sourced from the user data export feature provided by OJS, containing details such as usernames, email addresses, and researcher identifiers like Scopus ID and Scholar ID. The system processed these data by extracting and consolidating first and last names into a complete name format. All the information was then stored in a database.

Scopus search API

In this research, data extraction processes were conducted—namely, reviewer and abstract data extraction. This data extraction process utilized the Scopus search API (https://dev.elsevier.com/) and the elsapy library (https://github.com/ElsevierDev/elsapy) to retrieve data from Scopus. To use the API, it is necessary to obtain an API key and adhere to specific terms and conditions.

The initial step in data extraction involved reviewer extraction, which was accomplished by retrieving author IDs from OJS user data using the Scopus search API. The objective was to collect data from Scopus for each user registered in OJS. However, it is possible that some users might not have Scopus author IDs. In those cases, these users would not be included in the subsequent processing steps. The parameters used to retrieve the author ID were the first and last names. After receiving the response from elsapy, the obtained author ID data were then entered into the database.

The second step in data extraction involved obtaining the title and abstract data from articles authored by reviewers already listed in the database. During this process, the elsapy library was utilized to retrieve the reviewer’s Scopus ID using the author ID as a parameter. The Scopus ID was then used as a parameter in the Scopus API to extract the title and abstract of the article. This data was subsequently stored in the database.

Text preprocessing

The preprocessing steps performed in this study included combining the title and abstract, converting text to lowercase, removing stopwords, tokenization, and lemmatization [3]. First, the title and abstract of each article were merged into a single text block. This consolidation aimed to gather essential information from both elements, enhancing the efficiency of subsequent text analysis. Following this, the text was converted entirely to lowercase to ensure that the model or algorithm treated words identically, regardless of their case. Stopwords, which are frequently occurring words in language that carry minimal informational weight, such as “and,” “or,” and “the,” were then removed. This elimination helped to filter out elements that might be irrelevant for further analysis. The process of tokenization broke the text down into discrete units known as “tokens,” which could be words, phrases, or characters, depending on the tokenization method employed. The final step involved lemmatization, where tokens were analyzed to determine their root form, or lemma. This step was crucial for reducing variations of words to their base forms, allowing, for example, both “running” and “runs” to be represented by the lemma “run.”

TF-IDF

TF-IDF is an algorithm based on calculating the frequency of words that appear in a text. The more often a word appears, the more valuable it is considered for that particular text [4]. Where n_i,j is the number of occurrences of the word t_i in file d_j, Σ_kn_k,j is the sum of the occurrences of all words in the file dj, The formulas for TF, IDF, and TF-IDF are defined in equations 1,2, and 3.

(1)

tfi,j=ni,jΣknk,j

(2)

idfi=log⁡|D||{j:ti∈dj}|

(3)

tfidfi,j=tfi,j×idfiΣti∈dj|tfi,j×idfi|2

Cosine similarity

Cosine similarity is an algorithm that calculates the similarity or closeness between two vectors. This algorithm is computed using the dot product of vector X with Y, divided by the product of the magnitudes of vector X and vector Y [5]. The formula for cosine similarity is defined in equation 4.

(4)

cos⁡θ=X⋅Y‖X‖‖Y‖

System implementation

To obtain recommendations for reviewers, editors must input the titles and abstracts of manuscripts under review. The system the searches its database to identify the most relevant reviewers. The output from the system provides a list of potential reviewers who are researching the same topics as the manuscript in question. Notably, these reviewers are already registered with OJS, simplifying the process for editors, who only need to extend invitations through the OJS system. The align with the subject of the article under review, as illustrated in Fig. 2. Editors can use this information to effectively recruit reviewers. The proposed system identifies the 10 most pertinent articles based on their titles and abstracts and subsequently recommends reviewers from the authors of these articles. By doing so, it ensures that the most suitable reviewers are suggested for the research topics. This approach improves the review process by guaranteeing that all recommended reviewers are registered users within the OJS system.

The system was developed using Python (Python Software Foundation) and the Django (Django Software Foundation) web framework. The source code can be accessed at https://github.com/AgileRE-2023/Journal-Database-Reviewer. In addition, the User Guide can also be accessed at https://bit.ly/REMSUserGuide.

Measurement

This study compared the recommendations of the system with observations made by the JISEBI editorial team to assess the system’s performance. The system processed titles and abstracts from 30 research articles. Based on these recommendations, the JISEBI editorial team determined the most relevant author for the inputted journal data. To ensure data confidentiality, all developers involved with this software were required to agree to a data confidentiality statement.

The system underwent evaluation by the journal editorial team. The evaluation scenario was as follows:

We obtained data from 30 manuscripts under review by the JISEBI journal, comprising titles and abstracts. This data was input into our system, and we subsequently acquired the results of the reviewers’ recommendations. The JISEBI editorial team reviewed the list of recommended reviewers to assess whether each candidate possessed the necessary expertise in the research field relevant to the manuscript under review. Each candidate was evaluated for relevance. This assessment took place on January 2, 2024. Additionally, it is important to note that the first, second, and third authors of this document are members of the JISEBI editorial team. However, we did not participate in the process of annotating reviewer candidates. This task was performed by other members of the editorial team, ensuring that we remained uninvolved.

From these observations, precision, MAP, and MRR were calculated to evaluate the system’s performance [6–8]. These metrics help users select high-quality items from a set of available options. Precision, or confidence, reflects the proportion of positive predicted cases that are actual positives. It represents the ratio of relevant reviewer recommendations to the total number of recommended reviewers. In this study, we did not calculate the recall value because we were unable to obtain a false negative value from the recommendation results. This false negative value pertains to a relevant reviewer who does not appear in the recommendation results. MAP and MRR are metrics used to assess model performance in tasks, particularly in information retrieval. They take into account the order of all relevant recommendations. These metrics are commonly employed in evaluating systems that generate ordered results. The MAP metric was determined by calculating the mean of the average precision across all relevant recommendations.

Meanwhile, MRR was calculated by taking the reciprocal position of the first relevant recommendation. For this purpose, k is the total number of reviewer recommendations, n is the number of relevant recommendations, P(i) is the precision of recommendation at rank position i, rel(i) is an indicator function equaling 1 if the recommendation is relevant, and 0 otherwise, and rank refers to the rank position of the first relevant recommendation. The formula for precision (P), average precision (AveP), and reciprocal rank (RR) for each test data are defined in equations 5, 6, and 7. Meanwhile, the final precision, MAP, and MRR are the mean or average of all test data.

(5)

P=nk

(6)

AveP=1n∑i=1kPi×reli

(7)

RR=1rank

Results

The system’s results provided recommendations for reviewers by matching the title and abstract of the manuscript with the reviewer’s area of research. These results were obtained by entering data from manuscripts under review at JISEBI. Table 1 displays the system evaluation results, which include metrics such as precision, MAP, and MRR. According to the evaluation, the system achieved an average precision of 0.47, an average MAP of 0.77, and an average MRR of 0.87. Specifically, the precision score averaged 0.47, with the highest recorded score being 0.88 and the lowest being 0.1. The MAP score averaged 0.77, reaching a maximum of 1 and a minimum of 0.33, with the most common value being 1 and a median of 0.82. Similarly, the MRR score was 0.87, with values ranging from a maximum of 1 to a minimum of 0.33, and the most frequently occurring value was 1.

Discussion

Interpretation

The evaluation results demonstrated that the proposed system achieved an average precision of 0.47, a MAP of 0.77, and an MRR score of 0.87. These MAP and MRR scores suggest that the system is capable of providing relevant reviewer recommendations along with effective ranking. The high MRR score indicates that the initial recommendations are generally relevant. Similarly, the MAP score suggests that the rankings of all recommendations are predominantly relevant. This performance aligns with the primary objective of this research, which is to identify and rank reviewer recommendations based on the highest similarity.

Although the MAP and MRR scores are generally high, the precision of some recommendations is low, with an overall precision of 0.47. This low precision may be attributed to the varying levels of expertise among the available users and the manuscripts being reviewed. The recommendation system operates by comparing the similarity of manuscripts under review with the published data of potential reviewers. Given the broad range of existing research fields, this approach carries a significant risk of low precision. The system is designed to utilize data exclusively from OJS users. This focus provides a distinct advantage by ensuring that recommendations are highly relevant to the journal’s scope and its collaborative network. However, a notable limitation is that the number and diversity of available reviewers are confined to those within the OJS user base. Consequently, this restriction might result in a lack of available reviewers for certain specialized topics or research focuses.

Editors play a crucial role in deciding whether to adopt recommendations for reviewers. They must also continually assess the appropriateness of the expertise of potential reviewers. This process is facilitated by the system, which provides information related to the paper’s title from prospective reviewers on the recommendation page. Additionally, the title is clickable and directly displays the manuscript at its publication site using digital object identifier (DOI) information. This feature serves as a valuable decision support tool for editors when inviting reviewers.

In implementing the system, we utilized the top 10 publications pertinent to the manuscript under review. Thus, the number of recommended reviewers may not always be 10 if a candidate has authored multiple papers within these top publications. To address this, a similarity threshold can be introduced to maintain a consistent number of reviewer candidates. Implementing such a threshold could mitigate these limitations and improve the system’s effectiveness. However, the optimal threshold may differ based on the policies and user availability of each journal. Therefore, in this study, we established a criterion that limits the recommended reviewers to those who have authored the 10 most relevant manuscripts.

Comparison with previous studies

The proposed method introduces a reviewer recommendation system that utilizes cosine similarity, TF-IDF, and web data extraction to enhance system performance. Various studies have explored methods for recommending reviewers in academic journals and conference management systems. Protasiewicz [9] proposes a decision support system that recommends relevant reviewers by analyzing researcher profiles and calculating the cosine similarity between their expertise and the problem under review. Choi et al. [10] proposed an algorithm to recommend appropriate peer reviewers for academic manuscripts based on their scholarly activities and achievements. Koçak et al. [11] presented an approach that combines hesitant fuzzy VIKOR and TOPSIS for reviewer selection within a hierarchical decision-making framework. Meanwhile, Duan et al. [12] introduced a different approach called sentence pair modelling-based reviewer assignment, which aimed to effectively model the research fields of papers and reviewers and subsequently match them for reviewer assignment. Furthermore, Kalmukov [13] proposed a heuristic assignment algorithm to optimize reviewer selection based on various factors, including expertise and workload, to ensure efficient and effective peer review.

Limitations

Despite the advancements made by existing methods, there are still limitations in terms of accuracy, efficiency, or scalability. This study, however, primarily focuses on practical improvements in reviewer recommendation systems, particularly the ease of inviting reviewers. We analyzed the existing user data from the journal, which includes potential reviewers. Although the journal is aware of this data, the challenge often lies in determining which users are suitable to be invited as manuscript reviewers. Users typically list their preferred research topics, but this information can often be misleading. Moreover, access to this data is limited to editors for users who have registered as reviewers. In the proposed system, we can profile all user data in OJS that relates to scientific publications. This capability is extremely beneficial for editors in identifying potential reviewers within their database.

Conclusions

The above metrics demonstrate that the proposed system is effective, providing accurate recommendations that improve the quality of the peer review process. Although existing methods for recommending reviewers provide a range of approaches, this study focuses on practical advancements, particularly the simplicity of inviting reviewers who are already part of the journal network. By utilizing OJS user data and employing algorithms like cosine similarity and TF-IDF, the proposed system enhances the journal’s capability to pinpoint and invite suitable reviewers, thereby contributing to the production of high-quality scientific publications.

Notes

Conflict of Interest

No potential conflict of interest relevant to this article was reported.

Funding

The authors received no financial support for this article.

Data Availability

Dataset files are available from the corresponding author upon reasonable request. The data are not publicly available due to confidentiality restrictions, as they contain personal information such as names, email addresses, and usernames. Additionally, the data were obtained from the Open Journal Systems, where data confidentiality must be maintained.

Supplementary Materials

Fig. 1.

Reviewer recommender system flow. OJS, Open Journal Systems; API, application programming interface; TF-IDF, term frequency-inverse document frequency.

Fig. 2.

Example output of the system: recommending reviewers based on the title and abstract of the manuscript to be reviewed.

References

• 1. Hevner A, Chatterjee S. Design science research in information systems. In: Hevner A, Chatterjee S, editors. Design research in information systems: theory and practice. Springer; 2010. p. 9–22. https://doi.org/10.1007/978-1-4419-5653-8_2. ArticlePubMed
• 2. Siahaan D, Raharjana IK, Fatichah C. User story extraction from natural language for requirements elicitation: identify software-related information from online news. Inf Softw Technol 2023;158:107195. https://doi.org/10.1016/j.infsof.2023.107195. Article
• 3. Abidin Z, Junaidi A. Text stemming and lemmatization of regional languages in Indonesia: a systematic literature review. J Inf Syst Eng Bus Intell 2024;10:217-31.https://doi.org/10.20473/jisebi.10.2.217-231. Article
• 4. Liu C, Sheng Y, Wei Z, Yang YQ. Research of text classification based on improved TF-IDF algorithm. In: 2018 IEEE International Conference of Intelligent Robotic and Control Engineering (IRCE); 2018 Aug 24–27; Lanzhou, China. IEEE; 2018. p. 218–22. https://doi.org/10.1109/IRCE.2018.8492945. Article
• 5. Reswara CG, Nicolas J, Ananta M, Kurniadi FI. Anime Recommendation System using BERT and cosine similarity. In: 2023 4th International Conference on Artificial Intelligence and Data Sciences (AiDAS); 2023 Sep 6–7; IOPH, Malaysia. IEEE; 2023. p. 109–13. https://doi.org/10.1109/AiDAS60501.2023.10284693. Article
• 6. Chen AR, Chen TH, Wang S. Pathidea: improving information retrieval-based bug localization by re-constructing execution paths using logs. IEEE Trans Softw Eng 2022;48:2905-19.https://doi.org/10.1109/TSE.2021.3071473. Article
• 7. Isinkaye FO, Folajimi YO, Ojokoh BA. Recommendation systems: principles, methods and evaluation. Egypt Inform J 2015;16:261-73.https://doi.org/10.1016/j.eij.2015.06.005. Article
• 8. Turpin A, Scholer F. User performance versus precision measures for simple search tasks. In: Dumais S, Efthimiadis EN, Hawking D, Järvelin K, editors. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 2006 Aug 6–11; Seattle, WA, USA. Association for Computing Machinery; 2006. p. 11–8. https://doi.org/10.1145/1148170.1148176. Article
• 9. Protasiewicz J. A support system for selection of reviewers. In: 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC); 2014 Oct 5–8; San Diego, CA, USA. IEEE; 2014. p. 3062–5. https://doi.org/10.1109/SMC.2014.6974397. Article
• 10. Choi DH, Hyun JW, Kim YR. An algorithm for peer reviewer recommendation based on scholarly activity assessment. IEEE Access 2023;11:39609-20.https://doi.org/10.1109/ACCESS.2023.3263857. Article
• 11. Koçak S, İç YT, Atalay KD, Sert M, Dengiz B. The development of a reviewer selection method: a multi-level hesitant fuzzy VIKOR and TOPSIS approaches. J Ambient Intell Humaniz Comput 2023;14:3275-302.https://doi.org/10.1007/s12652-021-03466-5. Article
• 12. Duan Z, Tan S, Zhao S, Wang Q, Chen J, Zhang Y. Reviewer assignment based on sentence pair modeling. Neurocomputing 2019;366:97-108.https://doi.org/10.1016/j.neucom.2019.06.074. Article
• 13. Kalmukov Y. An algorithm for automatic assignment of reviewers to papers. Scientometrics 2020;124:1811-50.https://doi.org/10.1007/s11192-020-03519-0. Article

Figure & Data

References

Citations

Citations to this article as recorded by