Revealing Reviewers’ Identities as Part of Open Peer Review and Analysis of the Review Reports

This research article is aimed at comparing review reports, in which the identity of the reviewers is revealed to the authors of the papers, with those where the reviewers decided to remain anonymous. The review reports are gathered as part of the peer review process of the European Scientific Journal (ESJ). This journal maintains a single-blind peer review procedure and optional open review. Reviewers are familiar with the names of the Revealing Reviewers’ Identities as Part of Open Peer Review 2 Liber Quarterly Volume 3


Introduction
Peer review is the well-known gatekeeper of publishing sound research. However, it is not a flawless process but is followed by numerous inconsistencies (Smith, 2006) including gender biases or biases on the basis of geographical affiliation of authors (Kaatz, Gutierrez, & Carnes, 2014;Lee, Sugimoto, Zhang, & Cronin, 2013). Mahoney (1997) argues that reviewers even favour manuscripts that confirm their own perspectives and vice versa. Although the benefits of peer review cannot be disputed, the necessity of constant improvements is inevitable (Smith, 1999).
The traditional peer review concept, used by the majority of the academic journals, refers to single-blind or double-blind peer review. The names of the reviewers are not presented to the authors of the papers nor are the review reports made available to the public. However, the landscape of scholarly communication is changing and new forms such as open review are emerging (Rodgers, 2017).
Under the concept of open science, the open review is aimed at strengthening transparency and accountability of the internal processes of academic publishing (Bravo, Grimaldo, López-Iñesta, Mehmani, & Squazzoni, 2019). Ross-Hellauer (2017) concludes that most prevalent elements among open review definitions are open identities and open reports. While researchers in general support the idea of opening their reports, they are still sceptical on the effects of opening reviewers' identities.
A robust debate has been initiated about the advantages and disadvantages of open peer review. Supporters argue that openness serves as a corrective for the usual abuses that anonymous peer review allows. As examples we can mention overcoming various types of biases as well as providing the deserved credits to the reviewers instead of only a letter of appreciation (Groves, 2010). Moreover, open review transforms the rude tone of reviewers into constructive criticism (Le Sueur et al., 2020). Opening the peer review process can even secure a higher number of citations (Zong, Xie, & Liang, 2020). On the other hand, some studies argue that when their identity is disclosed reviewers restrain themselves from providing critical comments, replacing them with more generous ones, which negatively affects the quality of the peer review procedure. Those against the open concept remind that reciprocity affects human behaviour and open review neglects that fact. Therefore open review may work under ideal circumstances only (Khan, 2010).
Regardless of the concept a journal adopts, it is always valuable to have a clear account on reviewers' behaviour. They are key players in the process and better knowledge on their preferences would contribute to further improvement of peer review. Squazzoni et al. (2020) stated that research on peer review is necessary. It will not only improve the process for the authors, reviewers and editors, but it will also increase the reliability, rigour, and relevance of scientific literature. Therefore, this paper analyses reviewers' behaviour through the analysis of their reports. It examines the reports of the reviewers who decided to reveal their identity under the open review and those whose identities remained anonymous for the authors of the papers. The focus was on comparing the assessment of the manuscripts and reviewers' recommendations. The authors Liber Quarterly Volume 30 2020 concluded that the reviewers are more critical when their identity is hidden, i.e., reviewers are keener to reveal their identity when their remarks are more positive.

Literature Review
It is very likely that, when asked to choose, the reviewers will decide to remain anonymous rather than reveal their names to the authors of the papers (Van Rooyen, Godlee, Evans, Black, & Smith, 1999). Only a small minority of them will decide to unmask their identity (Bravo et al., 2019). On the other hand, Walsh, Rooney, Appleby, & Wilkinson (2000) argued that during a journal's randomised trial, over 76% of the reviewers decided to sign their review reports.
Revealing reviewers' identity had no important effect on the quality of the review reports, neither the recommendation of the reviewers, nor the time to review a paper. The difference is not significant. However, the reviewers were more likely to reject the manuscripts and less likely to suggest acceptance when their identity remained hidden from the authors of the papers (Van Rooyen et al., 1999). Furthermore, there was no difference in the quality of review reports when the reviewers were told that their signed review reports will be available online (Van Rooyen, Delamothe, & Evans, 2010). Walsh et al. (2000) concluded that even though signed review reports were of a higher quality, the reviewers made more favourable recommendation for the authors of the papers. Bravo et al. (2019) argue that open review did not compromise the peer review process. Even though the reviewers with more positive recommendations more often revealed their identity, there was no significant difference among the recommendations of the reviewers. On the other hand, open identities can solve many of the problems peer review is faced with (Godlee, 2002;Smith, 1999). Thelwall, Allen, Papas, Nyakoojo, & Weigert (2020) argue that when reviewers are able to read previous review reports on the same paper, as part of the open review process, it does not affect their comments and recommendations. However, editors should be cautious when assigning reviewers from the same country as the authors of the papers and under the open peer review procedure.
Research on the peer review process has been constantly growing. The findings presented in the analysis below can enrich the knowledge and the discussion on the openness of the peer review process. Therefore, the authors open their data, thereby making the findings more transparent and reliable.

Methods and Data
Data were gathered as a part of the peer review procedure of the European Scientific Journal (ESJ). This journal maintains a single-blind peer review model and reviewers are familiar with the identity of the authors but not vice versa. However, starting in June 2019, the journal adopted optional open review. With this option, reviewers may choose to reveal their identity to the authors of the paper they review. This journal was chosen for our project because it has a sufficiently large number of open reviews for allowing reliable conclusions, and because the authors -as members of its editorial boardhad easy access to the review data and they were familiar with the rigour and transparency of its review procedure.
The sample of 343 review reports were collected from May to July 2019. Some of the reports were made before the open review option was introduced. Under this, the reviewers were familiar with the journal's process and aware that their identities are hidden. In the other group, there are review reports after the open review adoption. In this group, reviewers have chosen to open their identity, to remain anonymous, or they simply skip answering the question. The reviewers are part of the ESJ's editorial board.
The data were transformed from descriptive to numeric so as to enable the statistical analysis of the sample. The editorial office is familiar with gender and institutional affiliation of the journal's reviewers and provided these data. The country of affiliation of the reviewers was identified as developing or developed according to the United Nations country classification. 1 Genders are identified as female or male, although the gender of some reviewers was not available. Basically, there are four categories of review reports recognised in this paper: PR1 -the review report was received before the open review option and the identity of the reviewers was not visible to the authors, PR2 -the reviewer approved to reveal his/her identity to the author, PR3 -the reviewer did not approve his/her identity be revealed to the author, and PR4 -reviewer did not respond to the question. The recommendations of the reviewers were categorized as 1. Rejection with assigned score, 2. Major revision and resubmission with assigned score, 3. Minor revision with assigned score and, 4. Acceptance with assigned score. Thus, these recommendations followed the evaluation sheets and assessments of each part of a paper such as: title, abstract, grammar, methods, body of a paper, conclusion, and references. Each of the categories was assessed between 1 (poor) to 5 (excellent) points. 35 points altogether was the maximum with 7 point as minimum.
All the data analysis was performed using the Python programing language based on NumPy, Pandas, and Scipy packages. The dataset and analysis are open and are provided in the link specified in the endnote. 2

Results
Results are presented in several sections analysing different parts of the peer review process. In the first section, a statistical analysis of a sample is provided with the structure regarding the country of the institutional affiliation of the reviewers as well as their gender. In the second section the recommendations are analysed regarding the peer review openness and in the end the rationality of recommendations regarding the evaluation sheets is assessed.

Statistical Analysis of the Sample
In this part of the paper, the statistics of the sample analysis is presented. The reviewers play a key role in the peer review process and assessment of the structure of the research sample can provide important information about the research group. The country of affiliation and gender are two main characteristics of the reviewers whose review reports were taken into consideration. The affiliated countries of the reviewers preparing the review reports are presented in Table 1.
Review reports made by reviewers from developing countries dominate with 63.6%, with the remaining 36.4% coming from reviewers from developed countries. These review reports were submitted by a group of reviewers that is composed of 63% unique reviewers from developing countries and 37% that are affiliated in developed countries. The share of affiliation between revisions and reviewers is very similar, indicating that both groups of reviewers were active in the revision process almost in the same degree.
The share of the review reports made by reviewers according to their gender is presented in Table 2.
A majority of the review reports were made by male reviewers (65.3%), and 31.8% were made by female reviewers; only 2.9% of the review reports were made by the reviewers with no available data on their gender. The group of reviewers submitting the review reports is represented in 64% by males, in 32% by females and in 4% with no available data about the gender. The share of gender between review reports and reviewers is very similar, indicating that the specified groups of reviewers were active in the peer review process almost in the same degree.
There is very little difference regarding the share between the group of reviewers and their recommendations and for this reason the recommendations can be taken into consideration as a research sample.

Analysis of Reviewers' Recommendations
It is worth mentioning that reviewers have reviewed the papers, which passed an initial screening by the journals' editorial office. Reviewers' recommendations regarding the papers they reviewed are analysed in this part. There were four possible options from which the reviewers could select their recommendation to the editorial office: rejection, major revision and resubmission, minor revision, or acceptance without revision. Points, as mentioned in the Methods and Data section, are assigned to each decision in order to enable a statistical analysis. Table 3 presents the percentage of reviewers' recommendations in a sample.
More than a half of the recommendations were with minor revision and only in 5.8% of the reports did the reviewers recommend a direct rejection of the paper. Reviewers' recommendations regarding the country of their institutional affiliation are presented in Table 4.
When every group is analysed separately, minor revision is the most popular decision between reviewers both in developing and developed countries. The rejection ratio is similar in both groups as well. However, there is a difference between the acceptance without revision decisions. Reviewers from  developed countries tend to recommend direct acceptance more often in their reports.
In Table 5 the percentage of reviewers' recommendations according to their gender are analysed.
Minor revision is the most popular decision regardless of gender. There is no significant difference in the reviewers' recommendations on the basis of gender.
When invited to review a paper, the ESJ reviewers are asked to fill an evaluation form and comment on the main parts of every paper. They should assess those parts by giving evaluation points. There are seven parts (title, abstract, grammar, methods, body of a paper, conclusion, and references) that reviewers should appraise in the evaluation form with 5 points as maximum and 1 as minimum. The total result may be maximum 35 points. The data can be grouped into 4 sets of recommendations according to the total points assigned. The results are presented in Figure 1.
When analysing Figure 1, a box plot reveals the ranges of points for each recommendation, median, first and third quartile, the violin plot shows the distribution of points and its density. As can be noticed for rejection, major and minor revision decisions, median and mean of points are close to each other. However, recommendations marked as acceptance without revision is characterized significantly by a higher median than mean. This is the result of a much higher concentration of points that are close to the higher boundary. It means that the acceptance without revisions was recommended only when maximum points in evaluation sheet were granted. This result can indicate a high quality of recommendations related to acceptance of a paper. The results provided in this section show that there is no big difference between recommendations made by males or females affiliated to the institutions operating in developed or developing countries. Moreover, when evaluation sheets are taken into consideration, the decisions are fair and made based on the average number of points, but in case of acceptance without revision only the highest number of points results in such a decision, meaning that reviewers accept only the highest quality texts.

Peer Review Reports and the Openness of Reviewers' Identities
This part of the paper is related to the analysis of the review reports according to the availability of the open identity option. Thus, PR1 represents the review reports received before the "open" option. PR2, PR3, and PR4 are related to the period when reviewers were asked about revealing their names in the peer review process. They could approve to reveal their names, reject such option, or do not prefer to answer to the question. In Table 6, statistical analysis of the reviewers' approvals is provided.
Most of the reports are in the group when the reviewer approved to reveal his/her name to the author, that is 33.8%. This is followed by the reports received before the open review was adopted as a part of the ESJ peer review process that accounts for 32.4%. The share of reviewers that did not approve their names to be revealed to the author accounts for 23.6%. The smallest group form the reviewers who skipped answering the question related to the openness of the identity. However, the identity of the reviewers who did not answer the question remained closed.
Taking into consideration only the reviewers that were able to choose whether or not to reveal their identity, it appears that 50% of them decided to reveal their names to the authors. This is presented in Table 7.
A high percentage of the reviewers (50%) decided to choose the open option and sign their review reports, but 15.1% of the reviewers skipped this question. The major reasons for skipping this question cannot be determined. However, the openness of the identity may not be an important issue to these reviewers. On the other hand, they perceived skipping the questions as an approval of the single-blind concept where the anonymity is guaranteed and is maintained by the journal.
In Table 8, the peer review type and the country of institutional affiliation of the reviewers are analysed and the results are presented.  The results presented in Table 8 indicate that regarding their reports the reviewers from developing countries approved to reveal their identities more often than their colleagues from developed countries. However, a majority of the reviewers from developed countries are in the group when the "open identity option" was not yet available.
Therefore, the table below will more clearly present reviewers' preferences on open identity in connection with the country of their institutional affiliation. Table 9 embraces only the reports where the open review option was available.
As can be seen, reviewers from developing countries more often decide to open their identities to the authors of the papers they review, while among the reviewers from developed countries there is only a small majority in favour of approving that their name be revealed to the author.  In the next step the recommendations and the gender are analysed and the results regarding the groups of peer review are presented in Table 10.
When the sample is analysed regarding the gender, the males dominate in the group of reviewers that approved to reveal their name to the author while females dominate in the group of reviewers that did not approve their name to be revealed to the author.
In Table 11 we repeat the same analysis, leaving out the reports from the period without a choice.
The results confirm that female reviewers are less likely to reveal their identity. On the other hand, male reviewers more often answered positively to this question.

Recommendations regarding the openness of the reviewers' identities
The recommendations made by reviewers may vary depending on the openness of the peer review. Table 12 presents the average points that reviewers awarded in the evaluation forms when assessing separate parts of the papers.
The average points related to recommendations for the whole sample is 2.93 and the highest is in the group where reviewers did not respond to the question about revealing their names. The lowest number of points, on the other hand, is in the group where the reviewers did not approve their names to be revealed to the authors. It can be seen that the PR1 average score (from the review reports received before the "open" option was presented as a part of the peer review process), is almost equal to the average for PR2, PR3 and PR4 together, indicating that when reviewers were not asked about revealing their name, their recommendations were not influenced by the "open" review option.
The average score in each peer review option regarding the points in the evaluation sheets is presented in Table 13.
The average points in a sample is 26.73. The highest is in the group where reviewers did not respond to the question, while the lowest is in the group where reviewers did not approve their names to be revealed to the authors. Upon the assessment of each part of a paper, the reviewer is making a recommendation to the editorial office. The reviewer can select one of the four options available. Acceptance without revision, minor revision, major revision, or rejection of a paper can be recommended. The statistics for recommendations of the reviewers are presented in Table 14 according to the openness of their identities. Table 14 shows that minor revision is the most common recommendation of the reviewers regardless of whether reviewers decided to open their identities or remained anonymous. When recommending major revision, the reviewers are not hesitating to open their identities, as the difference is insignificant.  However, when recommending rejection, the reviewers often decide to remain anonymous. Reviewers are also keener on opening their identities when acceptance without revision is suggested.
The Spearman rho correlation between the number of points in the evaluation sheet and the recommendations for the sample and peer review identity options is calculated and the results are presented in Table 15. Due to the nonlinear relation between the number of points and the reviewers' recommendations, the Pearson correlation is inapplicable.
The results presented in Table 15 show that there is a significant correlation in every group of peer review types between the recommendation (rejection, major revision, minor revision, acceptance) and the evaluation sheet points. The highest correlation is in the group where the reviewer did not approve his/her name to be revealed to the author and the lowest when the reviewer approved to reveal the name to the author. It can be concluded that there is a difference in the strength of this correlation depending on the openness of their identities.
In the next step the peer review correlation analysis in certain groups of peer review reports regarding the country and gender of reviewers and the results are presented in Table 16. When the recommendation and the correlation with the country of affiliation and gender are taken into consideration, it appears that there is no significant correlation between those factors and the reviewers' recommendations (p-value > .100). This confirms that both gender and the country of the affiliation did not influence the recommendations of the reviewers.

Rationality of Reviewers' Recommendations
The recommendations of the reviewers are subjective regarding the points awarded in the evaluation sheet. Such a situation can be related to the rationality and risk averse heuristic. It is expected that a larger range of points for a specific recommendation refers to a less rational recommendation. The results of the analysis of the whole sample are presented in Table 17.
The recommendations based on the evaluation sheets show the wide range of points that influence the recommendations of reviewers. The largest range is recognized in the case of major revision and resubmission and the smallest in the case of acceptance without revision. It can be concluded that those groups of recommendations are respectively the least and the most rational in the sample.  Figure 2 shows the relations between the total number of points and the number of different recommendations made for the final results. The same analysis is repeated for different peer review options regarding the identity of the reviewers.
In the first step, the group of PR1 (representing the review reports received before the "open" option was presented as a part of the peer review process) is analysed and the results are presented in Table 18.
Another analysis is made for the PR2 -group (where reviewers approved to reveal their names to the author). The results are presented in Table 19.
For PR3 (where reviewers did not approve their names to be revealed to the author) the results are presented in Table 20. For PR4 (where reviewers did not respond to the question) the results are provided in Table 21. Figure 3 presents the graphs of the peer review and the recommendations of the reviewers.
The analysis of the results can help to understand the recommendation process of reviewers in correlation with the openness of peer review. The largest range for rejection recommendation is found for PR3 when reviewers did not approve their names to be revealed to the author and is equal to 14. Reviewers not revealing their identity may be less certain about their   decisions. The smallest range is found for PR2 when reviewers approved to reveal their identity to the author and is equal to 0. The largest range for major revision and resubmission is in the group representing PR3, when reviewers did not approve their names to be revealed to the author and is equal to 16. Moreover, the smallest range for this type of recommendations is found in the group PR4 when reviewers did not respond to the question and is equal to 15. In the group of minor revision, the largest range is found in the group PR4 when reviewers did not respond to the question and is equal to 16. The smallest range is found in the group PR3 when reviewers did not approve their names to be revealed to the author. For the acceptance without revision, the largest range is found in the group PR1 when review reports received before the open review was presented as a part of the peer review process and PR2 when reviewers approved to reveal their names to the author and is equal to 13. The smallest range is found in the group PR3 when reviewers did not approve their names to be revealed to the authors and is equal to 6. PR3, where reviewers did not approve their names to be revealed to the author, is related to the largest range in the rejection and major revision decisions representing the lowest rationality in these situations. PR4, where reviewers did not respond to the question, is related to the largest range in the group of minor revision and acceptance. The largest range is in the PR1, where review reports were received before the open review option, and in PR2, where reviewers approved to reveal their names to the authors. The less rational decisions are in the group PR3, where reviewers did not approve to reveal their names to the author in the case of rejection and major revision. This can be related to the reviewers' lack of self-confidence or lower certainty related to the decisions.
PR2, where reviewers approved to reveal their identity to the author, shows the smallest range in the group of rejection. PR4, where reviewers did not respond to the question, shows the smallest range in the group of major revisions. PR3, where reviewers did not approve to reveal their names to the authors, shows the smallest range in the group of minor revisions and acceptance. The most rational decisions are in the group PR3, where reviewers did not approve to reveal their names to the authors, and this in the case of a recommendation for minor revision or acceptance.
The results on the rationality of decisions show that it depends on the option of openness taken into consideration. To improve the peer review process, the open peer review is emerging (Rodgers, 2017) and a significant share of reviewers, as it was presented in this research, decide to reveal their names to the authors.

Limitations and Discussion
The limitation of the study is that review reports of only one journal were analysed. Although the journal receives papers in various academic disciplines, no analysis on the basis of a specific academic domain was conducted. However, the paper reveals that the reviewers' preferences in general depend on whether their identity is open or they decided to remain anonymous.
There were review reports before and after the journal adopted open peer review process. With the introduction of open review, the reviewers were given the option to open their identity or to remain anonymous to the authors of the paper they review. This helps in analysing whether this option has changed the behaviour of the reviewers. The results have shown that the average grades reviewers awarded, as well as the total average of the points, before and after the open option, did not introduce any significant difference and the numbers remained the same. It means that the review reports were not influenced by the implementation of the open review option.
Furthermore, after the open review was implemented, half of the reviewers chose to open their identities. The other half decided to stay anonymous or simply skipped to answer the question. This is far above the result of the study where only 8.1% of the reviewers agreed to sign their review reports and open their identities (Bravo et al., 2019). However it is less than what Walsh et al. (2000) found, where over 76% of the reviewers were willing to reveal their identities. Furthermore, reviewers from developing countries approved to reveal their identities more often than the reviewers from developed countries. Female reviewers were more likely to remain anonymous while male reviewers more often chose the open option.
Reviewers appraised 7 aspects of each paper with points from 1 (poor quality) to 5 (good quality). The highest result was in the group where reviewers did not respond to the question about revealing their names. The lowest result was in the group where the reviewers did not approve their names to be revealed to the authors. Therefore, it can be stated that reviewers more often avoid revealing their names when a more critical approach to the reviewed papers is presented.
When it comes to the recommendations the reviewers made in their reports, it can be seen that a majority of them have opted to remain anonymous when their recommendation is rejection of the paper. The level of rejection rate remained similar before and after the open review option. However, most of the reviewers have chosen not to use the opportunity and hide their identities. This might look a logical decision initiated by numerous factors. However, the real reasons for hiding their identity when rejection is recommended were not part of this study.
Furthermore, the reviewers did not hesitate to open their identities when major revision or resubmission was recommended in their review reports. The difference was not significant. However, similarly to Bravo et al. (2019), the reviewers were more likely to open their names and sign the reports when their recommendations were more in favour of the authors of the papers, such as minor revision or acceptance without revision.
The largest range of points are found in the groups with for rejection or major revision, while the less rational recommendations are in the group PR3, where reviewers did not approve to reveal their names to the authors. The most rational decisions are in the group PR3 as well, for the recommendations for minor revision or acceptance.

Conclusion
The implementation of open review and the option given to reviewers to either choose between opening their identity or remaining anonymous did not affect the review reports. The average points appraisal and recommendations in the review reports before and after the open review did not show a significant difference.
Half of the reviewers decided to open their identities while the other half decided to remain anonymous or simply skipped to answer the question. A majority of the reviewers has chosen to remain anonymous when rejection of a paper was recommended. Contrary to this, when reviewers' recommendations are more favourable, such as acceptance with minor revision or acceptance without revision, they are more likely to open their identities. However, when major revision and resubmission was recommended, the difference in the number of those who open their names and those who remained anonymous was insignificant. Major revision and resubmission is not considered a very favourable approach.
Revealing reviewers' identities did not influence the recommendations and grades in their review reports. However, reviewers are not keen on revealing Liber Quarterly Volume 30 2020 their names when rejection of a paper is recommended. On the other hand, this option can increase the transparency and the reliability of the peer review procedure.
Some further research might focus on the reasons why reviewers decide not to reveal their names, especially when they suggest rejection of a paper.