Peer Review Citation Use Cases
The following use cases are supported by this profile:
- 1 Use Case 1 – Article Peer Review Open Review
- 2 Use Case 2 – Article Peer Review Closed Review
- 3 Use Case 3 – Conference Topic/Meeting Abstract Submission Review
- 4 Use Case 4 – General Review Service Recognition
- 5 Use Case 5 – University/Faculty and Department External Review
- 6 Use Case 6 – Annotation
Use Case 1 – Article Peer Review Open Review
An increasing number of publishers use open peer review. This can be mandatory or optional (for the authors and/or the reviewers). Additionally, sometimes the review may be public but the referee name is not, or vice versa, or both may be made public, or sometimes only the Editor’s name (who makes the decision to publish). A few referees will insist on signing their reviews even for a journal that otherwise uses anonymous peer review. Reviews may be co-‘authored’ by several reviewers. These open referee reports can be published as they are submitted (one by one), or combined with author and/or editor responses and with the other referee reports and published in one combined file on article publication. They can also be summarised into a single Editor-authored document and published together with the article. They can also take the form of reviews post-publication on articles already published in a journal elsewhere. There is a wide (and increasing) range of services that provides these types of reviews, ranging from anonymous to named, and from anyone being able to write the reviews to only invited experts. However, for the purpose of this project, it is probably best to only include invited expert post-publication review such that it still is ‘peer’-review. All these referee reports often come with a type of quantitative assessment, which will be standard across a journal, and often across a publisher, but the labels of these gradations will vary between publishers. Some of these reports are associated with unique identifiers but the majority are not. The various types of open reviews should be associated with the citation information of the article in question as it is typically already published when reviews are published.
Authors submit articles for publication to a journal. Reviewers are identified, sometimes from the list of journal Editorial Board members, or they may be selected outside of this list (often suggested by the authors or the Editor assigned to the article and checked for suitability) and then they are invited. Once they accept, the referees submit the reviews to the journal. These usually go to the assigned Editor who makes a decision on whether to publish the article based on this review in conjunction with other reviews from other reviewers. If the decision is that the article is not to be published then even if the journal uses open review, these referee reviews typically remain anonymous. Only if the article is published are the reviews published. In some journals though, the reviews submitted by the referees go straight to the journal who publish the reviews immediately alongside the article (which is already published), regardless of how positive or negative the review is. Hence all these reviews are published and there is no associated Editor decision.
Non-expert peer review (e.g. PubMed Commons comments, PubPeer, Reditt). Peer review on unpublished articles (e.g. on preprint services such as ArXiv, BiorXiv). Editorial Board members and Editors, if not specifically assigned to the manuscript in question.
Do we want to consider the author being able to be a source and the publisher verifies? Do we include Review Service reviews? e.g. Rubriq Many journals that use open review still have anonymous peer review for those articles they reject so we will need two mechanisms running for these journals. We may wish to not include the quantitative assessment information associated with the reviews given the breadth of gradations used.
Use Case 2 – Article Peer Review Closed Review
Most scholarly publishers still use a form of closed review. In this model, reviewers are usually selected by editor(s) who evaluate a submission separately and independently, at least initially. Editors or associate editors may also be used as a reviewer or in providing a recommendation to a higher editor or editor in chief for a decision. The reviews are generally not published with the paper, though in some cases, selected comments to large parts of the review may be eventually (e.g., elife; this borders on the open review file.) Some variations are: There may be multiple rounds of review, with the same or additional reviewers, before or more often after further revision or an author response. Usually in these cases, the reviewers see previous review comments, and if available author responses. This process can continue indefinitely. Some journals use an editorial board to help with pre-screening of papers. Some or all of the reviewers may be anonymous to the authors or chose to be anonymous. Some publishers or grant agencies require anonymity. In some cases, single reviews may be authored by multiple reviewers, although most editorial databases identify just one reviewer and don’t track joint reviews (usually handled in comments only) Some journals allow separate confidential comments to the editor(s) by the reviewers and/or separate confidential ratings. Some editorial management systems will close out reviewers who never return reviews but still indicate that the reviewer was asked (e.g., by entering “no response” as a review). In some cases, reviewers will return minimal, unhelpful comments. The amount of questions asked of reviewers varies greatly. Reviews can be of book proposals, scholarly papers, or internal reviews conducted by a manager (e.g., by agencies or companies). Grant proposal reviews are often handled similarly. Some conference reviews are also. The reports usually are only uniquely identified within each editorial management system. Reviewers, if not anonymous, are occasionally or haphazardly thanked by authors. Editors and associate editors are also only sometimes identified publicly as handling a paper, either by the authors or as an added note at the end of a paper (varies by journal/publisher).
The authors submits an article for publication, or a book proposal. The editor-in-chief or editorial staff assign an editor or associate editor (or both) to handle the paper. In some cases, there may be a pre-screening by the editors and/or editorial board which may lead to rejection immediately. If the paper is appropriate, the editor then selects and asks scientists to participate in a review within a deadline. Once the reviewer agrees, they are provided confidential access to the paper and related material, and mostly today, an interface to upload their review and any related materials. Once all the reviews are in, or the process is completed, the associate editor provides a recommendation to the editor, which may be in the form of an additional review, or the editor reaches a decision. In some cases an associate editor can act as a reviewer. After receiving a revision or response from the author, this process may be repeated. Some journals allow all the reviewers to see all of the other reviews and revise or add additional comments first.
Use Case 3 – Conference Topic/Meeting Abstract Submission Review
Many meetings and conferences use a peer review process for deciding the topics, papers, posters, or speakers that will be included in the meeting or conference. In some situations these reviewers are part of the organizing committee, and at times the reviewers are solicited from a broader pool. Once proposals are submitted, the reviewers read and comment on the proposals, often assigning both a quantitative and qualitative evaluation of the proposal (the review.) These reviews are often (though not always) shared with those submitting the proposal, though it is rare for the identity of the reviewer to be shared. Conference/meetings rarely have unique identifiers to distinguish them. Often even conference URLs are not persistent, making object reference for the review less specific than if an identifier or other unique attribute was present. In addition, while some larger meeting/conference organizers will have system-based approaches for tracking reviewer contributions, many use more informal processes, and may not have a system-based source for sharing review activities. Those performing reviews often review many (all?) of the proposals for a given conference time slot, track, or topic. As a result, review attribution is most applicable at the meeting/conference level (or sub-level), vs assigning attribution to specific proposals reviewed. In addition, conference/meetings are generally infrequent (annual or semi-annual), so even aggregated attribution over a relatively long period of time (say a year) is unlikely to be sufficient to maintain anonymity.
Individuals/organizations submit proposals, often using some form of a Request for Proposals (RFP) process. Reviewers are identified and selected by the meeting or conference organizer, and the proposals are distributed for review. The meeting/conference organizer collects the reviews done and shares these reviews with those who submitted the proposals. The conference organizer thanks the reviewers, and may keep them on a list of people to reach out to for future reviews. (OPEN QUESTION: Is this process this informal, or are there more formal processes that we can point to?)
This use case does not consider a system-to-system approach for sharing review information (quantitative/qualitative results) about a proposal with the person/organization that submitted the proposal. For the purposes of this use case, it is probably best to only consider the largest meeting/ conference organizers, perhaps focusing exclusively on those who use system-based approaches for managing reviewers and their reviews.
Use Case 4 – General Review Service Recognition
Acknowledgement of peer review service varies from organization to organization. While some institutions such as NIH list the names and affiliations of those reviewers who serve for a given individual study section meeting, many others do not provide that level of specificity, including NIH in other situations. A certain level of anonymity is expected by peer reviewers who offer their services, yet recognition for their efforts is also important as it is a critical factor in promotion/tenure considerations. A healthy compromise to address these competing desires is general review service recognition. This can be done in a number of ways. For example, acknowledgement of service can be included on an organization’s website or in a special page/section of a published journal or meeting program by simply listing and thanking those who served for a given period of time and/or for a general effort. This likely would be done annually for funding agencies or for large meeting organizers but could be done more frequently (quarterly, semi-annually) for publishers given the much larger number of manuscript submissions and reviews that they would attract.
This type of information could be shared annually or more frequently depending on the organization. For small funding agencies, a yearly “thank you for serving” posting on a website may be sufficient. For larger funding agencies such as NIH, a list of reviewers who participated in small review panels on a somewhat broad scientific topic (eg, behavioral neuroscience) is provided three times a year. Publishers may acknowledge their reviewers more frequently given the volume of work. Meeting organizers are more likely to list/thank their abstract/session reviewers in a special section of their meeting program.
This case is not appropriate for open article reviews or for federal government reviews where they are required by federal law to publish the names of reviewers for a specific study section or program.
Is a general acknowledgement sufficient? Would reviewers want more information provided, such as number of reviews/meetings completed per year? Would organizations want that type of information released?
Use Case 5 – University/Faculty and Department External Review
There is no recognition for external reviews provided for faculty and other organizational units that undergo periodic review. Most tenure and promotion processes, presumably, engage external reviewers. Many departmental and other academic organizational units are reviewed periodically (most commonly in a decennial period) and engage external reviewers. In each case, the reviewers are asked to process considerable amounts of information and, in the case of unit reviews, spend time on campus conducting interviews with stakeholders.
This information is not shared currently. If it is shared, it would likely be shared annually. Given the ease of connecting a reviewer to a faculty review case, it is unlikely that the specific university review source would be named. Though reported singly, they would be displayed without source attribution or just summed annually. In the case of unit reviews, the reviewer is known and sharing that information publicly would not violate any promised anonymity.
Would simple annual counts of faculty tenure/promotion reviews be sufficient?
Use Case 6 – Annotation
Annotation represents either a single action, or potentially a set of multiple actions linked together. For instance, a single “review” might be composed of a number of discrete interactions (a general overall review for an article, followed by a dozen sentence or paragraph level annotations providing specific feedback, and becoming the launch points for threaded conversation by others in those locations). Lets imagine that a known researcher with an ORCID, creates a detailed post-publication annotation at arXiv.org on a breaking paper on whether neutrinos can exceed the speed of light. This annotation then is voted up, and generates substantial follow on discussion both in threaded form in situ, and in a variety of external places subsequently. It becomes “known” as the place where key insights and questions were first raised about the premise of the underlying paper, and is as valuable a part of the academic record of that Researcher as any other citation or review. This annotation has (at a minimum) a URL, since all annotations (and all replies to all annotations) in the open annotation model are web objects. It may also have a DOI or other stable scholarly identifier.