Aries Systems Corporation

As the golden standard for evaluating research, peer review is an essential part of the publishing process. Publications rely on knowledgeable research experts to volunteer their time and effort to conduct rigorous and impartial examinations of submitted content. This work is often unpaid and performed on top of their existing heavy workloads in order to advance their professional careers and research in their field. Additionally, the high influx of submissions often outweighs the capacity of publication’s pool of reliable Reviewers. Not only is this an unsustainable model long-term, but it also adds to the increasing pressures from business publishers for their editorial staff to reduce time to publication in order to boost productivity and research output. This often creates an environment where Editors opt for their trusted Reviewers, prioritizing familiarity and reliability over the effort of searching, evaluating, and inviting early-career researchers they are unacquainted with. Consequently, this poses a challenge for new researchers who aspire to gain experience as a Reviewer in their field. To support success in the future of peer review, it is critical for publications to build a pipeline of the next generation of strong Reviewers, foster coaching and mentorship opportunities for senior and junior researchers, and ensure they are guided and recognized along the way.

Although the backbone of the peer review process, Reviewers tend to get less focus than the Author and Editor roles during the editorial workflow. To further clarify the gaps and brainstorm possible Aries- and publication-lead solutions, Aries explored this topic during the new Discovery Roundtable session at this year’s Editorial Manager User Group meeting (EMUG 2023). An interactive workshop designed to help inform our product development roadmap, the Discovery Roundtables session included facilitated group conversations focused on three key topics. In honor of Peer Review Week (PRW) 2023, which is dedicated to the theme “Peer Review and the Future of Publishing,” we are highlighting discussions held on each topic at EMUG 2023 in a three-part blog series. As the third and final installment in the series, this summary delves into the thoughts and opinions shared by the Editorial Manager user community on fostering mentorship in peer review.

Using the “Customer Journey Map” methodology to structure the discussion, the groups focused on the junior and senior Reviewer personas and these three major stages in the peer review workflow: Pre-Review (getting invited to review), During Review (conducting and submitting the review), and Post-Review (receiving feedback, recognition, and insights). Insights gained from EMUG serve as valuable resource for identifying areas within EM that may require enhancements and evaluating the feasibility of any proposed solutions for inclusion in the Aries roadmap. To help fuel ideas, we also asked attendees to consider the following:

• When a Reviewer declines, do you often receive suggestions for who may be a qualified replacement? How often do Editors take onboard those suggestions and what do they use to validate suggested replacement (e.g., Reviewer Ratings)?
• What recognition opportunities do you have in place for Reviewers and do you already have a mentorship program in place?
• Do you ask Reviewers to disclose if they had assistance with the submitted review and, if so, do you have a process for including this person as a potential Reviewer for future submissions?
• Would you consider using Reviewer Ratings as a mode of identifying Reviewers that could benefit from mentorship? Alternatively, would you use this to match high-value Reviewers with junior Reviewers? What rating questions would be of most interest in making this determination? (e.g., what do your Editors find most valuable in review content?)
• Can you identify any tools that may help Reviewers submit quality reviews?

Starting at the first journey stage, “Getting Invited to Review,” attendees noted likely scenarios in which both junior and senior Reviewers would receive an invitation to review in their field – many of which are related to networking, participation, and connections in the industry. These include attending conferences, being a postdoc, authoring a paper and submitting to a journal in which they would like to review for, directly emailing a journal to volunteer their services as a potential Reviewer, being proposed by the submitting Author, being a member of industry associations or databases such as Scopus or Web of Science, and more. A few ideas proposed put the responsibility on the publisher persona, such as conducting “open calls” for Reviewers as a way to increase diversity and minimize selection bias, and establishing a formal “buddy system” in which junior and senior Reviewers are invited and partnered together to split the work and recognition of the review as a form of mentorship and coaching, and being mindful that most Reviewer search/matching tools surface candidates with the highest rankings and experience at the top and that early-career researchers that deserve a chance may appear lower on the results roster. While there is room for improvement in this space, it was made clear from the discussion that the focus areas are further downstream in the workflow.

There are more available opportunities to enhance Reviewer success during the “Conducting and Submitting the Review” stage. Many attendees stated they require the Reviewer to enter personal keywords and classifications upon registration to EM, which can help with discoverability and matching for future submissions. Attendees also expressed interest in implementing a more structured approach to encouraging Reviewers to independently update their keywords/classifications as their expertise and experience grows, either through manual notices from the editorial office or automatic pop-up prompts from the system. Many of the groups put an emphasis on the theme of training and guidance for Reviewers. Using custom instructional text, customizing the Reviewer form, and drafting clear email communications is critical to ensuring the Reviewer is informed of all expectations prior to starting their evaluation. Embedding or linking helpful resources such as tutorial videos, online help portal, and documentation through HTML or in-product tool tips on pages such as the Reviewer Main Menu and Submit Review and Comments are also channels in which publications can offer support.

Training programs like Web of Science Reviewer Certification can help increase researcher discoverability, however, several groups at EMUG voiced a potential opportunity for this to be leveraged. Publications can prompt their Reviewers to indicate in their EM profiles that they have completed certifications, which can then be pulled by Editors when inviting Reviewers through a new field or through the use of flags. Furthermore, attendees emphasized a desire for a more formal way of connecting mentors with mentees. Some users noted that existing EM functionality can be leveraged, such as enticing senior researchers who are already invited to review to recommend junior researchers via a custom question in EM, or using Additional People Details custom fields can be used to identify interest in being a mentor or mentee. While not guaranteed viable solutions, additional ideas attendees raised that would fit their “ideal world” workflows included discussion forums between senior and junior Reviewers or direct content commentary/review, Reviewer chains (similar to existing Editor chains) in which a senior Reviewer can automatically invite a junior associate to complete the assignment, and ways to link junior and senior review data together to track performance.

Aries staff and users identified that the “Post-Review” stage in the journey map was where the most improvement can be made towards Reviewer advancement. Once the Reviewer has completed their analysis, it is uncommon for them to receive additional communications from the journal beyond a “Thank you” for their services. To expand and solidify the loop of information at the end of the workflow, publications can establish and maximize opportunities for formal feedback, recognition, or insights from the journal either on their performance or the fate of the manuscript. Attendees offered many ideas to facilitate feedback channels, both through existing EM functionality and by suggesting entirely new potential solutions. One group recommended using the Decision Term as the basis for triggering a Reviewer evaluation, while another group proposed inserting the Reviewer’s ratings into the Thank You letter via merge fields based on data pulled from Additional Manuscript Details (AMD). Similarly, it was suggested that the Reviewer rating be expanded to include mentor/mentee ratings, and that ratings would be more flexible and adjustable to include options like a 5-point Likert scale rather than just the existing 1-100 scale. Additional brainstormed ideas included the ability for Reviews to keep a record of their review as to later return to critique themselves or use as a reference when reviewing revised manuscripts, the ability to have people flags assigned automatically based on Reviewer performance statistics in EM, and the ability for Editors to leave direct notes/commentary on archived records of reviews to praise and offer advice for future improvements.

Along a similar vein, it was indicated during the discussion that closing the loop on the paper itself was also important to the Reviewer. Given the Reviewer had dedicated significant time and effort towards the manuscript, it is expected that they would feel interested in the outcome of the paper after they have completed their review. Publications can set up automated letters to Reviewers on the Editor’s decision, when it was published (if accepted), what channels it was being distributed/promoted, access to the final piece (if allowed), and potentially even the performance of the piece (e.g. how many readers/reach, is it being cited in other works, has it been retracted or corrected post-publication, etc.). Shifting to Reviewer acknowledgement, many users at EMUG indicated they have at least one method of offering recognition to their Reviewers. Most commonly, publications run regular reports and deposit this activity to third-party platforms such as Kudos, ORCID, ReviewerCredits, and Publons/WOS or post a shout-out to the Reviewers on social media during industry events (such as Peer Review Week). Additional recognition ideas proposed by the groups include establishing publisher-specific incentive programs (that include perks such as waived open access APCs), yearly promotion to editorial board and to the associate Editor based on high-performing reviews, use of identifiers on badges at society annual meetings to recognize reviewers from other journals, offering certificates, and more!

“As we celebrate Peer Review Week 2023 this week, it’s valuable to reflect on our time at EMUG 2023. Our new Discovery Roundtables session was a hit!” stated Beth Fitzhenry-Neale, Aries Product Manager. “Through collaborative brainstorming alongside our EM customers, we identified the potential key elements needed to advocate for the next wave of Reviewers – who are the foundation for the future of scholarly publishing itself.”

“Partnering with both publishers and their Reviewers to facilitate mentorship, recognition, and outreach in peer review demonstrates our commitment to collaborative solutions,” continued Jaclyn Fitzgerald, Aries Product Manager. “Together, we can cultivate and embrace a future in which there is more robust support dedicated to their success, during Peer Review Week and beyond.”

For Editorial Manager/ProduXion Manager users unable to attend EMUG 2023, we invite you to please share your thoughts, ideas, and feedback with us on this topic to contribute to our strategic product roadmap!

Peer Review Week (PRW), an annual, community-led global event, explores topics related to current trends, advancements, challenges, and opportunities related to peer review as part of the larger publishing and research ecosystem. Held September 25-29, PRW 2023 will celebrate “Peer Review and the Future of Publishing,” earning the top vote through a worldwide poll of the scholarly community against proposed themes. Aries Systems is a proud member of the PRW Steering Committee. Follow along with our three-part blog series this week!

Get Involved!

#PeerReviewWeek

#FutureofPublishing

Twitter: @PeerRevWeek

LinkedIn: @Peer Review Week

Join the Committee: contactprweek@gmail.com

*Suggested solutions and enhancements generated from the EMUG 2023 Discovery Roundtables workshop offer valuable input towards Aries’ strategic vision, but are not guaranteed to be implemented on the product development roadmap