Aries Systems Corporation

By Emily Babcock, Systems Support Services Coordinator for J&J Editorial

Wrangling a reviewer database can be a daunting undertaking. Regardless of its size, the average journal will have at least several hundred (if not thousand!) reviewers, all with differing academic backgrounds, qualifications, and perspectives. While this variety is one of the factors that makes the peer review system shine, there are numerous variables to take into account. And though you cannot control the reviewers themselves, a few thoughtful adjustments to your Editorial Manager® system can greatly improve your workflow. By enhancing the searchability, quality, and maintenance capabilities of your peer review workflow you can ensure that your journal delivers quality, trustworthy results to your authors.

J&J Editorial and Aries Systems have partnered to help Editorial Manager users take advantage of the full functionality of the system to address these specific issues. In honor of Peer Review Week 2020 and theme of “Trust in Peer Review”, below are some of our most recommended strategies to achieve high quality, trustworthy results:

Searchability – you won’t be able to deliver quality reviews if you can’t track down the right reviewers! Make sure your system has the proper framework to support quick and easy reviewer invitations.

• Developing a good system of Classifications or revisiting your existing list of Classifications is a great place to start. Free-text keywords are all well and good, but pre-determined Classifications allow the journal greater control over how both users and submissions are categorized. Better control leads to better results!
• Be sure to leave plenty of room in your Classification numbering scheme. We recommend using increments of 10 at a minimum or even 100 depending on the scope of your nomenclature.
• Require reviewers to select at least 3-5 Classifications during registration and require authors to do the same for their article during the submission process. With both reviewers and authors selecting classifications, the staff or editors handling reviewer invitations can easily search for matches with a click of a button.
• People Flags and People Notes. People Flags are searchable, reportable, and can serve as an independent identifier outside of Reviewer Role designations. They are easy to swap out, you can assign as many as you’d like to a profile, they attract the eye when scrolling through a reviewer list, and they add a fun bit of color and whimsy to your daily journal work!

People Notes are another useful feature that can provide more context to a reviewer’s profile. You can add any kind of notation you’d like, such as “Editorial Board Alumni” or “Expert on Animal Models”. We recommend using Detailed People Notes specifically, as this will attribute a specific operator and time stamp on the note, which gives added context as to why it was left.

Detailed People Notes are also searchable in Enterprise Analytics Reporting (EAR) and can be configured to appear in reviewer search results.

NOTE: While People Notes are useful, be cautious about the type of content included. Per GDPR regulations, a user can request the details of their profile at any time, including their People Notes. Additionally, People Notes can be made viewable for a variety of roles in the system. As such, including negative content is discouraged.

Quality Review Content – Now that you can find your reviewers, how can you guarantee that they deliver constructive, specific, and relevant feedback?

• Use Custom Reviewer Questions and Review Forms to ask for targeted feedback.
  • Example questions include, but are not limited to: “Is the methodology sound?”, “Are the references cited relevant to the study?”, “Should this article undergo statistical review?”, “Is the grammar usage accurate?”, and anything else you can possibly think of!
  • These can be simple Yes/No answers, scales of 1 – 10, or a variety of other answer value types.
  • Review Questions can be configured to appear on different kinds of Review Questionnaires, which can be customized to the Reviewer Role and Article Type. The answers to these questions can also be configured to populate directly into your author’s decision letter.
• Link your Reviewer Guidelines directly to the Reviewer Invitation Letters and Reviewer Form. This will help your reviewers better understand what’s expected of them up front.
  • If you don’t already have reviewer guidelines, consider drafting some! They are a powerful tool for attracting, engaging, and retaining a strong reviewer base.
  • We recommend coding the HTML to have the link open in a new browser tab or window so that your reviewers don’t lose their place on the review form itself or have to worry about navigating back and forth.
  • Per the Custom Question point above, you can make a custom question for reviewers to confirm that they have read and understood the guidelines. This provides you another area to insert the link and can draw additional traffic to those guidelines if a reviewer hasn’t looked at them yet.

Reporting and Maintenance – Regardless of the measures taken in the above two categories, quality control measures will still be required to maintain your reviewer pool.

• Enable Reviewer Rating functionality for Editors and make the ratings mandatory to complete a decision. All Editors can be required to rate reviewers on whatever scale your journal wishes to use and that data will be calculated into an average score over the reviewer’s lifetime in the system. Be sure to provide your editorial board with a scoring rubric to help enforce some consistency in how they score each reviewer. Many journals find using something similar to academic grading (A – F, 60 – 100) useful. This can be hard coded directly into the ratings configuration screen for quick reference as seen below.

• Consider creating qualitative Custom Review Questions for Editors to complete when filling out their decision form. While the review ratings are great from a quantitative standpoint, additional questions can be used to help with other editorial office initiatives to interact with their reviewers. For example, editors can be asked if a reviewer on the manuscript should be eligible for a special award or accolade of some kind. Or the editor could recommend the reviewer to participate in a mentorship program or focus group as a part of general journal strategy. The editorial office or society staff can follow up on these questions after a decision has been rendered and take the appropriate action.
• Familiarize yourself with the various reporting options that focus on reviewer performance and metrics.
  • Reviewer Performance Report: This will give you a quick table view of everyone who has reviewed in a given timeframe and also show you how often they’ve agreed/declined invitations, how often they’ve been late, how many reminders they typically receive, and much more! Found under the Reports > Summary and Annual Reports
  • Reviewer Thank You Report: A canned EAR report that will quickly pull a list of everyone who has reviewed in the preceding year and orders them based on number of completed reviews. Extremely useful for showing gratitude to your reviewers! Found under Reports > Enterprise Analytics Reporting > Standard Reports
  • Journal Accountability Report: While helpful for a myriad of things, this report can include general reviewer averages you can use to assess your general reviewer policies and tactics – average days turnaround, average days late, number of invitations, number of declined reviewers, and much more! Found under the Reports > Summary and Annual Reports

With these tips, tricks, and best practices in your toolkit, you will be well on your way to ensuring you have a trustworthy, streamlined, and high-quality peer review management process. After years of managing journals at J&J, I can attest that these process recommendations can produce a better review process, while creating greater satisfaction for users.

For more information on Editorial Manager configuration options, please contact your Aries Systems Account Coordinator.

Emily Babcock is the Systems Support Services Coordinator for J&J Editorial, an independent publishing services company dedicated to providing professional support to editorial offices, publishers, and societies. She oversees the business division managing client-driven configuration support projects as well as consulting projects that enhance a client’s efficient use of their journal management systems.