Elements of a Good Review

Expertise

Reviewers are chosen because of their subject matter expertise. A reviewer with expertise has a high level of knowledge, skills, and experience in the subject and is often recognized as an authority in their field.
They have a deep understanding of the theories, concepts, and practices within their field, and can apply this knowledge to solve complex problems or advance the field through research and innovation.
This expertise allows a reviewer to provide insightful and constructive feedback as well as identify any potential flaws or limitations in the methodology or analysis.
A potential reviewer may not have the exact expertise required but could still meaningfully contribute to the review process. In this case, the reviewer is encouraged to contact the handling editor or Program Committee Chair(s) who will determine if the reviewer should proceed with the review.

Conferences vs. Journals
For the conference review process, reviewers can sometimes bid on papers to review (typically via a Likert scale, but also via interest and expertise). Papers can also be assigned by a matching algorithm that uses existing published papers as input. Sometimes reviewers provide areas of expertise to help with the matching.

For the journal review process, reviewers have an account in the peer review system of a respective publication with a series of keywords selected indicating their expertise and interests. Handling editors will generally select reviewers using these keywords and/or their own knowledge of the reviewer.

Impartiality

Being an impartial and fair reviewer requires an unbiased evaluation of the research by setting aside any personal biases or preconceptions and evaluating based solely on quality, accuracy, and relevance.

Approaching the review process with an open mind allows the reviewer to provide constructive feedback, including both the strengths and weaknesses, in the form of clear and specific suggestions for improvement.

Things a reviewer should reflect on before starting their review:

Their eligibility: Ask themself if they know the author or are they in categorical opposition to the premise or content of the paper. If yes, they should contact the handling editor/PC Chair.
Their implicit biases: Ask themself what expertise they are bringing to the paper, and what expertise they lack that might be needed. Are there aspects of their identity or outlook that might limit their ability to interpret the work's methods and significance?

Inclusivity

It is important for reviewers to keep inclusivity in mind when reviewing a paper by considering a wide range of perspectives and experiences when evaluating research.

The reviewer must assess the research based solely on its quality, accuracy, and relevance, regardless of the background or identity of the authors or at what institution the research was conducted.

Reviewers should be aware of potential biases or knowledge gaps that may exist in the research and consider how the research may impact different populations, including those who may be underrepresented in the field.

“Anyone, from any background, should feel encouraged to participate and contribute to ACM. Differences – in age, race, gender and sexual orientation, nationality, physical ability, thinking style and experience – bring richness to our efforts in providing quality programs and services for the global computing community."
—ACM official statement on Diversity, Equity, and Inclusion

Confidentiality

To maintain confidentiality, a reviewer must not share any identifying or substantive information with others about the paper being reviewed—including with colleagues, friends, or family. Submissions may not be used for training in reviewing and pre-published results may not be used for any reason.

PC members may not access and/or download in bulk submissions for any purpose other than to perform a review.

The practice known as “sub-reviewing” or “co-review” is permitted only with the explicit written approval of the Editor(s)-in-Chief, Senior Associate Editor, Program Committee (PC) chair(s) of the publication, or other individual authorized to assign reviewers on behalf of the publication. It is a requirement that the identity of the sub-reviewer be declared in writing to those approving the written request to utilize a sub-reviewer. The reviewer shall be responsible for the content of the review performed by the sub-reviewer.

The identities of the reviewers must not be revealed to the authors during review processes that are not “open”; in double-anonymous peer review, the authors’ identities are hidden from the reviewer as well.

In a double-anonymous review, the reviewer should not actively try to de-anonymize the submission (e.g., by performing an internet search on the paper’s title which may lead to a preprint version of the paper with identifying information).

The reviewer should not communicate directly with the authors of the paper, and any feedback or comments must be provided through the appropriate channels. The reviewer also should avoid any language or statements that could reveal their identity.

Reviewers should not disclose information about a paper’s confidential previous review history which they may know because of their role as a reviewer or decision maker at another publication. If there is supposition of plagiarism, simultaneous submission, or other research misconduct, the reviewer should notify the individual who assigned them to review.

Reviewers may not upload confidential ACM submissions into any generative AI or LLM system or any system managed by a third party that does not promise to maintain the confidentiality of that information. However, reviewers are permitted to use enterprise versions of LLMs or other third-party systems to improve the quality and readability of their reviews provided the system ensures the confidentiality of that information.

🔎 Case Study: Co-review/Sub-review 🔎

Alex, a graduate student for Dr. Kovac, was asked to perform the review of a paper Dr. Kovac was invited to review. Alex, eager to gain experience and please his mentor, diligently evaluated the manuscript. However, the AE was not informed.

After sending his completed review to Dr. Kovac, who then submitted the review to the journal, Alex began to be concerned about the unreported delegation. Would his input be perceived with the same level of credibility as a review conducted by a seasoned researcher? What did the lack of acknowledgment say about a potential imbalance of power between him and his mentor?

How should Alex proceed? Click here to find out.

Timeliness

Submitting reviews before or on the due date is essential to ensure the integrity of the peer review process and maintain the publication schedule of the publication or publisher.

When accepting an invitation to review, a reviewer agrees to complete their evaluation of the paper in the timeframe specified in the reviewer instructions.

If there are any issues or concerns that may affect their ability to complete the review on time, the reviewer must communicate with the publication’s editorial office.

If the reviewer determines they are no longer able to complete the review, it is important that they contact the publication’s editorial office as soon as possible so that another reviewer can be secured.

Conferences vs. Journals

Conference submissions are typically reviewed by a smaller group of experts, or a Program Committee (PC), and submissions are evaluated for relevance to the conference theme, novelty, and potential to contribute to discussions. Submissions are typically organized in batches and papers are often reviewed more quickly to accommodate a conference schedule.

Journal peer review is often a more extensive and time-consuming process, including multiple rounds of review.

Back | Next

Module 1: Peer Review Overview

►Elements of a Good Review

Expertise
Impartiality
Inclusivity
Confidentiality
CASE STUDY: co-review/sub-review
Timeliness

Levels of Anonymity and Confidentiality in Peer Review
ACM Publication Policies
ACM Peer Review Workflow

Module 1: Peer Review Overview

Elements of a Good Review

Expertise

Impartiality

Inclusivity

Confidentiality

🔎 Case Study: Co-review/Sub-review 🔎

Timeliness

Module 1: Peer Review Overview

Module 2: Assessing Your Suitability to Review

Module 3: Review Touchstones

Module 4: Evaluating the Paper

Module 5: Submitting Your Review

Module 6: Artifact Review and Badging

ACM Peer Reviewer Certification

Case Study: Co-review/Sub-review