Review Touchstones

General Review Criteria

When publications provide review criteria to their reviewers, those criteria should be thoroughly considered before the review is started. Doing so gives the reviewer a framework for writing their review and allows the AE and the author to more easily understand any critiques.

Although review criteria can be specific to a publication, there are some general items that a reviewer should keep in mind:

Relevance. The research presented in the paper should be within the scope of the field and the publication.
Significance, novelty, and originality. The research should be important to the field as well as novel and original.
Soundness, validity, and accuracy. The research should have valid methology and be accurate.
Clarity. The research should be presented in a manner that is clear and easy to understand.
Reproducibility. Research should be reproducible. Reproducibility ensures that the methods used in the research are transparent and can be repeated by others, which helps establish the validity and reliability of the results. When reviewing a paper, the reviewer should notice whether the methods section is detailed enough to allow for an evaluation of the reproducibility.

Research Integrity

While conducting a review, the reviewer should take notice of any potential ethical issues, some of which are listed below:

Self-plagiarism. Authors should always communicate when they reuse and/or expand upon their own work. For extended versions of previously published work, the paper must be substantially revised. Generally, this means that 25% of the paper should be material not previously published, to avoid what ACM would deem self-plagiarism.
Excessive self-citation. When an author occasionally cites their own work in a paper, it is not problematic; however, doing it excessively can artificially inflate the impact of an author’s work and can be seen as self-promotion. Furthermore, it may indicate other problems with the manuscript, such as downplaying or ignoring others' contributions.
“Ghost” or "guest" authorship. A paper should be written and revised by one, some, or all the authors listed on it. Look for inconsistencies in writing style or tone, a lack of clarity about the authors' contributions, or an absence of acknowledgments for writing assistance.
- Ghost authorship is defined as co-authorship given due to reputation or influence of someone who did not contribute significantly to a Work in order to increase the likelihood of acceptance of the submitted Work.
- Guest authorship occurs when co-authorship is given as a reward, payment, or incentive to someone who did not contribute significantly to a Work. This is otherwise known as "quid pro quo authorship." An example would be accepting an offer to be listed as co-author on a Work in exchange for performing a pre-submission review of the Work.
Tortured phrases. When a paper contains oddly worded phrases that may be the result of words getting translated from English into a foreign language and then back into English, such as “counterfeit consciousness” instead of “artificial intelligence”. See Retraction Watch for more examples.

Content-Specific Ethical Issues

There are ethical issues that are specific to the content of the research, such as:

Human subjects. If the research involves human subjects, there may be ethical considerations related to informed consent, privacy, and confidentiality.
Decision software. The use of computer systems in research may raise questions about bias and fairness. If the system is designed to make decisions or predictions about human behavior, there may be concerns about whether the system is biased against certain groups or whether it is making accurate predictions.
Data collection. There may be ethical concerns related to the use of data collected during the study. If the data is being used to develop or improve a commercial product, there may be concerns about who owns the data and how it will be used.
Population bias. Ethical concerns should be raised when a paper shows bias toward or against a particular population. Research about or affecting any population, regardless of location or socioeconomic status, should be impartial in nature.

Data or Figure Manipulation

A reviewer can detect data or figure manipulation by carefully examining the data and figures presented in the manuscript. Some ways a reviewer can detect such manipulation include:

Comparing similar experiments or data sets to see if there are any inconsistencies or anomalies
Checking the statistical analyses to ensure that they are appropriate and correctly applied
Examining the raw data to see if it matches the presented figures and graphs
Checking for any unusual patterns or similarities between different data sets or figures
Looking for any evidence of image manipulation, such as duplicated or altered images

Machine-Generated Writing

ACM allows the use of generative artificial intelligence (AI) tools by authors in some situations, and such use must be disclosed.

Disclosure can be a footnote for small portions of text, a note in the acknowledgments section, or an appendix or supplementary material document for large portions of the submission.
The use of software to improve the quality of author-generated text, such as a word processing program or an online editing system such as Grammarly, does not need to be disclosed.

Reviewers should be aware of the possible use of AI in the event that such use is not disclosed, and they should notify the handling editor of suspected, undisclosed AI use. Indications of such use can include:

There are inconsistencies in style or tone.
The text contains unusual phrasing or vocabulary.
There is a lack of coherence in the writing from section-to-section, paragraph-to-paragraph, and/or between the writing and the charts, figures, and/or data presented in the paper.
The writing uses repeated phrases or sentences.
There are errors or inaccuracies related to the field or topic being discussed, indicating a lack of understanding of the subject matter.

🔎 Case Study: Machine-Generated Writing 🔎

Dr. Tanaka was assigned as a reviewer for a paper submitted to an ACM publication. As she began reading the paper, she noticed some things about the writing that gave her pause, such as that key experimental details were missing from the Methods and the same phrases were used in several places throughout the text, among others. The figures also had some irregularities.

Dr. Tanaka suspected that the author had used an LLM to generate the text, and even the figures, of the paper.

How should Dr. Tanaka proceed? Click here to find out.

Reporting Ethical Violations

A reviewer may encounter evidence of possible misconduct in the process of reviewing a paper. Violations can fall into several categories, for example:

Plagiarism
Redundant publication or self-plagiarism
Author misrepresentation
Content falsification
Not disclosing use of LLMs

It is important that any suspected violation be reported to ACM. ACM provides a web-based form for reporting violations here. In many cases, the reviewer should first consult with the individual who assigned them to review, and that individual, likely in consultation with the EIC and/or PC Chairs, will determine the appropriate course of action and who, if anyone, should report the violation. When reporting a claim, the the following information should be provided:

Their name
Relationship to the work
Detailed written description of claim
Detailed evidence supporting the claim

It is helpful in an investigation for the reviewer to provide copies of the potentially plagiarized work, how to obtain unpublished work, and any other relevant information.

Information regarding submitting claims and ACM’s subsequent process surrounding investigations can be found here.

🔎 Case Study: Reporting Ethical Violations 🔎

Dr. Schwartz is assigned as a peer reviewer for a paper submitted by a team of researchers led by Dr. Owusu. As she meticulously reviews the manuscript, she notices substantial data manipulation, with several images appearing to be duplicated or altered to enhance the study's results. This raised serious concerns about the validity of the research presented.

Despite recognizing the severity of the ethics violation, Dr. Schwartz hesitates to report the misconduct. Dr. Owusu is a prominent figure in the field, and if he recognizes that Dr. Schwartz authored the review, she fears there would be retaliation, including damage to her professional reputation, or even strained relationships with colleagues. The fear of reprisal leaves her uncertain about how to address the ethical breach.

How should Dr. Schwartz proceed? Click here to find out.

Back | Next

Module 3

►Review Touchstones

General Review Criteria

Relevance
Significance
Soundness
Clarity
Reproducibility

Research Integrity

Content Specific Ethical Issues
Data or Figure Manipulation
Machine-Generated Writing
CASE STUDY: Machine-Generated Writing

Module 3: Review Touchstones

Review Touchstones

General Review Criteria

Research Integrity

Content-Specific Ethical Issues

Data or Figure Manipulation

Machine-Generated Writing

🔎 Case Study: Machine-Generated Writing 🔎

Reporting Ethical Violations

🔎 Case Study: Reporting Ethical Violations 🔎

Module 3

►Review Touchstones

Module 4: Evaluating the Paper

Module 5: Submitting Your Review

Module 6: Artifact Review and Badging

ACM Peer Reviewer Certification Exam

COMPLETE

✓ Module 1: Peer Review Overview
✓ Module 2: Assessing Your Suitability to Review

Module 3: Review Touchstones

Review Touchstones

General Review Criteria

Research Integrity

Content-Specific Ethical Issues

Data or Figure Manipulation

Machine-Generated Writing

🔎 Case Study: Machine-Generated Writing 🔎

Reporting Ethical Violations

🔎 Case Study: Reporting Ethical Violations 🔎

Module 3

►Review Touchstones

Module 4: Evaluating the Paper

Module 5: Submitting Your Review

Module 6: Artifact Review and Badging

ACM Peer Reviewer Certification Exam

COMPLETE

✓ Module 1: Peer Review Overview ✓ Module 2: Assessing Your Suitability to Review

✓ Module 1: Peer Review Overview
✓ Module 2: Assessing Your Suitability to Review