Evaluating the Paper

Methodology

Every research paper should have a detailed Methods section where the author describes the materials used in the study as well as how the study was conducted.

A reviewer should evaluate whether the methods make sense with respect to the research being conducted and whether there are any limitations that affect the outcome of the study.
Reproducibility is one of the hallmarks of solid research. Based on the description of the methods, it should be clear whether another researcher can reproduce the work.

Figures

Figures often illustrate the methods or results and can give the reader a visual of the author’s discussion of the results. The reviewer should keep the following points in mind:

The figures should be a necessary element of the paper. If the information presented in the paper is sufficient to communicate the methods/results, then a figure that serves the same purpose may not be necessary.
The information in the figures should reflect what is discussed in the text without confusing the reader. Figures should contain no surprises.
The image should be clear and readable.
The reviewer should be aware of possible image manipulation or duplication without permission.
The captions for the figures should not be too wordy or have information irrelevant to the figure. A figure with multiple parts should have an explanation of each part.
Third-party images should be appropriately credited and cited.

Tables

Tables present data and results in tabular form. When evaluating the tables in a paper, the reviewer should keep the following in mind:

The tables should be necessary to the paper and reflect the information discussed in the text.
Although the methods and results can and should be summarized in the text, a table presents specific data.
Tables should be organized in a simple and logical manner.
All columns should have headings that are relevant to the data in the columns.
The table captions should not be too wordy.
Tables can also have footnotes that serve as additional explanation of items in the table body or caption.

Results and Analysis

The Results section of a paper should present the results of the study in a logical order that follows the order of the Methods section.

The reviewer should be able to refer to the information presented in the Methods when reading the Results, and no information should be missing between the two sections.
The reviewer should be able to refer to the Introduction to ensure any questions raised there have been answered in the Results. It is also important that there are no results presented that do not have any basis in the data presented.
When evaluating the results, the reviewer should consider the statistical methods used in the study. Those methods should be appropriate to the experiment.
It is also important for the reviewer to consider the robustness of the data. Although some authors provide supplemental files containing the raw data, there should be enough data points in the paper to demonstrate reliable results.

Discussion and Conclusion

Discussion

The Discussion explains the meaning of the results achieved in the study.
The reviewer should evaluate whether the Discussion is in line with the Results and whether they agree with the author’s assessment of the research’s importance to the field.
Also important is for the author to point out in the Discussion any shortcomings of their study. The reviewer should be aware of whether the author includes this information.

Conclusion

The Conclusion section summarizes the information presented in the paper. It also gives the author the opportunity to present a final word on the study.
The reviewer should make sure the conclusion does not contain any ideas that have not been supported elsewhere in the paper.

References and Supporting Information

References

References to other published works that support the study are a necessity for conveying the solidity of the methods and legitimacy of the results.
Reviewers do not need to check each reference for validity. However, they should make sure that the references are in a consistent citation style, and the corresponding reference for each citation should make sense to what is being presented in that portion of the text.
As part of their review, reviewers should identify any relevant papers that are missing from the reference list, which may include some of the reviewer’s own papers.
Reviewers shouldn't criticize a paper for not referencing their own work, unless that work is objectively highly relevant. Inappropriate citation recommendations are prohibited and are treated as a policy violation by ACM.

Supporting Information/Supplementary Material

Some authors provide extra text, tables, or figures with their papers. These items are not necessary for the paper to be complete, and the reviewer should be able to conduct a thorough review without them.
The reviewer should view the supporting information anyway because it may provide more detailed data or references or even figures that may help in gaining a better overall understanding of the paper.

🔎 Case Study: References and Supporting Information 🔎

Dr. Thompson has recently published a paper in her field of study. Subsequently, she was invited to review a paper written by a prominent author group on a topic related to her own recent work. Her recently published paper has not been cited in the paper she is reviewing.

Dr. Thompson considers whether to recommend that the authors reference her work in their paper. She is aware of the importance of maintaining objectivity and avoiding any conflicts of interest in the peer review process. She also thinks it is possible that the authors may not be aware of her work on the topic because it was so recently published.

How should Dr. Thompson proceed? Click here to find out.

Back | Next

Module 4: Evaluating the Paper

Evaluating the Paper

Methodology

Figures

Tables

Results and Analysis

Discussion and Conclusion

References and Supporting Information

🔎 Case Study: References and Supporting Information 🔎

Module 4

►Evaluating the Paper

Module 5: Submitting Your Review

Module 6: Artifact Review and Badging

ACM Peer Reviewer Certification Exam

COMPLETE

✓ Module 1: Peer Review Overview
✓ Module 2: Assessing Your Suitability to Review
✓ Module 3: Review Touchstones

Module 4: Evaluating the Paper

Evaluating the Paper

Methodology

Figures

Tables

Results and Analysis

Discussion and Conclusion

References and Supporting Information

🔎 Case Study: References and Supporting Information 🔎

Module 4

►Evaluating the Paper

Module 5: Submitting Your Review

Module 6: Artifact Review and Badging

ACM Peer Reviewer Certification Exam

COMPLETE

✓ Module 1: Peer Review Overview ✓ Module 2: Assessing Your Suitability to Review ✓ Module 3: Review Touchstones

✓ Module 1: Peer Review Overview
✓ Module 2: Assessing Your Suitability to Review
✓ Module 3: Review Touchstones