Artifact Review and Badging
An experimental result is not fully established unless it can be independently reproduced. ACM promotes the integrity of the research ecosystem through encouraging a formal audit for artifacts associated with the work published in ACM journals and conference proceedings.
Authors who agree to such additional review, and whose work meets established standards, are rewarded with appropriate labeling both in the text of the article and in the metadata displayed in the ACM Digital Library.
ACM Artifact Review and Badging Terminology
- Artifact. A digital object that was either created by the authors to be used as part of the study or generated by the experiment itself (e.g., software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, scripts used to analyze results).
- Repeatability. A researcher can reliably repeat their own computation or other results.
- Reproducibility. An independent group can obtain the same result using the author's own artifacts.
- Replicability. An independent group can obtain the same result using artifacts which they develop completely independently.
ACM Badges
ACM recommends three separate badges related to artifact review be associated with research articles in ACM publications:
- Artifacts Available
- Artifacts Evaluated: Functional or Reusable
- Results Validated: Reproduced or Replicated
These badges are considered independent and any one, two, or all three can be applied to any given paper depending on review procedures developed by the journal or conference.
Example of ACM Conference Badging Workflow
Module 6
Artifact Review and Badging
Artifact Review and Badging
Repeatability
Reproducibility
Replicability
ACM Badges
Example of ACM Conference Badging Workflow
ACM Peer Reviewer Certification Exam
****
COMPLETE
✓ Module 1: Peer Review Overview
✓ Module 2: Assessing Your Suitability to Review
✓ Module 3: Review Touchstones
✓ Module 4: Evaluating the Paper
✓ Module 5: Submitting Your Review