Artifact Review and Badging

An experimental result is not fully established unless it can be independently reproduced. ACM promotes the integrity of the research ecosystem through encouraging a formal audit for artifacts associated with the work published in ACM journals and conference proceedings.

Authors who agree to such additional review, and whose work meets established standards, are rewarded with appropriate labeling both in the text of the article and in the metadata displayed in the ACM Digital Library.

ACM Artifact Review and Badging Terminology

Artifact. A digital object that was either created by the authors to be used as part of the study or generated by the experiment itself (e.g., software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, scripts used to analyze results).
Repeatability. A researcher can reliably repeat their own computation or other results.
Reproducibility. An independent group can obtain the same result using the author's own artifacts.
Replicability. An independent group can obtain the same result using artifacts which they develop completely independently.

ACM Badges

ACM recommends three separate badges related to artifact review be associated with research articles in ACM publications:

Artifacts Available
Artifacts Evaluated: Functional or Reusable
Results Validated: Reproduced or Replicated

These badges are considered independent and any one, two, or all three can be applied to any given paper depending on review procedures developed by the journal or conference.