Reproducibility

Ensuring the reproducibility of results is essential in experimental sciences. Unfortunately, as highlighted recently, a large proportion of research results are hardly, if not at all, reproducible, raising reasonable doubts on the research lead by scientists around the world.

To address this important issue, the ACM has introduced a new policy on result and artifact review and badging. The policy defines clearly the terminology to be used to assess results and artifacts but does not specify the review process.

This wiki is the follow-up of the ACM SIGCOMM 2017 Reproducibility Workshop () and the goal is to craft recommendations on research results and artifacts review process for conferences and journals of the SIGCOMM interest group.

To achieve this goal, the workshop was structured as a forum where presenters discussed with attendees their experience with reproducibility and addressed the following questions:

How to deal with artifacts
How to change mentalities

Reproducibility seen by authors

Two testimonials of ACM CoNEXT'16 authors:

  • David Hancock and Jacobus van der Merwe. 2016. HyPer4: Using P4 to Virtualize the Programmable Data Plane. In Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies (CoNEXT '16). ACM, New York, NY, USA, 35-49. DOI: https://doi.org/10.1145/2999572.2999607 see the testimonial
  • Yuliang Li, Rui Miao, Changhoon Kim, and Minlan Yu. 2016. LossRadar: Fast Detection of Lost Packets in Data Center Networks. In Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies (CoNEXT '16). ACM, New York, NY, USA, 481-495. DOI: https://doi.org/10.1145/2999572.2999609  See the testimonial

How to move on

The ACM provides a clear definition of artefact review and badging https://www.acm.org/publications/policies/artifact-review-badging, that's a good start if you are new in this topic.

We will use this wiki to share our ideas in order to provide a set of recommendations in the form of an editorial article in ACM Computer Communication Review.

Mailing list: http://sympa.inria.fr/sympa/info/reproducibility-sigcomm

 

Tags:
Created by Damien Saucez on 2017/01/24 15:14