Reproducibility at SIGCOMM

Last modified by Damien Saucez on 2017/06/14 08:22

What is this Wiki about?

Ensuring the reproducibility of results is essential in experimental sciences. Unfortunately, as highlighted recently, a large proportion of research results are hardly, if not at all, reproducible, raising reasonable doubts on the research lead by scientists around the world.

To address this important issue, the ACM has introduced a new policy on result and artifact review and badging. The policy defines clearly the terminology to be used to assess results and artifacts but does not specify the review process.

The goal of the workshop is to craft recommendations on research results and artifacts review process for conferences and journals of the SIGCOMM interest group. To achieve this goal, the workshop is structured as a forum where presenters will discuss with attendees their experience in reviewing research results and artifacts.

Topics of particular interest include, but are not limited to:

  • Reviewing experience;
  • Enabling environments and tools for reproducibility;
  • Methods and tools to anonymize datasets;
  • Testbeds for reproducible research;
  • Platforms to share experimental results and artifacts.

The workshop will try to answer questions rose by the ACM such as:

  • Should reviews occur before or after acceptance of a paper?
  • How many reviewers should there be?
  • Should the reviewers be anonymous, or should they be allowed to interact openly with the authors?
  • How should artifacts be packaged for review?
  • What specific metrics should be used to assess quality?

To prepare the workshop we propose this open forum.

The official workshop webiste is

Let's start!

This workshop is yours but to bootstrap the discussions, we provide here a list of papers presented at ACM CoNEXT 2016 and for which we identified great artifacts.

If you are authors, don't hesitate to add your papers!

Questions for the workshop

During the workshop we will constitute workgroups aiming at answering questions specific to reproducibility and how to integrate it to the ACM SIGCOMM community. Please go here to propose your questions.


Created by Damien Saucez on 2017/01/24 15:14