Registration and project submissions will be handled by our OpenReview Portal.

Claiming a paper

For this challenge, you first have to submit a claim on a NeurIPS accepted paper. In the portal, you will be able to see a list of papers which have been “claimed” and a list of papers which have not been claimed, or “unclaimed” for the challenge. We encourage you to select papers which are yet to be claimed, however, you can also claim a paper which has been claimed by a team.

Once you submit the form to claim a paper, we will post an anonymous comment to the paper which will show the claim made by your team. Thus, while claiming a paper, you can see how many teams are working on it.

Expect to complete the following information when you submit your claim:

  • Team Name (will be public) & Team Member information (will be private in the claim)
  • University / Institution (public)
  • Track : See Task Description
  • Reproducibility Plan: We expect you to write a short proposal (500 words max) of your reproducibility plan of action for the selected paper. In this proposal, write concisely how you would approach the problem. This proposal essentially helps to narrow down your deliverables regarding the challenge. This proposal will not be made public, neither it will be used in our reviews.
  • Application for Compute: If you are a student then you will be eligible for free CodeOcean GPU accelerated development host. CodeOcean has provided us a limited amount of free compute in addition to what is already available in the free tier for academic users. We will be granting the additional compute sources to teams on the basis of the Reproducibility Plan. Otherwise, you can ask your instructor to apply for Google Cloud Credits citing our challenge. Check Resources for more information.

Submitting your project

We will use the same OpenReview portal to submit your Reproducibility projects. Click on “Add Reproducibility Challenge NeurIPS 2019 Report” to submit your work. In the field of “NeurIPS paper ID”, paste the paper ID of the forum for the paper. (For example, if the link is https://openreview.net/forum?id=B1V7ccl2Br, then the corresponding paper ID is B1V7ccl2Br). Select the appropriate track you want to submit your report on.

Please check Important Dates for the deadline for submission of your report. Your report should be of maximum 8 pages excluding references, in NeurIPS style format. You can also submit supplementary materials. Code submission is mandatory along with your supplementary materials.

Reviewing Criteria

We will be selecting reproducibility reports which present a clear, well motivated, and thorough reproducibility effort. The report should very briefly discuss the main ideas of the paper and not simply re-iterate the main work. Your report should clearly mention the key findings of your effort, and if necessary include discussion / commentary / suggestions to the readers re-implementing this work on its replicability / complexity.

Reports will be reviewed on the following criteria:

  • Concise Problem Statement
  • Reproducibility code
  • Communication with authors
  • Hyperparameter Search
  • Ablation Study
  • Discussion on results
  • Recommendations for reproducibility
  • Clarity of writing