Supporting the Reproducibility Challenge participants

At W&B, we strongly believe that research should be reproducible and accessible to all, so we’re excited to support people participating in the Reproducibility Challenge. We believe the initiative they’re running is a very important one, and we’re happy to do what we can to support participants, on our own accord.

Specifically, we're looking to support efforts to make sure every NeurIPS, ACL, CVPR, ECCV, ICLR, ICML, EMLNLP paper is reproducible and that the claims made are verifiable by tracking the model performance and predictions in W&B. If you'd like to help us achieve that, please join the #ml-reproducibility channel in our Slack community, where we're coordinating these efforts.

We realize some of these papers are computationally expensive to reproduce, and we're happy to offer participants $500 prize money per paper reproduced, as long as it meets the guidelines.

How to participate

See Reproducibility Challenge for more details.

• Pick a paper from this list. Join the #ml-reproducibility channel in our Slack community.
• Use existing code released by the authors, or create a minimal implementation, to verify the main claims and results made by the paper.
• Instrument W&B to your code (typically takes 15 mins)
• Write a short summary of your findings and ablation studies as a Report (typically takes an hour) – was the paper reproducible? Did you uncover any key insights?
• Ensure you meet the ML Reproducibility checklist criteria.
• Submit your findings:
  • to the reproducibility challenge (make submission), and
  • as a Report to the W&B Gallery.
• Selected authors will have their work featured in the Gallery and in the W&B community.

Timeline

See Reproducibility Challenge for more details.

• Final submission deadline: January 29th, 2021
  • You can submit your reports at any time before the deadline
• Authors notified about prize money: February 28, 2021
• Deep Learning Salon showcasing top reports: March 9, 2021