Announcing MLRC 2025, our first in-person conference

We are excited to announce the 8th iteration of the Machine Learning Reproducibility Challenge, MLRC 2025, which will also be the first, in-person conference, hosted at Princeton University, New Jersey, USA on August 21st, 2025!

The Machine Learning Reproducibility Challenge (MLRC) is an annual conference for reproducibility research in the Machine Learning community. MLRC has been running as an online conference for the last seven years (v1, v2, v3, v4, v5, v6, v7). This limits the incentives to submit to the conference, as online mode doesn’t offer the authors to showcase their work and network among researchers in the same domain. We have been systematically trying to address this issue by improving the submission and publication process, and partnering with several conferences over the years, either by a workshop, or more recently through a Journal-to-Conference mode with NeurIPS for the last couple of iterations.

The success of the MLRC poster sessions at these conferences, and the recent success of COLM, motivated us to “graduate” MLRC into an in-person conference, starting this iteration. MLRC 2025 will be a one-day single track conference, with a mix of invited talks, oral presentations, and poster sessions. We hope the conference will provide the much needed avenue for discussing and disseminating reproducibility research and allow participants and attendees to network over a common goal of improving the science of Machine Learning through reproducible methods. We are excited to partner with Princeton University, and thank Princeton AI Lab & Arvind Narayanan for facilitating the venue.

As for the nomenclature of the conference, historically we have had one year backdated, as in MLRC 2023 actually happens in 2024, due to incorporating papers published in 2023. As we move on to be an in-person conference, to closer align with the format of ML conferences and also in favor of broadening our scope, we are therefore dropping the version 2024 and moving directly to MLRC 2025.

We therefore announce the call for papers for MLRC 2025. We invite submissions which conduct novel, unpublished research of reproducibility of machine learning methods and literature, including but not limited to :

• Methods and tools to foster reproducibility research in Machine Learning
• Generalisability of published claims: novel insights and results beyond what was presented in the original paper, from any paper (or set of papers) published in top ML conferences and journals.
• Meta-reproducibility studies on a set of related papers.
• Meta analysis on the state of reproducibility in various subfields in Machine Learning.

Submissions must be first accepted at TMLR to be considered in the MLRC 2025 Proceedings. Please read the author guidelines and submission guidelines from TMLR to get the submission format and to understand more about the reviewing process. Existing papers related to the scope (with reproducibility certification) already published at TMLR are also welcome for the consideration of the committee.

While TMLR aims to follow a 2-months timeline to complete the review process of its regular submissions, this timeline is not guaranteed. If you haven’t already, we therefore recommend submitting your original paper to TMLR by February 21st, 2025. We aim to announce the accepted papers by June 27th. We have set a cutoff deadline for accepting TMLR decisions one week prior to the announcement deadline, allowing ample time for you to ensure your paper has received the decision at TMLR, and update our forms accordingly. For logistical purposes, this date will be a hard deadline, and unfortunately we would not be able to accommodate any late decisions from TMLR post this date. Therefore, we encourage you to submit early to TMLR, and contact the TMLR Action Editors well in advance if your paper hasn’t been reviewed or is pending decisions. If you miss the cutoff deadline, we encourage you to still go through the TMLR review cycle, as then your paper once published will be eligible for the next year’s iteration (MLRC 2026). If you already have a relevant published TMLR paper which has not been showcased at MLRC 2023, you can directly submit it now to our system for consideration for MLRC 2025.

Important dates

• Submit to TMLR OpenReview: https://openreview.net/group?id=TMLR
• Deadline to share your intent to submit a TMLR paper to MLRC: February 21st, 2025 at the following form: https://forms.gle/REgwJQBP8ZXQEaJk7
• This form requires that you provide a link to your TMLR submission. Once it gets accepted (if it isn’t already), you should then update the same form with your paper camera ready details.
• Cutoff deadline for receiving TMLR decisions: June 20th, 2025
• Deadline for announcing accepted papers: June 27th, 2025
• Conference day: August 21st, 2025 at Princeton University, NJ, USA

In the following months, we will share more updates about the conference session, invited talks, program and registration. We are excited that this will be a first, in-person conference specifically focused on reproducibility in machine learning research, which will foster the research and discussion on reproducible methods, analysis, insights and further strengthen and promote the scientific understanding of Machine Learning.

We are looking for co-organizers and volunteers! If you wish to help us in organizing this in-person conference, or would like to nominate organizers / volunteers, please submit the following form. You can also contact us at mlrc-2025@googlegroups.com or reproducibility.challenge@gmail.com if you have any feedback / suggestions.

Looking forward to a successful conference next year!

Koustuv Sinha, General Chair

on behalf of the MLRC 2025 Organizers