Code of Ethics

MLRC utilizes COLM Code of Ethics, which itself utilizes ICLR Code of Ethics, which is restated here.

Responsible Stewardship The Machine Learning Reproducibility Challenge (MLRC) is committed to promoting good conduct and a reflexive and responsible approach to research and its applications, especially with respect to research and development related to reproducibility, methods, investigations and tools in machine learning and artificial intelligence. This Code is a vital part of our continuing effort to encourage reflection of the wider impacts of work that is considered by the conference, and to encourage work that consistently supports the responsible stewardship of trustworthy research that advances knowledge, public good and well-being.

Aims This Code of Ethics provides general ethical principles, applicable to both individual researchers and to organizations that carry out, fund, host, or are otherwise involved in research, and associated with COLM. The Code should not be seen as prescriptive but as a set of principles to guide ethical, responsible research.

Positive Action This code applies to all contributors to MLRC, including reviewers, authors, speakers, organizers of the conference, tutorials, workshops, sponsors, and attendees. MLRC’s contributors are expected to acknowledge this code, and whenever necessary, include a discussion in their contributions that expands on the wider impacts of their work, using this Code as one source of ethical considerations.

General Ethical Principles

Contribute to Society and to Human Well-being

  • Researchers must acknowledge that all people globally are stakeholders in computing, and that we should use our skills for the benefit of society, its members, and our natural environment.
  • Research should minimize negative consequences, including threats to health, safety, personal security, and privacy. In order to do so, it must take into consideration a multiplicity of socio-economic factors and geographies.
  • When the interests of multiple groups conflict, the needs of those less advantaged should be given increased attention and priority.
  • Researchers should consider whether the results of their efforts will respect diversity, will be used in socially responsible ways, will meet social needs, and will be broadly accessible.

Uphold High Standards of Scientific Excellence

  • Researchers and organizations should strive for excellence when conducting research and aim to produce and disseminate work of the highest quality. This implies a commitment to open enquiry, intellectual rigor, integrity, and collaboration.
  • Findings must be reported accurately and honestly. Researchers must not make deliberately false or misleading claims, fabricate or falsify data, or misrepresent results. Methods and results should be presented in a way that is transparent and reproducible.
  • Where human subjects are involved in the research process (e.g., in direct experiments, or as annotators), the need for ethical approvals from an appropriate ethical review board should be assessed and reported.
  • All contributions to the research must be acknowledged, and agreements relating to intellectual property, publication and authorship must be complied with.

Avoid Harm

  • Here, “harm” means negative consequences. Well-intended actions, including those that accomplish desired outcomes, may lead to harm.
  • When that harm is unintended, those responsible are obliged to undo or mitigate the harm as much as possible. Avoiding harm begins with engaging with application domain experts, engagement with the communities that the research is intended to serve, and a careful consideration of potential impacts on all those affected.
  • When harm is an intentional part of the system, those responsible are obligated to ensure that the harm is ethically justified.
  • Harm to the natural environment, whether in the process of producing research or in its application, should also be considered. In all cases, ensure that all harm is minimized.
  • The consequences of data aggregation and emergent properties of systems should be carefully analyzed, including those that can become integrated into the structure of society. Researchers have an additional obligation to report any signs of system risks that might result in harm. For reporting, see the section at the end on Concerns and Remediation.

Be Honest, Trustworthy and Transparent

  • Researchers should be honest about their qualifications, and about any limitations in their competence to complete a task.
  • Researchers should provide full disclosure of all pertinent system capabilities, limitations, and potential problems to the appropriate parties, including any party that may deploy the system.
  • Researchers should be open and transparent about any circumstances that might lead to either real or perceived conflicts of interest or otherwise tend to undermine the independence of their judgment. Researchers must consider their competing interests, including from sources of the funding, and report any possible conflicts.
  • Researchers should not misrepresent work that might be competing or related, and should not misrepresent an organization’s policies or procedures.

Be Fair and Take Action not to Discriminate

  • The values of equality, tolerance, respect for others, and justice govern this principle. The COLM Code of Conduct provides additional details of expected behavior at the Conference.
  • Fairness requires that even careful decision processes provide some avenue for redress of grievances.
  • Researchers should foster fair participation of all people—in their research, at the conference and generally—including those of underrepresented groups.
  • The use of information and technology may cause new, or enhance existing, inequities. Technologies and practices should be as inclusive and accessible as possible and researchers should take action to avoid creating systems or technologies that disenfranchise or oppress people.

Respect the Work Required to Produce New Ideas and Artifacts

  • Researchers must show respect for colleagues, research participants, society, ecosystems, cultural heritage and the environment.
  • Developing new ideas, inventions, creative works, and computing artifacts creates value for society, and those who expend this effort should expect to gain value from and receive credit for their work.
  • Researchers should therefore credit the creators of ideas, inventions, work, and artifacts, and respect copyrights, patents, trade secrets, license agreements, and other methods of protecting authors’ works.

Respect Privacy

  • The responsibility of respecting privacy applies to machine learning research in multiple ways. Researchers should be familiar with the various definitions and forms of privacy and should understand the rights and responsibilities associated with the collection and use of personal information.
  • Data should be used in ways consistent with their licenses. Researchers should only use personal information for legitimate ends (e.g., those consistent with approval from an ethics review board) and without violating the rights of individuals and groups.
  • The ethical considerations in this Code should supersede technical legality in the use of data and technologies, i.e. researchers must go beyond the minimal ethical requirements (avoid ethics shirking). This requires taking precautions to prevent re-identification of anonymized data, unauthorized data collection or data collected without consent, ensuring the continuous accuracy of data, understanding the provenance of the data, and protecting it from unauthorized access and accidental disclosure.
  • Data should be collected under appropriate ethical approvals and such approvals must be acknowledged in papers and other contributions.

Honor Confidentiality

  • Researchers and reviewers are often entrusted with confidential information such as trade secrets, client data, non-public business strategies, financial information, research data, pre-publication scholarly articles, and patent applications. Researchers should protect confidentiality except in cases where it is evidence of the violation of law, of organizational regulations, or of the Code.
  • In these cases, the nature or contents of that information should not be disclosed except to appropriate authorities. Researchers should consider thoughtfully whether such disclosures are consistent with the Code.

Concerns and Remediation

This Code should be read in concert with the MLRC Code of Conduct. Concerns can be raised regarding any conduct and contributions that contravene this Code. Where ethical concerns are raised during the review of research contributions, or in the conduct of scientific communication and exchange, this Code serves as an additional basis for remediation of possible violation. MLRC reserves the right to reject and refuse the presentation of any scientific work found to violate the ethical guidelines put forth here, at any point. For raising concerns and for remediation of the same, please reach out to reproducibility.challenge@gmail.com, as described in the Code of Conduct.

Researchers will already have made commitments to codes of ethics and conduct, and to ethical research practices, through their research institution or other professional affiliations, and should remind themselves and consult those codes.

Codes that will be of broader relevance and interest, and that have informed this Code, include:

Ongoing Review

This Code will be reviewed regularly, initially on an annual basis. MLRC welcomes feedback from organizations and researchers on the current version, to inform the review. Last Reviewed, December 2024.

Definitions

Research For the purposes of this Code, “research”1: “… is to be understood as the original investigation undertaken in order to gain knowledge and understanding. It includes work of direct relevance to the needs of commerce, industry, and to the public and voluntary sectors; scholarship;2 the invention and generation of ideas, images, performances, artifacts including design, where these lead to new or substantially improved insights; and the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products and processes, including design and construction. It excludes routine testing and routine analysis of materials, components and processes such as for the maintenance of national standards, as distinct from the development of new analytical techniques. It also excludes the development of teaching materials that do not embody original research.”

Organizations and Researchers For the purposes of this Code, “organizations” refers to any bodies that conduct, host, sponsor or fund research; employ, support or host researchers; teach research students; or allow research to be carried out under their auspices. “Researchers” refers to any person who conducts and contributes to research, including but not limited to: an employee; an independent contractor or consultant; a research student; a visiting or emeritus member of staff; a member of staff on a joint clinical or honorary contract; or project managers and coordinators.

  1. by the Research Assessment Exercise (Research Assessment Exercise 2008, p. 5) ↩︎

  2. Scholarship… is defined as the creation, development and maintenance of the intellectual infrastructure of subjects and disciplines, in forms such as dictionaries, scholarly editions, catalogues and contributions to major research databases. ↩︎

Last updated on Dec 12, 2024