A new report by the Commons Science, Innovation and Technology
Select Committee highlights concerns over the reproducibility of
scientific research.
The report notes that despite the Government's increased focus on
research and innovation and the largest-ever increase in public
investment in research and development, the integrity of some
scientific research has been called into question because of
difficulties in reproducing claimed findings of experiments or
analyses of data. This has led to concerns over what some have
called a 'reproducibility crisis'.
The Committee welcomes the establishment of a new Committee on
Research Integrity, which was a recommendation of the predecessor
Committee’s 2018 report. However, it expresses concern about the
absence of reproducibility as a priority in the new
organisation's strategy.
The report emphasises the need for a comprehensive and rigorous
assessment of the scale of the problem in the UK and which
disciplines are most affected, and in particular the impact of
deploying Artificial Intelligence—and other increasingly complex
software—on reproducibility in research.
The report recommends the establishment of a sub-committee of the
Committee on Research Integrity focused solely on questions of
reproducibility in research.
The Select Committee’s report addresses recommendations to
funders, including UK Research and Innovation. It says funders
should consider whether grants awarded provide the resources
necessary to ensure that work they fund is reproducible.
The report recommends that funders should require reproducibility
as a condition of grants awarded, as well as training researchers
in research integrity and the need to ensure reproducibility. The
report also backs the wider use of the 'resume for researchers'
format in funding calls, including evidence of commitment to
reproducibility.
The report calls on publishers of academic journals to ensure
that there is no bias towards publishing studies whose findings
are novel over those whose results confirm previous studies. It
also calls on publishers to commit to more timely publication of
retractions of published articles and corrections of errors.
Committee Chair Rt Hon MP said:
“Scientific progress requires transparency: being able to
reproduce analysis to check research findings are
robust.
“With the annual publicly-funded R&D budget set to
reach £20 billion next year, it is especially important that
findings stand up to scrutiny.
“But there are concerns that many scientific findings are
difficult or impossible to replicate, with AI posing new
challenges to transparency.
“We call for a clearer focus from research funders,
publishers and academic institutions to ensure that research is
transparent and reproducible.”
Key recommendations:
- Reproducibility as a condition of grants awarded for
empirical research
- Review the Research Excellence Framework assessment criteria
to assure that transparency is a prerequisite of top-scoring
research.
- Publishers should review their record of publishing
confirmatory studies
- Publishers should commit to more timely publication of
retractions of published articles and the correction of errors no
more than two months after they become known
- Require, outside exceptional circumstances, the deposition of
research data and code in open-access repositories alongside the
publication of research outputs
- A coordinated policy on minimum protected research time for
research staff
- A three-year minimum contract for post-doctoral researchers
in universities
- Mandatory reproducibility training for researchers at
undergraduate, postgraduate, and early career researcher stages
- A pilot programme to fund replication studies
- Dedicated funding for the presence of statistical experts and
software developers in research teams
- Develop a ‘registered report’ model in which funding
proposals and prospective publication are peer-reviewed before
results are obtained
Recommendations to the UK Committee of Research Integrity:
- Publish an annual state of the nation report with action
plans and timings for implementing measures
- Establish a sub-committee focused solely on questions of
reproducibility in research
- Investigate the impact of deploying AI—and other increasingly
complex software—onreproducibility in research