Scientific rigor broadly means good experimental practice. It means that other people can replicate your work and understand exactly what you did in the course of your experiments.
Core Aspects
Research is done in different ways. Some studies are planned out from start to finish. However, innovative discoveries sometimes come from studies that begin with preliminary experiments that progress through more systematic studies. The latter is an iterative process: an experiment here, another there, and if the preliminary experiments are promising, more experiments are done. Somewhere along the line you realize that additional control groups are needed, and different groups may be compiled by experiments done over time. At some point, data are wrapped up in a package and tied with a bow and sent off to a journal. But then sometimes editors and reviewers require additional experiments, and so the iterative process continues.
When pre-planning can be done, it’s important to plan what groups are needed, how many animals/subjects/individual experiments are required, how to compile groups to avoid unrecognized bias (randomization for example), how to collect data that avoids unrecognized bias, blinding, data inclusion/exclusion, handling of outliers, and much more.
But scientific rigor goes beyond that. It also pertains to full and transparent reporting. You need to explain exactly how you did an experiment — not just in the general sense, such as that you had five groups — but how did those groups get constructed and when were they actually done? Transparency is important not only so people can repeat your study, but also so readers can understand the implications and possible caveats of the exact actions you took.
Importance
Scientific rigor is the basis for confidence in results for scientists and trust for science by the public.
If we understand that all aspects of rigor were followed, then we have greater confidence than if we read a paper in which the presentation doesn’t completely outline what was done to ensure rigor.
Concerns about scientific rigor have arisen because of highly publicized failures to replicate across many different fields, some of which have formed the basis of clinical trials. Scientists in many different fields are working to understand the reasons for the “replication crisis” and to develop ways to enhance rigor going forward. The TMEDR series put together by SFN represent one part of our professional society’s approach to address this very critical issue.
Challenges
There are challenges in improving and maintaining scientific rigor due in part to perverse incentives built in to our systems.
People who are in academic institutions or industry need to finish things by deadlines, including grant deadlines and paper submissions for promotion and tenure. This can be a perverse incentive to terminate studies prematurely.
Another challenge is that journals have been reducing methods sections, and some journals have eliminated published methods entirely, relegating methods to supplemental information. The challenge going forward is to re-instate thorough review practices that focus on scientific rigor, not just “broad interest” or “impact.”
It’s almost certain that improving rigor will require attention and action at all levels. Also critical is coming to grips with what a failure to replicate actually means. In most cases, a failure to replicate should not be a repudiation of the original study. Instead, the reasons for failures to replicate likely lie in unrecognized and perhaps subtle differences in procedure or biological variables. If the critical differences can be identified, this advances our understanding of the underlying biology.
About the Contributor
