Scientific rigor broadly means good experimental practice. It means that other people can replicate your work and understand exactly what you did in the course of your experiments.

Core Aspects

Research is done in different ways. Some studies are planned out from start to finish. However, innovative discoveries sometimes come from studies that begin with preliminary experiments that progress through more systematic studies. The latter is an iterative process: an experiment here, another there, and if the preliminary experiments are promising, more experiments are done. Somewhere along the line you realize that additional control groups are needed, and different groups may be compiled by experiments done over time. At some point, data are wrapped up in a package and tied with a bow and sent off to a journal. But then sometimes editors and reviewers require additional experiments, and so the iterative process continues.

When pre-planning can be done, it’s important to plan what groups are needed, how many animals/subjects/individual experiments are required, how to compile groups to avoid unrecognized bias (randomization for example), how to collect data that avoids unrecognized bias, blinding, data inclusion/exclusion, handling of outliers, and much more.

But scientific rigor goes beyond that. It also pertains to full and transparent reporting. You need to explain exactly how you did an experiment — not just in the general sense, such as that you had five groups — but how did those groups get constructed and when were they actually done? Transparency is important not only so people can repeat your study, but also so readers can understand the implications and possible caveats of the exact actions you took. 

Importance

Scientific rigor is the basis for confidence in results for scientists and trust for science by the public. 

If we understand that all aspects of rigor were followed, then we have greater confidence than if we read a paper in which the presentation doesn’t completely outline what was done to ensure rigor. 

Concerns about scientific rigor have arisen because of highly publicized failures to replicate across many different fields, some of which have formed the basis of clinical trials. Scientists in many different fields are working to understand the reasons for the “replication crisis” and to develop ways to enhance rigor going forward. The TMEDR series put together by SFN represent one part of our professional society’s approach to address this very critical issue.

Challenges

There are challenges in improving and maintaining scientific rigor due in part to perverse incentives built in to our systems.  

People who are in academic institutions or industry need to finish things by deadlines, including grant deadlines and paper submissions for promotion and tenure. This can be a perverse incentive to terminate studies prematurely. 

Another challenge is that journals have been reducing methods sections, and some journals have eliminated published methods entirely, relegating methods to supplemental information. The challenge going forward is to re-instate thorough review practices that focus on scientific rigor, not just “broad interest” or “impact.”

It’s almost certain that improving rigor will require attention and action at all levels. Also critical is coming to grips with what a failure to replicate actually means. In most cases, a failure to replicate should not be a repudiation of the original study. Instead, the reasons for failures to replicate likely lie in unrecognized and perhaps subtle differences in procedure or biological variables. If the critical differences can be identified, this advances our understanding of the underlying biology.

About the Contributor

Oswald Steward, PhD

Oswald Steward is a professor of anatomy and neurobiology at the University of California at Irvine School of Medicine.

What does it mean to conduct rigorous science? This video explores the key elements: replicability, experimental design and bias, data analysis protocols, and data management and reporting. Watch to learn why these topics are important and how to incorporate best practices into your everyday research.

Read more about SfN’s scientific rigor policy positions and how the Society is evaluating scientific rigor in the field. Also access SfN’s training modules about scientific rigor in the Neuronline collection, Promoting Awareness and Knowledge to Enhance Scientific Rigor in Neuroscience.

Scientific rigor means implementing the highest standards and best practices of the scientific method and applying those to one’s research. It is all about discovering the truth.

Scientific rigor involves minimizing bias in subject selection and data analysis. It is about determining the appropriate sample size for your study so that you have sufficient statistical power to be more confident about whether you are generating false positives or missing out on false negatives. It’s about conducting research that has a good chance of being replicated in your own lab and other laboratories.

The idea is to figure out under what range of conditions a particular outcome is generated — the wider the range, the higher the likelihood that the outcome reflects a scientific truth. If results can be replicated in one’s own lab, but cannot be reproduced in another lab, does not generalize well in other situations, or does not hold up over the test of time, the outcome might not be that important.

What’s at Stake

We want to know and like to think that our research is generalizable and robust. If the research is not conducted with scientific rigor, then generalizability and robustness are not going to happen.

And, if neuroscience research is not conducted with scientific rigor, then it’s a big waste of time, money, and energy in pursuing outcomes that are not real. We also risk the possibility of missing great insights because we didn’t conduct the research in a way that would allow us to detect those insights.

Some of the issues around blind analysis and randomization of subjects are particularly important for clinical applications because lives are going to depend on their outcomes.

We also want science to be objective and empirical. But we’ve known for a long time that when an experimenter knows what treatment group a patient, rat, or a mouse is in, then that knowledge can influence the way they look at the data or analyze the data. It’s not intentional, but it is experimenter bias. The same is true for patients. If they know they are in one group and not the other, then bias is introduced. It is therefore critical to know how large the placebo effect is. Ultimately we are after the truth. If people — either the experimenter or the subject — are not blind to their treatment group, that introduces bias that will cloud the truth.

Challenges

The challenges are mainly around the time it takes to be careful and rigorous in research. PIs are faced with many responsibilities ranging from teaching, completing administrative/committee work, administering grants, keeping the lab funded, and publishing papers. They are pushed and pulled in all directions. However, it takes time to ensure that there is scientific rigor, and implementing the best practices of scientific rigor will almost certainly increase the time to publication of one’s results.

My Lab’s Approach

Before any experiment is conducted, the student or postdoc has to write a plan of study so they think purposefully about their study in advance. The plan of study includes, among other points: rationale for the study, an answer to whether it is discovery science or hypothesis-driven (both have their place), sample size, determination of statistical power, detailed methods and procedures, and how data will be analyzed.

We also periodically read and discuss three papers that span 110 years of science. One is John Platt’s 1964 Science paper, “Strong Inference,” which talks about hypothesis testing and the importance of having alternative hypotheses and an experimental design intended to falsify (not support) your hypothesis. The second paper written in 1897 by Thomas Crowder Chamberlin discusses the method of multiple working hypotheses. More recently, a 2014 paper by Douglas Fudge, “50 Years of Platt’s Strong Inference,” revisits principles outlined in the 1964 Platt paper.

Students find the idea of designing experiments that will falsify your hypothesis, given a particular outcome, a challenging exercise — and so do I. We have a lot of fun doing this.

Forward Look

Through the Training Modules to Enhance Data Reproducibility grant (I am a co-PI), SfN seeks to raise awareness in the neuroscience community about the importance of scientific rigor and its application to our discipline. If we are going to make strides that stand the test of time, it is essential.

The next generation of neuroscientists needs to be aware of these important points. Of course, awareness will make the discipline stronger. We also need to provide necessary training to conduct research in a rigorous way.

In addition, the relationship between young investigators and trainees and their mentors is really important in learning about scientific rigor. It is my responsibility as a mentor and trainer to raise young neuroscientists in a way that makes scientific rigor and strong inference habits of mind and practice.

It is also important for young researchers just learning the ropes to seek training in this area. This can teach them to question both their own assumptions and those of others in their lab, and to do what they can as budding scientists to enhance the scientific rigor and credibility of their research. 

About the Contributor

Cheryl Sisk, PhD

Cheryl Sisk is a University Distinguished Professor in the neuroscience program at Michigan State University.

Rigorous conduct of science is the cornerstone of the scientific endeavor, touching on established practices for experimental design, data analysis, and transparency, as well as other issues like publishing and funding pressures. Knowing how to address these issues is critical for a successful career in science. This workshop explores practical examples of the challenges and solutions in conducting rigorous science from the real-life examples of neuroscientists at various career stages. It focuses on development of the interpersonal, scientific, and technical skills necessary to address various issues in scientific rigor, such as what to do when you can't replicate a published result, how to get support from a mentor, and how to cope with various career pressures that might affect the quality of your science. For more information about scientific rigor, visit SfN's hub of Training Modules to Enhance Data Reproducibility.

This training module is supported by Grant Number 1R25DA041326-01 from the National Institute on Drug Abuse (NIDA). The original contents of this module are solely the responsibility of SfN and do not necessarily reflect the official views of NIDA.

Speakers

John H. Morrison, PhD

John H. Morrison, PhD, is the director of the California National Primate Research Center and Professor of Neurology and Neuroscience at UC Davis. Morrison earned his bachelor’s degree and PhD from Johns Hopkins University, and completed postdoctoral studies in the laboratory of Floyd E. Bloom at the Salk Institute for Biological Studies. Morrison’s research program focuses primarily on the neurobiology of aging and neurodegenerative disorders such as Alzheimer’s Disease, particularly as they relate to cellular and synaptic organization of cerebral cortex. Within this broad arena, his lab works specifically on the interactions between endocrine factors (e.g., estrogen, stress steroids) and aging and the synaptic determinants of cognitive aging. He has published over 300 articles on cortical organization, Alzheimer’s disease, the neurobiology of cognitive aging, and the effects of stress on cortical circuitry. Deeply committed to the public communication of neuroscience, he was a member of the BrainFacts.org Advisory Board, which helped guide the development of BrainFacts.org from its inception, prior to becoming Editor-in-Chief.

Barbara Lom, PhD

Barbara Lom, PhD, is professor and chair of biology at Davidson College where she studies the growth and elaboration of axons and dendrites in the developing nervous system. She is also the founding editor-in-chief and editorial board member of The Journal of Undergraduate Neuroscience. Lom has received numerous honors and awards, including the Distinguished Mentor Award from the Faculty for Undergraduate Neuroscience and the Hunter-Hamilton Love of Teaching Award from Davidson College.

Deena Walker, PhD

Deena Walker, PhD, is a postdoctoral fellow in the laboratory of Eric Nestler at the Icahn School of Medicine at Mount Sinai. Her research focuses on cellular and molecular mechanisms underlying sex differences in adolescent development, with a focus on motivation and reward. She has published extensively on how perturbations during early life influence the development of sex-specific brain regions. She plans to investigate how stress during adolescence alters sex-specific transcription, circuitry, and behavior in adulthood in her independent laboratory.

Erin McKiernan, PhD

Erin McKiernan, PhD, is a researcher at the National Institute of Public Health in Mexico. She works in experimental and computational neuroscience, and is an advocate for open access, open data, and open science.

Phillip Popovich, PhD

Phillip Popovich, PhD, is a neuroscience professor and the director of Ohio State University’s Center for Brain and Spinal Cord Repair. His lab studies spinal cord injury (SCI) and it recently showed that SCI has significant effects on shaping the functional properties of innate (e.g., macrophages) and adaptive (T and B lymphocytes) immune cells. His research may provide new insight into novel immune-modulatory therapies.

Peter R. Rapp, PhD

Peter R. Rapp, PhD, is the chief of the Neurocognitive Aging Section in the Laboratory of Behavioral Neuroscience at the Intramural Research Program of the National Institute on Aging and is an adjunct professor of psychological and brain sciences at Johns Hopkins University. He also serves as editor-in-chief of the journal Neurobiology of Aging. Rapp’s laboratory focuses on the neurobiology of cognitive aging, with particular emphasis on the mechanisms that support successful neuroadaptation in aging.

Literature focused on scientific rigor is ever-growing. In addition to the resources found in the Promoting Awareness and Knowledge to Enhance Scientific Rigor in Neuroscience collection, the articles below can help you explore various issues, solutions, and approaches to implement high standards of scientific rigor in your research.

Rigorous Science: a How-To Guide
Arturo Casadevall and Ferric Fang propose a working definition of scientific rigor: the “theoretical or experimental approaches undertaken in a way that enhances confidence in the veracity of their findings, with veracity defined as truth or accuracy.” They also suggest how to improve scientific rigor and include it in research training.

Scientific Rigor and the Art of Motorcycle Maintenance
Marcus Munafó et al. discuss the CHDI Foundation’s new policies, such as providing a manual with pre-clinical experimental best practices and developing online training resources, to explore how the scientific community can consider addressing scientific rigor issues.

Reproducibility in Science
Glenn Begley and John Ioannidis summarize many of the reported issues with reproducibility in the biomedical sciences and suggest solutions. They also address the challenges to instituting these solutions, including bureaucracy and increased time demands facing researchers.

A Manifesto for Reproducible Science
Marcus Munafó et al. propose measures to improve research efficiency and robustness and discuss how to approach implementation as the practice of science grows and changes.

Reproducibility2020: Progress and Priorities
In this report, Leonard Freedman et al. summarize efforts to improve reproducibility in study design, data analysis, reagents and reference materials, lab protocols, and reporting and review. They outline solutions to reproducibility problems in these areas and highlight how different parts of the scientific community can contribute to these undertakings.

Reproducibility in Computational Neuroscience Models and Simulations
Focusing on reproducibility issues specific to computational neuroscience, Robert McDougal et al. draw from software development best practices, such as strong commenting and documentation, to suggest solutions for their field.

The topics of scientific rigor and data reproducibility have been increasingly covered in the scientific and mainstream media, and are being addressed by publishers, professional organizations, and funding agencies, including NIH. This webinar — the first in a series titled Promoting Awareness and Knowledge to Enhance Scientific Rigor in Neuroscience — will address topics of scientific rigor as they pertain to pre-clinical neuroscience research.

• A better understanding of the issues surrounding scientific rigor and the lack of data reproducibility in basic neuroscience research
• Example best practices for designing pre-clinical experiments and planning for data collection
• An overview of the new grant sections required by the NIH to address issues of experimental rigor and data reproducibility

This training module is supported by Grant Number 1R25DA041326-01 from the National Institute on Drug Abuse (NIDA). The original contents of this module are solely the responsibility of SfN and do not necessarily reflect the official views of NIDA.

Speakers

Oswald Steward, PhD

Oswald Steward is founding director of the Reeve-Irvine Research Center for Spinal Cord Injury at the University of California, Irvine. He is Reeve-Irvine Professor of Anatomy and Neurobiology and holds additional joint appointments in the departments of neurobiology and behavior and neurosurgery. His research focuses on how neurons create and maintain their connections, how synapses are modified by experience and injury, and the role of genes in neuronal regeneration, growth, and function. He received his PhD in psychobiology/neuroscience from the University of California, Irvine.

Katherine Button, PhD

Katherine Button is an associate professor at the University of Bath, UK. Her main research interests lie in mechanisms underlying anxiety and depression and their treatment, but she has a side interest in research rigour and reproducibility. She received her undergraduate degree in neuroscience from Cambridge University and her PhD in Psychiatry from the University of Bristol, where she also held two post-doctoral fellowships before moving to her current role at the University of Bath.