Scientific rigor is fundamental to two key goals of the scientific endeavor: knowledge creation and the translation of discoveries into new therapies. Research must be well-designed and implemented, accessible, and reproducible to reap the benefits of new discoveries.
For this virtual conference, SfN has partnered with NIH and neuroscientists around the world to provide you — no matter your career stage — with training resources to enhance rigor in experimental design, data analysis, and reporting of your research findings. This virtual conference is provided open access to the field and is supported by Grant Number R25DA041326 from the National Institute on Drug Abuse (NIDA). The contents of this series are solely the responsibility of the Society for Neuroscience and do not necessarily reflect the official views of NIDA.
The insights, experiences, and practical approaches for enhancing the rigor and transparency of neuroscience research you will hear from experts in a breadth of disciplines will cover various topics, including:
Differences between early stage discovery, hypothesis development, and formal hypothesis testing, and how to incorporate rigor at all stages.
Experimental design and implementation to minimize bias.
Considerations for data gathering and analysis, such as sample size planning, group compilation, statistical power, and the meaning of statistical significance.
NIH training resources and requirements for improving rigor and reproducibility in biomedical research.
Transparent publishing practices and their effect on enhancing rigor in neuroscience research.
How the field can incentivize rigorous research.
Visit the Neuronline collection, Promoting Awareness and Knowledge to Enhance Scientific Rigor in Neuroscience, to watch six related webinars on demand and explore additional articles and videos.
Click on each session title to reveal its description.
Distinguished faculty will discuss how they implement elements of scientific rigor and transparency in their research and training programs for their students and postdocs. As working scientists, Li-Huei Tsai, Cheryl Sisk, Emanuel DiCicco-Bloom, and Os Steward will share how they transition from testing and validating protocols and collecting preliminary data for hypothesis development, to designing a hypothesis-testing experiment that fully incorporates issues of scientific rigor, such as:
Defining necessary controls.
Power analyses.
Random assignment to groups.
Blinding.
Data collection to minimize unintentional bias.
Plans for data analysis.
Session 2: Overcoming Statistical Paralysis
Speakers: Brian Caffo, Os Steward Time: 9:25 a.m. – 10:00 a.m. EDT
Brian Caffo will share his perspective as a biostatistician on incorporating rigor and transparency into a study design — regardless of the form it takes. He will define key breakdowns in the statistical analysis of data, describe how those breakdowns can undermine reproducibility and rigor, and present simple rules for navigating statistical analyses. After his presentation, Brian will be joined by Os Steward for a short question-and-answer session.
Session 3: Identifying and Mitigating Bias in Research
Many sources of bias can affect the different stages of the experimental process — from design and execution, through data analysis and interpretation. These biases can range from obvious ones — such as what can occur during selection of samples — to unconscious biases that may be more difficult to identify — such as decisions that may bias you towards your original hypothesis, rather than following the path of where your data lead. In this session, Rita Balice-Gordon, who works in the pharmaceutical industry, and Theresa Hernández, who studies complementary and alternative medicines, will share insight into how they identify and mitigate various sources of bias in their research programs, particularly when exploring new therapeutic approaches. Presentations will be followed by a short question-and-answer session.
Session 4: How to Adopt Evolving Best Practices for Rigorous Data Collection, Management, and Analysis
Researchers from different neuroscience disciplines will address how they approach data collection and analysis with rigor and transparency and how they are integrating best practices for managing and analyzing data into their work. Meaghan Creed will focus on how she designs behavioral and electrophysiology experiments with her analyses in mind — from replicate groups to sample sizes. Next, based on her work in the human imaging field, Deanna Barch will cover best practices in hypothesis-driven and data-driven analytic approaches, including methods for addressing power and multiple comparisons. Finally, Mary Harrington will discuss her decision to share her complete data sets and analysis code with the scientific community through the Open Science Framework, and how she and her undergraduate students stay up-to-date on evolving open science research practices to enhance transparency. Individual presentations will be followed by a panel discussion and Q&A with all speakers.
Session 5: Rigor and Reproducibility – Perspectives from the NIH
NIH continues to update and refine its requirements and resources for training graduate students and postdoctoral fellows in the areas of rigor and reproducibility. In this session, Kristine Willis from the National Institute of General Medical Sciences will share the latest updates regarding NIH requirements in research and training grant applications for enhancing reproducibility and rigor. She will also discuss training resources that support NIH’s ongoing effort to promote reproducible and rigorous research, quantitative and computational skills development, and the sociology and ethics of decision-making related to reproducibility and rigor.
Transparent reporting of experimental methodologies and results is critical for enhancing rigor and increasing reproducibility of neuroscience research. In this session, Marina Picciotto, Elena Porro, and Malcolm Macleod will address recent changes in scientific publishing to enhance transparency. Speakers will discuss initiatives, such as revised reporting guidelines from JNeurosci, Cell Press’ STAR Methods, and the Nature Publishing Group’s author checklist, that can help standardize reporting of experimental details and statistical analyses. Emerging initiatives — such as preprints and registered reports and protocols — will also be discussed. Brief individual presentations will be followed by a panel discussion and Q&A with all speakers.
Session 7: Making Open Science Work for You
Speakers: Marcus Munafò, Kate Button Time: 2:50 p.m. – 3:50 p.m. EDT
Researchers can enhance the rigor and reproducibility of neuroscience research by making their research protocols, data, and analyses more transparent and accessible. In this session, Marcus Munafò and Kate Button will focus on the actionable steps individual scientists — regardless of their field — can employ to integrate rigor and transparency into their research programs, emphasizing open science initiatives, such as pre-registration of protocols and open data practices. Individual presentations will be followed by a moderated Q&A discussion period, which will provide an opportunity to explore how various open science practices can be implemented in different settings.
Session 8: Recognizing and Addressing Perverse Incentives in Science
Rigorous conduct of neuroscience research is influenced by many factors beyond the actual conduct of science. These factors, often called “perverse incentives,” can include:
Pressure on scientists to increase the breadth or flashiness of their research while minimizing caveats and sacrificing rigor.
Pressure to rush research into publication to meet career or funding deadlines.
Disincentives for publishing negative results or replication studies, and many more.
In this closing session, Os Steward, Emanuel DiCicco-Bloom, and Cheryl Sisk will engage conference attendees in a discussion about these and other factors that warrant collective action by the field. They will emphasize the shared responsibility all stakeholders —funding agencies, publishers, scientific academy, and others — play in recognizing these perverse incentives and rewarding efforts that enhance rigor and reproducibility.
Speakers
Oswald Steward, PhD
Oswald Steward is founding director of the Reeve-Irvine Research Center for Spinal Cord Injury at the University of California, Irvine. He is Reeve-Irvine Professor of Anatomy and Neurobiology and holds additional joint appointments in the departments of neurobiology and behavior and neurosurgery. His research focuses on how neurons create and maintain their connections, how synapses are modified by experience and injury, and the role of genes in neuronal regeneration, growth, and function. He received his PhD in psychobiology/neuroscience from the University of California, Irvine.
Brian Caffo, PhD
Brian Caffo is professor of biostatistics at the Johns Hopkins University (JHU) Bloomberg School of Public Health and director of the graduate program at JHU Biostatistics. Caffo is the co-founder of the JHU Statistical Methods and Applications for Research in Technology (SMART) working group, which develops statistically principled methods for new technologies with an emphasis on brain imaging, wearable computing, and biosignals. He earned his BS in mathematics and statistics, MS in statistics, and PhD in statistics from the University of Florida.
Theresa D. Hernández, PhD
Theresa Hernández is an associate dean for research and professor of psychology and neuroscience at the University of Colorado, Boulder. Hernández’s research on traumatic brain injuries (TBIs) utilizes basic and clinical neuroscience approaches to improve treatment outcomes and explores novel interventions stemming from complementary and alternative medicine. She received her PhD from the University of Texas at Austin.
Rita Balice-Gordon, PhD
Rita Balice-Gordon, PhD, is the head of neuroscience research at Sanofi Genzyme. She leads teams of scientists prosecuting targets relevant to neurodegeneration, neuroinflammation, and central nervous system rare disease. Balice-Gordon is on the National Institute of Child Health and Human Development Board of Scientific Councilors and SfN’s Government and Public Affairs Committee, and she was elected as a fellow of the American Association for the Advancement of Science (AAAS) in 2015. She earned her BA in biological sciences from Northwestern University and PhD in neurobiology from the University of Texas at Austin.
Deanna Barch, PhD
Deanna Barch is a professor and department chair of psychological and brain sciences at Washington University in St. Louis. Her research uses functional MRI, structural MRI, and cognitive neuroscience methods to examine the neural basis of disturbances in cognitive control and emotional processing in individuals with mental health illnesses such as schizophrenia and depression. She received her BA from Northwestern University and MA and PhD from the University of Illinois.
Meaghan C. Creed, PhD
Meaghan Creed is an assistant professor in the department of pharmacology at the University of Maryland School of Medicine. Creed earned her HBSc and her PhD from the University of Toronto. She completed her postdoctoral training at the University of Geneva.
Mary Harrington, PhD
Mary Harrington is a professor of life sciences and director of the neuroscience program at Smith College. Her research on the effects of disrupted circadian rhythms utilizes novel techniques to measure gene expression in vivo, animal models of fatigue, and cell culture. She earned her BS in psychology from Pennsylvania State University, MA in psychology from the University of Toronto, and PhD in psychology from Dalhousie University.
Kristine Willis, PhD
Kristine Willis is a program director in the Division of Genetics and Molecular, Cellular, and Developmental Biology at NIH, where she oversees grants in the area of mutagenesis and DNA repair. She also manages postdoctoral fellowships in genetics and developmental biology. Willis was previously an assistant research professor at Georgetown University. She earned her BS and PhD in biology from the University of Southern Mississippi and conducted postdoctoral research at the University of Toronto.
Malcolm Macleod, PhD
Malcolm Macleod is a professor of neurology and translational neuroscience at the Center for Clinical Brain Sciences at the University of Edinburgh. Macleod’s main interests lie in the development and application of systematic review and meta-analysis to the analysis of data from animal studies modelling neurological diseases such as stroke. He received his BS and PhD from the University of Edinburgh.
Marina Picciotto, PhD
Marina Picciotto is the Charles B.G. Murphy Professor of Psychiatry in the Child Study Center of Neuroscience and Pharmacology at Yale University, where she has been on the faculty since 1995. Dr. Picciotto’s research interests lie in understanding the role of molecular signaling in complex behavior, focusing on the function of acetylcholine and its receptors in learning, development, and circuit function. She received her undergraduate degree in biological sciences from Stanford University and her PhD in Molecular Neurobiology from Rockefeller University. She completed her postdoctoral training at the Institute Pasteur in Paris.
Elena Porro, PhD
Elena Porro is the senior deputy editor of Cell and the editorial director at Cell Press. She received her PhD in cell and developmental biology from Harvard University, where she worked in the lab of Li-Huei Tsai studying the cell biology of neuronal migration in brain development.
Marcus Munafò, PhD
Marcus Munafò is a professor of biological psychiatry and director of the Tobacco and Alcohol Research Group at the University of Bristol. His research focuses on the genetic and cognitive influences on addictive behavior and investigates the pathways into and consequences of health behaviors and mental health. In addition to his research expertise, he also has interests in the role of incentive structures in science and the extent to which these shape the robustness and reproducibility of scientific research. Munafò earned his undergraduate degree at the University of Oxford, his MSc in health psychology, and PhD at the University of Southampton.
Katherine Button, PhD
Katherine Button is an associate professor at the University of Bath, UK. Her main research interests lie in mechanisms underlying anxiety and depression and their treatment, but she has a side interest in research rigour and reproducibility. She received her undergraduate degree in neuroscience from Cambridge University and her PhD in Psychiatry from the University of Bristol, where she also held two post-doctoral fellowships before moving to her current role at the University of Bath.
Li-Huei Tsai, PhD
Li-Huei Tsai is a professor of neuroscience and the director of the Picower Institute for Learning and Memory at the Massachusetts Institute of Technology. Tsai is also a fellow of American Association for the Advancement of Science (AAAS), a member of the National Academy of Medicine, and an Academician of the Academia Sinica in Taiwan. She was previously a professor at Harvard Medical School and the investigator at the Howard Hughes Medical Institute. Her research interests lie in elucidating the pathogenic mechanisms underlying neurological disorders that impact learning and memory. Tsai received her PhD from the University of Texas Southwestern Medical Center and completed her postdoctoral training at Cold Spring Harbor Laboratories and Massachusetts General Hospital.
Cheryl Sisk, PhD
Cheryl Sisk is a professor of psychology and director of the interdepartmental graduate neuroscience program at Michigan State University. She is also the treasurer of the Society for Behavioral Neuroendocrinology and president of the Association of Neuroscience Departments and Programs. Sisk’s research program focuses on how the gonadal steroid hormones of puberty shape the development of the adolescent brain and the maturation of social behaviors. She earned her PhD in biopsychology and neuroscience from Florida State University and completed her postdoctoral training at Northwestern University.
Emanuel M. DiCicco-Bloom, MD
Emanuel DiCicco-Bloom is a professor in the department of neuroscience and cell biology at Robert Wood Johnson Medical School. He currently practices at Childrens Health Institute and is affiliated with Robert Wood Johnson University Hospital, Robert Wood Johnson University Hospital at Hamilton, and Saint Peter's University Hospital. DiCicco-Bloom’s research interests are in the regulation of developmental and adult neurogenesis and cell cycle mechanism. He graduated from Weill Cornell Medical College and completed a residency at New York Presbyterian Hospital.
Shiva Singh, PhD
Shiva Singh is chief of the undergraduate and predoctoral training branch of the Division of Training, Workforce Development, and Diversity at National Institute of General Medical Sciences (NIGMS). He previously served as director of biomedical research and training programs at Alabama State University. Singh earned his BS and MS in plant sciences from G.B. Pant University of Agriculture and Technology in India and his PhD in microbiology from Auburn University. He completed his postdoctoral training at Auburn University and Argonne National Laboratory.