American Epilepsy Society (AES)
Written Comments to
NINDS Request for Information:
Developing an Online Educational Resource for
Training in the Principles of Rigorous Research
Notice Number: NOT-NS-20-062
AES Written Comments were reviewed and approved by the AES Research & Training Council and the AES Executive Committee in July 2020. Comments were authored by the NINDS RFI Workgroup, Education on Rigor, with expertise spanning education and basic,
translational, and clinical research: Barbara Jobst MD (Chair), Ravindra Arya MD DM, E. Martina Bebin MD MPA, Shilpa Kadam PhD, David Loring PhD, and Jamie Maguire, PhD.
Submitted on July 31, 2020
Question 1:
What are the most important principles of rigorous research for all scientists to
understand?
Our comments reflect the collective expertise and viewpoint of the AES, whose 4500 members are clinical care providers and scientists who share a commitment to improved outcomes for children and adults with epilepsy, seizures, and related disorders.
All components of rigorous research drive towards a shared key goal of reproducibility. Principles of rigorous research span every step of research from experimental design and study execution to analysis and publication. The education on such principles
should be entwined with education on research ethics.
While some principles of research rigor are universal, others are specific to a given field of research and must therefore rely upon the experts within that field to inform and develop education on rigor. For example, in epilepsy research, new models
of genetic or acquired epilepsies continue to emerge, often by PIs who may be excellent scientists but lack training in the requirements of rigorous characterization of an epilepsy model. Epileptiform events might be “demonstrated” with
only a sample EEG, omitting the resource-intensive yet essential steps of EEG monitoring protocols with use of blinded scoring, quantification of frequency, diurnal distribution, or natural history, let alone universal standards for rigor such as
appropriate sample sizes, consideration of relevant biological variables, appropriate matched controls.
Universal principles in research rigor can be often found in courses on experimental design and philosophy of science, although such courses are not always accessible or well advertised at institutions. Courses on statistics are more commonly accessible
but often fail to comprehensively cover key principles such as sources of bias and confounding, the meaning of null hypothesis testing and use of probability levels (e.g., p<.01), as well as effect sizes, confidence intervals, clinical meaningfulness,
chance findings and the importance of replication. Important principles are often omitted from formal course work. The influence of base rates regarding diagnostic predicative accuracy is typically poorly understood by trainees and junior investigators.
The methods of systematic reviews could be a valuable teaching tool even for those who will not perform the reviews themselves, as it reinforces the consideration of all available research rather than selective narrative reviews. It also emphasizes
replicability of findings, and at least indirectly, embraces the “file Drawer” problem.
Important principles are often omitted from formal course work. The influence of base rates regarding diagnostic predicative accuracy is typically poorly understood by trainees and junior investigators. Principles on data sharing and data transparency
are essential for reproducibility. Checklists from the Equator Network can aid in their implementation. Distinctions should be reinforced between hypothesis driven versus exploratory research to generate hypotheses. With pre-defined hypothesis –
maximum details for sample size estimation, methods, data analyses and transparency for interpretations should be provided and subsequently reported with the appropriate EQUATOR checklist.
While an understanding of key principles is important, changes to infrastructure and incentives could help empower researchers to transform such principles from aspirational goals to consistent practice. Below, we suggest some potential strategies.
Preregistration of research plans could be more heavily adopted to better enable peer review of research methodology prior to study execution, increase the likelihood of publishing “null” results, and underscore the importance of replication.
To assure rigor of research the review of grant proposals and manuscripts for publication should be shifted to prioritize reproducibility and rigor. Despite increasing recognition of the importance of replication, grant proposals are evaluated almost
exclusively upon “novelty.” Proposals explicitly designed at replication are discouraged and typically triaged. Lack of novelty should not automatically be considered a study weakness. If these rigor and reproducibility are listed separately
and specifically on the review template sheet, this will redirect the entire study section to not only discuss it but also focus on it themselves in their grant writing. Similar changes might be made to the publication process so that reproducible
data results could be prioritized equally with non-reproduced yet novel data. A culture of publishing negative results of well conducted studies, especially clinical trials will set the culture of rigorous research.
Ultimately, primary investigators and institutional leaders must create an atmosphere to celebrate rigor at the department and research institutions, with celebration of rigorously generated null results for trainees and recognition of senior personnel
who mentor in rigorous design. Such atmospheres can be fostered through financial support (e.g. the NIH and other agencies) and the creation of a “score” for rigor so that the success of a research portfolio can be more easily judged by
its rigor as well as its impact factor.
Ultimately, primary investigators and institutional leaders must create an atmosphere to celebrate rigor at the department and research institutions, with celebration of rigorously generated null results for trainees and recognition of senior personnel
who mentor in rigorous design. Such atmospheres can be fostered through financial support (e.g. the NIH and other agencies) and the creation of a “score” for rigor so that the success of a research portfolio can be more easily judged by
its rigor as well as its impact factor.
Question 2:
For each career stage (undergraduates, graduates, post-docs, and faculty), what
are the best teaching practices to use (i.e. how to convey information in a lasting
and impactful way)?
The following comments reflect the collective expertise and viewpoint of the American Epilepsy Society (AES), whose 4500 members are clinical care providers and scientists who share a commitment to improved outcomes for children and adults with epilepsy,
seizures, and related disorders.
Principles.
1) Learners may have different goals for learning principles of rigorous research at different stages of their career and at different time points. These goals may include both appraisal of research (e.g. literature review for clinical decision making)
and generation of research (e.g. designing and implementing a research study to address a knowledge gap).
2) The dynamics of availability of information resources, their access, and the manner in which learners at different career stages and at different times interact with these resources, are constantly shifting.
3) The overarching objective of any educational program is to optimally match the goals of individual learners at a given time, their level of training in research methodology, and the pedagogic technique. Any such educational
program should have enough resilience to accommodate constantly changing dynamics of learning as mentioned above.
Educational Resources.
A majority of the following teaching and learning resources are available in different formats such as print, online, and offline soft version. The earlier education occurs in someone’s career, the more those concepts are likely to be engrained
into the researcher’s core principles.
Principles of rigorous research should definitely be communicated at the graduate level for preclinical research and at the resident and fellow level for clinical research. Within clinical education research relevant teaching is not always emphasized
and principles of rigorous research should be integrated into resident and fellow education.
Undergraduate and medical school education (beginner level below) is very broad and should include some basic principles of rigorous research but probably do not need to go into excessive detail.
At a post-doc and faculty level there should already be awareness of the principles of rigorous research but this cohort would benefit from updated learning.
| Teaching/Learning Resource | Best Utility for Career
Stage* |
| Textbooks | Beginner |
| Didactic lectures (including webinars) | Beginner to Mid-level |
| Live interactive sessions/Q and A | Mid-level to Advanced |
Mentored sessions for study design/research proposal review
| Mid-level to Advanced
|
| Mock study sections | Mid-level to Advanced |
| Original articles related to research methodology (e.g. new or domain-specific statistical computational methods) | Advanced
|
| Web based interactive e-learning | All levels |
Review articles
| All levels
|
RSS feeds, small “information nuggets”
| All levels
|
Irrespective of the career stage, we have used the following arbitrary classification.
- Beginner: Exploring interest in rigorous research.
- Mid-level: Having a specific research objective, looking to critically appraise the literature, or design a research study to address that objective.
- Advanced: Previous experience with successful design and implementation of rigorous research. Also includes reviewers and journal editors.
Potential Strategies to Maximize Research Education.
1) Infrastructure to engage all stakeholders. Professional societies and other 3rd party
organizations can sometimes provide training resources tailor-made to the needs of a
given field. For example, the American Epilepsy Society offers a portfolio of e-learning
resources, including an introductory curriculum for clinical research in epilepsy and
mentorship on grantsmanship. The NIH could foster the engagement of such organizations
by a) offering incentives to investigators or institutions to use such resources, for example
as part of the training requirements for a given grant, and b) offering funding opportunities
for education development.
2) Defining rigorous research and the core curriculum. Perhaps the first goal to develop
an effective educational program will be to establish a core curriculum encompassing the
common minimum tenets of rigorous research. These core tenets should transcend across
specific domains, should be suitable for any level of learning, and include research
methodology, ethics, integrity, and transparency. One possible way to develop such a
curriculum could be Delphi process including all stakeholders.
3) Incentivize learning. Reinforce the value of learning to trainees through financial and
career incentives.
4) Diversity and consistency of exposure. For any educational program to transform the
rigorousness in biomedical research, it will be desirable to offer a diversity of resources to
individual learners, so that they can choose the best format, depending on their specific
objective, career stage, and time availability. These learning formats could include all the
resources mentioned above including web based learning. Also, it will be important to
ensure that these diverse resources emphasize the same core principles of rigorous
research.
5) Opportunities for teaching and learning research methodology. The teaching and
learning of principles and methodology of rigorous research should not be restricted to
sessions solely intended for that purpose. Instead, any clinical or academic opportunity
can be harnessed for the impetus on rigorous research. In addition to dedicated sessions,
discussions in clinic, at the bedside, and in case conferences could be utilized to illustrate
rigorous research-oriented thought process.
6) Repurposing journal clubs. Journal clubs are essential components of residency and
fellowship academics, which are also attended by faculty members. They can be utilized
to emphasize good research methodology by enhancing the participation of statisticians
or research methodology experts. In addition to the subjective content of the journal
articles being reviewed, it is highly advantageous to spend time on critically analyzing the
research methods used by the investigators, learning about the statistical or other
quantitative techniques used therein, and potentially discussing avenues for improvement
in the same.
7) Panels for protocol and grant reviews. Given the variability in expectations, resources,
and sophistication in resident and fellow research, trainees and early career investigators
may benefit from workshops to discuss their research proposals with peers and
experienced researches.
8) Sustained individual extra-mural mentorship. This is exemplified by the AES EPIPORT
(Epilepsy Patient Oriented Research Training) program, where the trainee sends in a draft
proposal and is matched with an external mentor, who then provides ongoing guidance
for developing the funding proposal and its proposed experimental design.
9) Creation of a statistical toolkit. Clinical investigators, particularly those early in their
training and career have variable proficiency in statistical methods. For those with limited
statistical background, it may be valuable to have an online statistical resource to perform
power estimation and some straight-forward statistical tests. Many such online tools are
available, and may be adopted, or a new tool can be created for this purpose. Funding for
statistical consultants is also highly effective.
10) Formal training in research methodology during residency and fellowship.
Including even a small formal component of research methodology training during earlier
states of professional career, can inculcate a culture of rigor and critical thinking to adapt
and create scientific evidence for clinical practice. Incentives and infrastructure are key,
for example to include a research methodology component in institutional assessments
and board exams. Research training should be customizable to meet the needs of different
programs. For example, the needs of a neuroscience graduate student differ from that of
a child neurology resident motivated to critically evaluate and integrate research into her
practice, or a neuro-radiology resident with computer science background who wants to
pursue image-processing research.
| Practices with evidence for success in teaching research methodology |
1) Focus on real clinical and scientific applications.
2) Focus on learners’ actual needs.
3) Balance active and passive learning.
4) Connect new knowledge to old (what the learners already know)
5) Match, or take advantage of, the circumstances including the clinical setting and time
6) Balance preparedness with opportunism
7) Build learners’ abilities for future self-learning Modified from:
Straus, S. E., Glasziou, P., Richardson, W. S., & Haynes, R. B. (2018). Evidence-Based
Medicine E-Book: How to Practice and Teach EBM. Elsevier Health Sciences.
Pinsky, L. E., Monson, D., & Irby, D. M. (1998). How excellent teachers are made: reflecting
on success to improve teaching. Adv Health Sci Educ., 3(3), 207-215. |
Question 3:
For each career stage, what is the ideal delivery structure and how often should
these principles be reinforced?
The following comments reflect the collective expertise and viewpoint of the American
Epilepsy Society (AES), whose 4500 members are clinical care providers and scientists who share a commitment to improved outcomes for children and adults with epilepsy, seizures,
and related disorders.
Training in the Principles of Rigorous Research is essential for maintaining scientific integrity
and public trust in the scientific community. Efforts to develop an online educational
resource will ensure standardization of training and compliance. We have developed
recommendations to ensure effective training throughout the scientific community. Training
in the Responsible Conduct of Research (RCR) is an excellent example of essential training
which has been effectively administered throughout the scientific community. We
recommend following a similar training structure for Training in the Principles of Rigorous
Research:
1. Require that Training in the Principles of Rigorous Research, including fundamentals
of study design, is required for predoctoral and postdoctoral NIH National Research
Service Award (NRSA) Fellowship applications.
2. Require that all trainees supported by NIH funds receive training in the Principles of
Rigorous Research, refreshed every four years, and that such training is addressed in
progress reports for NIH grants.
3. Propose that the ACGME incorporate training in research and research rigor,
including fundamentals of study design, as a required component of accredited
clinical fellowships.
4. Add a section in NIH grants for the PI to outline issues specific to their grant that
relate to the broader concerns of rigor, then require that the PI discuss those issues
with the individuals working on the grant.
This periodic training can be reinforced by helping investigators access the training that they
need at the time that they need it for their own goals. For example, an online library of
concise training modules, with a robust searchable interface, could allow investigators to
access refreshers on a given topic at the time that they need it, for example when designing
or analyzing a given experiment. This concept of a toolkit is outlined further in our response
to Question 4 (below).
Question 4:
If delivered as an online resource, what is the most effective balance between
different educational media (e.g. text, video, interactive activities)?
The following comments reflect the collective expertise and viewpoint of the American
Epilepsy Society (AES), whose 4500 members are clinical care providers and scientists who
share a commitment to improved outcomes for children and adults with epilepsy, seizures,
and related disorders.
Training in the Principles of Rigorous Research is essential for maintaining scientific integrity
and public trust in the scientific community. Efforts to develop an online educational
resource will ensure standardization of training and compliance. The platform for delivery of
this information is critical. A completely online didactic format may have the appearance of
a “checkbox” training rather than actually fulfilling the goal to educate the scientific
community in the Principles of Rigorous Research. We have developed recommendations to ensure effective training throughout the scientific community. We recommend a multi-media
approach to Training in the Principles of Rigorous Research as outlined below:
1. Develop an online Toolkit for Training in the Principles of Rigorous Research,
including fundamentals of study design. Compiling essential tools for planning and
executing rigorous research in a single approved location would be a valuable
practical asset The toolkit could incorporate power analysis calculators, statistical
recommendations, technical and biological replicates, and recommendations for
blinding and counterbalancing proposed experiments as well as providing educational
feedback about the specific recommendations. The toolkit might also enable access
to field-specific resources, such as the preclinical common data elements developed
by the ILAE/AES Joint Translational Task Force to foster reproducibility in
translational preclinical epilepsy research. Although many online tools are available,
their validity has not been independently established and have not been compiled
into a comprehensive resource. Further, as part of this toolkit, there could be feature
where trainees could be presented various basic or clinical research vignettes
associated with each topic module and asked to complete educational “quizzes” with
feedback to test and reinforce their understanding of important research concepts.
2. Interactive online education is ideal to ensure understanding of the trainees. Short
video modules (10-15 minute max) with high production quality would be an
effective strategy, incorporating cartoons or infographics rather than a long video of
one person talking,
3. Interactive research studies would also be effective. The video could be designed to
be interactive such that the trainee could make decisions on their course of action
and then go through training regarding the potential consequences of those choices.
This interactive video-based online training would help reinforce the relevance of the
training. Example of poorly designed or poorly analyzed research should also be
presented with discussion of incorrect conclusions and study inference based upon
faulty study design.
4. Recommendations for standard operating procedures and best practices for labs to
reinforce best practices in the lab.