Interdisciplinary Approaches and Strategies from Research Reproducibility 2020: Educating for Reproducibility

Abstract

Research Reproducibility: Educating for Reproducibility, Pathways to Research Integrity was an interdisciplinary, conference hosted virtually by the University of Florida in December 2020. This event brought together educators, researchers, students, policy makers, and industry representatives from across the globe to explore best practices, innovations, and new ideas for education around reproducibility and replicability. Emphasizing a broad view of rigor and reproducibility, the conference touched on many aspects of introducing learners to transparency, rigorous study design, data science, data management, replications, and more. Transdisciplinary themes emerged from the panels, keynote, and submitted papers and poster presentations. The identified themes included lifelong learning, cultivating bottom-up change, “sneaking in” learning, just-in-time learning, targeting learners by career stage, learning by doing, learning how to learn, establishing communities of practice, librarians as interdisciplinary leaders, teamwork skills, rewards and incentives, and implementing top-down change. For each of these themes, we share ideas, practices, and actions as discussed by the conference speakers and attendees.

KEYWORDS:

• Interdisciplinary education
• Replicability
• Research integrity

1 Introduction

Cultivating the next generation of researchers is a key component of fostering a culture of research integrity. Research integrity training primarily focuses on the responsible conduct of research (RCR), covering a wide swath of topics designed to impart to trainees the principles of research rigor, ethical frameworks for evaluating different options, and practices that uphold community standards (Kligyte et al. 2008). Using these values and ethical frameworks, trainees can be expected to make better choices when confronted with difficult choices (Mumford et al. 2008). However, one area that RCR training has covered less, if at all, is reproducibility.

Reproducibility and replicability are cornerstones of research integrity and science, ensuring that experimental procedures can be performed by others, and that the steps and decisions involved in published results are presented openly and honestly (Claerbout and Karrenbach 1992; Stagge et al. 2019; The Turing Way Community 2021). Although used interchangeably in this article, “reproducibility” and “replicability” can refer to different aspects of the research process. As defined in the 2019 National Academies of Science, Engineering, and Medicine report, Reproducibility and Replicability in Science, “Reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis” and “is synonymous with computational reproducibility,” whereas “Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data” (National Academies of Sciences Engineering and Medicine 2019).

Recent concerns about the reproducibility “crisis” have led to increased public, scientific, and governmental scrutiny of this topic (Collins and Tabak 2014; Baker 2016; Harris 2017; National Academies of Sciences Engineering and Medicine 2017). To date, concerns about research reproducibility have surfaced in dozens of disciplines that range from economics to neuroscience (Button et al. 2013; Ioannidis and Doucouliagos 2013). Individual researchers and research results are often publicly questioned about reproducibility concerns, shaping widespread, public conversations about the integrity and even validity of the scientific process (Dominus 2017; Yong 2017; Bartlett 2018; Lee 2018; Marcus and Oransky 2018). Though outright fraud and serious misconduct remain rare, large-scale replication projects, surveys, and statistical analyses of published research confirm that a much larger proportion of scientists engage in “questionable research practices” (Fanelli 2009), such as data dredging (p-hacking) or HARKing (hypothesizing after results are known) (Kerr 1998; Head et al. 2015).

Simple solutions for achieving reproducible and replicable science are inherently impossible because of the sheer complexity of science. Indeed, culture change can require many different actions, from multiple stakeholders, who each have different priorities (Nosek 2019). Without a broad consideration of the interrelated components of the research enterprise, initiatives may “fail because they are too focused on one aspect of the system, commonly technology, and fail to analyze and understand the complex interdependencies that exist” (Socio-Technical Centre 2021). Even for a specific problem, such as making research data findable, accessible, interoperable, and reusable/reproducible (FAIR) at a particular institution, a multitude of stakeholders may be involved, including the office of research, research compliance unit, information technology (IT), researchers, diverse academic units, and the library (Erway 2013). Bringing together these stakeholders’ many perspectives is essential to ensure successful culture change.

Similarly, although reproducibility cuts across all research disciplines, the issues and ideal solutions may be discipline-agnostic, just as in other facets of RCR education. For that reason, coding and data science-focused courses have emerged in this space and have proven to be helpful to nearly all researchers (Wilson 2014; Teal et al. 2015; Pawlik et al. 2017). For example, skills like “data acumen,” defined as the ability to “[c]ombine many existing programs or codes into a ‘workflow’ that will accomplish some important task; ‘[i]ngest,’ ‘clean,’ and then ‘wrangle’ data into reliable and useful forms; [t]hink about how a data processing workflow might be affected by data issues; [q]uestion the formulation and establishment of sound analytical methods; and [c]ommunicate effectively about properties of computer codes, task workflows, databases, and data issues” are essential for all disciplines relying on any kind of data (National Academies of Sciences Engineering and Medicine (U.S.). Committee on Envisioning the Data Science Discipline: the Undergraduate Perspective 2018). Though these courses can provide some of the essential tools and skills, they do not answer the larger questions about how institutions can and should support students navigating situations involving questionable research practices or other ethical and policy questions. Whereas students, trainees, and other early career researchers look to established mentors for guidance in identifying and tackling research projects, these experts may be less familiar with navigating reproducibility and RCR—researchers at all stages can benefit from education and skill building.

For these reasons, the University of Florida Health Science Center Libraries, in partnership with the George A. Smathers Libraries, hosted a two-day virtual conference on December 2–3, 2020 to focus on the problem of reproducibility education. We brought together representatives from multiple stakeholder groups spanning disciplines to discuss challenges, opportunities, and existing programmatic examples around reproducibility education. The conference, Research Reproducibility 2020: Educating for Reproducibility (2020b), was funded by the U.S. Department of Health & Human Services’ Office of Research Integrity. This article reports on the common themes running through conference presentations; additional details about presenters’ specific training programs can be found in the full recorded presentations (2020a). In this article, as in the conference, we purposefully use a very broad definition of reproducibility that encompasses computational reproducibility, inferential reproducibility, methods reproducibility, replicability, rigor, and transparency.

2 Methods

The meeting was designed to meet three main objectives:

1. Increase knowledge of educational approaches to foster research integrity through reproducibility

2. Provide attendees with networking opportunities for brainstorming and collaboration and provide a forum for discussion

3. Build awareness of reproducibility as a component of res- earch integrity and responsible conduct of research education

2.1 Participants

The conference was designed to bring together educators, administrators, students, and researchers from a wide range of disciplines and institutions. Since the conference was initially planned as an in-person event, there was also a local and regional focus in bringing together participants and speakers from across the University of Florida (UF), the State of Florida, and nationally. With the change to a two-day virtual conference due to COVID-19 restrictions, we were able to dramatically increase the number of participants from outside of Florida.

429 attendees registered for the virtual conference, from 21 countries and 182 unique organizations and institutions. This expansive representation among attendees demonstrates how interest in reproducibility and reproducibility education has permeated the research community. With speakers and presenters coming from disciplines including behavioral science, bioethics, biology, biostatistics, biomedical informatics, clinical psychology, data science, economics, geographical data science, human and molecular genetics, information science, linguistics, medicine, metabolomics, neuroscience, open science, psychology, public health, and social work and from across career stages, discussions and presentations spanned a multitude of perspectives.

2.2 Conference Design

The conference schedule encompassed a variety of modalities and types of content (2020b). Additional conference details, including planning and evaluation materials, are available on our OSF site (Rethlefsen et al. 2021).

2.3 Analysis of Themes

To identify themes, planning team members (MLR, HFN, SLM, KAM, PS, HY) reviewed all of the videos and content produced from the conference, individually for submitted content and in pairs for the keynote and panels. After individual analysis, the group came together for multiple discussions to identify and refine common themes (Table 1). The themes below do not encompass every component of the conference, but rather strive to highlight interdisciplinary approaches and topics discussed by multiple presenters. Ultimately, the themes were organized using the Center for Open Science’s “Strategies for Culture Change,” introduced to the attendees by the conference’s keynote speaker, Brian Nosek, Co-Founder & Executive Director, Center for Open Science (Nosek 2019, 2020). The “Strategies for Culture Change” pyramid developed by Nosek has five levels, from bottom to top: Infrastructure, User Interface/Experience, Communities, Incentives, and Policy (Nosek 2019, 2020). Each level is associated with a guiding principle (in the same order): Make It Possible, Make It Easy, Make It Normative, Make It Rewarding, and Make It Required (Nosek 2019, 2020). Grouping by these guiding principles was a way to show the multiple levels at which people are already working to improve reproducibility.

Table 1 Thirteen approaches and key lessons for interdisciplinary research reproducibility education organized by Nosek’s “Strategies for Culture Change” guiding principles (2019, 2020).

Approach	Key Lesson
	Make It Possible
Practice lifelong learning	Foster an expectation that research practices be reproducible, even if that means changing them over time
Cultivate bottom-up change	Build broad support through localized and targeted approaches to embed and promote open science practices
Sneak in learning	Restructure curricula to embed, tailor, and align reproducible practices with discipline specific research norms
Just-in-time learning	Incorporate reproducible practices in task- and context-oriented training to promote learner satisfaction and retention
	Make It Easy
Engage learners where they are	Target training by career stage and audience type
Balance practical skills development with conceptual learning	Promote broad, flexible mental models by aligning specific practices with theoretical concepts
Learn by doing	Use hands-on experiences in replicating studies to show the importance of transparent and clear reporting of methods
Learn how to learn	Guide learners to reflect on their own thinking, to promote critical thinking, self-directed learning, and peer-learning
	Make It Normative
Establish communities of practice	Establish groups that meet regularly, to maintain, share, and expand on skills in a social setting
Librarians as interdisciplinary leaders	Librarians are uniquely equipped as early leaders and collaborators of open science and reproducible research practices
Effective teamwork skills	Reproducible research practices also enhance the ability of researchers to collaborate and resolve conflict
	Make It Rewarding
Incentives and rewards	Offering incentives and rewards can help shift research culture toward greater reproducibility
	Make It Required
Implement top-down change	Top-down mandated efforts can be used as a pathway to introduce and/or improve rigor and transparency in reporting practices

3 Results

3.1 Make It Possible

3.1.1 Practice Lifelong Learning

Panelists emphasized that embedding reproducibility into their own research was a continuous process, requiring ongoing learning and adjustment of practices. This provided the opportunity to lead by example in their own labs and classrooms and foster an expectation that research practices be reproducible, even if that means changing them over time. One speaker, Rachel Hayes-Harb, summed these up as “intentions to make our own research more reproducible” (Avery et al. 2020). Incorporating this approach to reproducibility in undergraduate education was also an effective entryway—these students could not only learn skills quickly, but if taught concepts around reproducibility, would carry them as part of how they think about research in the future. Reaching undergraduates can help address research culture problems by “embedding research skills and sensibilities development within the responsible conduct of research and open science values and practices” from the beginning, enabling a bottom-up approach to shifting culture (Avery et al. 2020).

3.1.2 Cultivate Bottom-up Change

Panelists described cultivating bottom-up support for reproducible education with localized approaches to embed and promote it. Building a network of like-minded educators is one approach; as described by panelist Brian Avery et al. (2020), identifying colleagues to partner with on the development of reproducible education acts to “bring it out of isolation.” For learners, embedding an infrastructure that promotes and instills open science research practices allows them to integrate these practices with their new discipline-specific knowledge (Avery et al. 2020). Hayes-Harb explained that, as the cornerstone of educational institutions, students’ expectations for social justice, transparency, and trustworthy science should be decision drivers (Avery et al. 2020). Leading by example, mentoring, embracing learners’ values and promoting them as change agents initiates bottom-up change toward reproducible research cultures (Avery et al. 2020; Ball, Riegelman, and Teal 2020).

3.1.3 Sneak in Learning

An emerging discussion in both educator panels was intentional integration of reproducibility training into existing education. For instance, two speakers working with undergraduate students described integrating replication studies into existing curricula in economics and linguistics (Avery et al. 2020; Ball, Riegelman, and Teal 2020). Educators explained that they do not seek permission or support to restructure traditional methods; instead they restructure, “sneak” reproducible practices in, and justify positive outcomes afterward (Avery et al. 2020). These “sneak-in” approaches offer opportunities to embed, tailor and align with discipline-specific research norms, and teach applicable skills and tools. Student speakers expressed a preference for an embedded research reproducibility infrastructure (Allen et al. 2020).

Speakers working with students were asked to discuss managing challenges with introducing and garnering support for reproducibility, when it can be seen as new content that competes for time in the curriculum. Educators across disciplines favored an approach that restructures traditional courses, rather than adding or removing content from the overall curriculum (Avery et al. 2020). For example, this could include using automated tools to calculate a t-test result, rather than traditional, by hand methods (Avery et al. 2020). Student panelists also expressed a preference for restructuring and perceived add-on approaches diminish the value of reproducible research practices (Allen et al. 2020).

3.1.4 Just-in-Time Learning

One speaker, Tracy Teal, explained that many open science practices focus on computational tools, and users’ past disempowered experiences with technology can negatively bias their perceptions (Ball, Riegelman, and Teal 2020). To address this challenge, educators teaching across career stages described creating positive open science learning environments and just-in-time approaches to training are important (Wickes 2020). Positive learning environments make it possible to build confidence, improve self-awareness, and enhance quality of life (e.g., reduce stress and increase efficiency). Just-in-time approaches focus on learning that is task- and context-oriented, such as those used in the Data Carpentries workshops. This style ensures training content is accessible, approachable, aligned, and applicable; which promotes learners’ confidence and satisfaction (Ball, Riegelman, and Teal 2020).

3.2 Make It Easy

3.2.1 Target Learning by Career Stage and Where Learners Are

Presenters emphasized that general concepts around reproducibility are interdisciplinary, that a myriad of skills are required to make research reproducible, and that more detailed, discipline-specific knowledge is required as students and researchers embark on their career paths. One of the lightning talk/poster presenters, Elizabeth Wickes (2020), summed up the dilemma, saying that people at all levels of their careers need to have “on-ramps to learning.” She noted that while introducing skills and concepts to people early may be ideal, middle and senior career individuals want to learn also and educators need to approach them with a growth mindset. Overall, a clear theme emerged that tailoring educational efforts to different career stages and specific audiences was key to engaging learners where they are.

Several presenters worked directly with undergraduate students, including working with them to conduct replication studies as part of coursework and in the lab. Two speakers specifically addressed using and conducting replication studies as a way to engage undergraduate researchers (Avery et al. 2020; Mears 2020). A trio of undergraduates who conducted a replication study as part of one of a similar course presented about their research (Burr, Goldsmith, and Cowan 2020). Undergraduates are a particularly good group to target for these opportunities, because they can simultaneously build their experimental knowledge and scientific reasoning while also learning some of the theory and background behind the experiment.

Many more presenters addressed educating graduate students about reproducibility, both within master’s degree and doctoral degree programs, generally in interdisciplinary settings. This included discussion about credit courses (Avery et al. 2020; Ball, Riegelman, and Teal 2020; Dunleavy and Lacasse 2020; Lapato and Exner 2020), theses (Granell, Sileryte, and Nüst 2020), and workshops (LaPolla and Surkis 2020). Education focused on graduate students was more frequently designed to build specific skill sets, including version control, programming, and data cleaning. Others required their graduate students to produce computationally reproducible research as part of their course-based projects or theses. Several speakers noted graduate students were at a key stage of needing to learn how to learn, so that teaching them how to access and use resources to teach themselves was critical and well-received by students.

Targeting graduate students can help to spread knowledge, skills, and culture. One set of presenters specifically noted that training graduate students had an additional outcome: the students’ learning influenced their faculty advisors’ interest in learning about reproducibility concepts and skills (Lapato and Exner 2020). Richard Ball noted that Project TIER’s new pilot project was to train graduate teaching assistants who work directly with undergraduates. Not only would that training benefit the undergraduates, but it could also influence how the graduate students being trained conduct their own research (Ball, Riegelman, and Teal 2020).

Another commonly targeted demographic was early career researchers, who need and want education, but who have greater limitations on their ability to participate in longitudinal or time-intensive training than undergraduate and graduate students. Multiple ways of engaging early career researchers were discussed, including ReproducibiliTea-style journal clubs that require limited time commitments (Ball, Riegelman, and Teal 2020; Chen et al. 2020; Lapato and Exner 2020), multi-hour to multi-day workshops (Ball, Riegelman, and Teal 2020; Johnson and Goodman 2020), credit courses (Avery et al. 2020), and fellowship-style programs that require more time and effort, or may require the faculty member to obtain release time (Ball, Riegelman, and Teal 2020; Winfree and Barratt 2020).

3.2.2 Balance Practical Skills Development with Conceptual Learning

While the instructional methods described throughout the conference were largely hands-on and application-based, speakers highlighted the importance of tying this practical learning to conceptual learning (Avery et al. 2020). One team of speakers added new skills training to an existing responsible conduct of research course, previously just covering the more theoretical elements of research integrity required by the National Institutes of Health (Johnson and Goodman 2020). The course was updated to include practical training on writing transparently and without “spin” and creating transparent, reproducible code (Johnson and Goodman 2020). Additionally, several speakers grounded their own teaching and evaluation practices using theoretical frameworks (Gottlieb 2020; Wickes 2020).

3.2.3 Learn by Doing

As discussed above, speakers from across various settings and disciplines found it effective and rewarding to teach learners the concepts of reproducible research by having them replicate studies themselves. This hands-on experience builds learners’ experimental knowledge and refines their scientific reasoning skills. In the case of a psychology course, for example, experimental replication can help students produce new knowledge by building a bridge between the theory they are learning and the phenomenon they are attempting to describe (Mears 2020). In attempting to reproduce studies, learners see first-hand what some of the barriers to reproducibility are: missing underlying data, inadequate description of workflows, and lack of access to specific software (Zaringhalam and Federer 2020). Some of these and other data management issues may not be apparent at the small scale that early researchers are familiar with (e.g., working with single spreadsheets, small datasets); learning by doing illuminates these bigger issues (Avery et al. 2020). Performing replications led learners to reflect not only on the broader reproducibility issues facing research, but on their own practices. In their workshop for NIH researchers, for instance, Zaringhalam and Federer (2020) noted that participants saw that their own papers are not reproducible and noted in evaluations that this will affect how they approach their research in the future.

3.2.4 Learn How to Learn

Several speakers identified metacognition or learning to learn as another key to learners moving forward as critical thinkers and thoughtful researchers. In a data science course for biomedical students, for example, Lapato and Exner (2020) added a session on metacognition, spending the time to teach learners how to teach themselves and think about how they would approach self-guided learning in the future. Learning to learn was mentioned in the undergraduate/graduate education panel as a step in helping students become independent critical thinkers (Avery et al. 2020). Likewise, at the student experience panel, speakers discussed the benefits to students teaching others about reproducible research concepts even as they are still learning themselves (Allen et al. 2020).

3.3 Make It Normative

3.3.1 Establish Communities of Practice

Several speakers discussed the importance of communities of practice, or groups of people who meet regularly to practice previously learned skills, acquire new knowledge through peer-teaching, and socialize with colleagues (Allen et al. 2020; Avery et al. 2020; Ball, Riegelman, and Teal 2020; Chen et al. 2020). Communities can operate in different modalities, including in-person and virtual. As noted by the speakers, these communities make regular practice and growth into a social experience and can help to maintain and expand participant capabilities. They can also accelerate the adoption of new approaches, by normalizing ways of thinking and doing within a group of like-minded peers.

Across the panels (Allen et al. 2020; Avery et al. 2020; Ball, Riegelman, and Teal 2020), multiple benefits to communities of practice were mentioned:

• Communities of practice can reinforce training that may have been initially acquired in more formal settings, as well as fill in learning gaps that can result from short-form training activities, such as workshops or boot camps.

• With rapid development of new tools and platforms for reproducibility and open science, communities of practice provide a mechanism for sharing information rapidly, for example, code review for computational research (Avery et al. 2020)

• As a social group of peers, communities of practice enable individuals to discuss lab culture and ethical issues in a low-intensity setting, without the fear of retaliation or conflict that may occur if issues are brought up directly as a confrontation or reported to, for example, ombudspeople or university research integrity offices.

Even with these benefits and interest, recruiting interest in communities of practice and sustaining their activities can be challenging. As might be expected, ReproducibiliTea journal clubs, discussed by Lapato and Exner (2020), Chen et al. (2020), and Riegelman (Ball, Riegelman, and Teal 2020), have a reusable set of materials for jumpstarting local chapters. Although some decrease in attendance over a semester is natural, most attendees are highly motivated and small incentives such as providing food and tea can be helpful. In addition, framing activities as practices that make research easier and more effective can be more attractive to audiences than focusing on reproducibility explicitly (Ball, Riegelman, and Teal 2020).

3.3.2 Librarians as Interdisciplinary Leaders

Librarians in academic libraries and health sciences libraries are creating effective, interdisciplinary opportunities for educating about reproducibility and open science concepts and skills (Ball, Riegelman, and Teal 2020; Lapato and Exner 2020; LaPolla and Surkis 2020; Wickes 2020; Zaringhalam and Federer 2020). Multiple speakers throughout the conference mentioned the role of librarians as early leaders and collaborators in their institutions. Several speakers mentioned libraries’ role as an educational hub on campuses, serving as touchpoints for interdisciplinary skills and researchers. It was also emphasized that librarians’ established partnerships and connections enabled their educational programs to reach a broader audience (Ball, Riegelman, and Teal 2020; Lapato and Exner 2020; LaPolla and Surkis 2020). This might be through individual consultations, library workshops, guest lectures or workshops for a specific class or cohort, hosting reproducibility-focused journal clubs, teaching credit courses, organizing code-a-thons aiming at reproducing published studies, or hosting other events (Ball, Riegelman, and Teal 2020; Lapato and Exner 2020; LaPolla and Surkis 2020; Wickes 2020; Zaringhalam and Federer 2020).

Librarians bring skills to the reproducibility space that others may not. Their special knowledge and skills concern organization, description (metadata), interoperability, taxonomies and ontologies, literature searching, systematic reviews, scholarly communication (including use of reporting guidelines), and publishing/dissemination of research (Ball, Riegelman, and Teal 2020; Lapato and Exner 2020). In addition, librarians have the ability to translate and explain complex reproducibility concepts in basic terms, due to their experience teaching interdisciplinary learners.

3.3.3 Effective Teamwork Skills

Although reproducible research is often framed as an external benefit (i.e., the research is more reliable and useful to others), the practices that promote research reproducibility can also facilitate working on a research project alone or in collaboration (Ball, Riegelman, and Teal 2020). For example, documenting the biological materials using Research Resource Identifiers and tools like Addgene can make it easier to write up the methods section when publishing, and also helps ensure that data can be aggregated or compared across experiments that are done in different locations (e.g., a collaborator’s lab) or at different times (e.g., a follow-up study exploring related questions) (Abitua and Bachle 2020). Indeed Taylor and Marsiske (2020) describe how principles in clinical trials can be implemented in single laboratory studies to provide a foundation for rigor, reproducibility, and transparency.

Establishing collective norms for research has impacts beyond just reproducibility, but also extends to better pathways for resolving conflict and other issues. As discussed in the student experience panel, there are differing ideas about which practices are acceptable, which are considered questionable research practices, and which may rise to the level of scientific misconduct (Allen et al. 2020). Creating and communicating shared norms in a collaboration or lab setting can head off these issues, and also enable more effective teamwork by clearly laying out responsibilities and rewards (including authorship order). In addition, these policies can also provide information on research processes when working with grants offices, IRB panels, systematic review services, and more. For trainees, this can be invaluable, as many do not start out being familiar with the intricacies of research (Allen et al. 2020).

3.4 Make It Rewarding

3.4.1 Incentives and Rewards

In his keynote presentation, Nosek (2020) identified rewards and incentives as another necessary pillar in creating broad cultural change and widespread adoption of reproducible research practices. One challenge here is that the commonly held values of transparency and research quality do not match the current rewards. Researchers are rewarded for high rates of publication, with positive results more likely to get published. Together, these rewards incentivize selective reporting and quantity of research output over quality, making questionable research practices more likely. Nosek (2020) discussed two publisher and funder models designed to shift the incentives: registered reports and registered proposals. Registered reports shift peer review to occur after study design and before data collection, which incentivizes asking important research questions and developing good study design, rather than producing positive results or focusing purely on innovation. Registered proposals are reviewed by journal publishers and funders together and, when accepted, guaranteed both funding and publication, again incentivizing good study design and reproducible practices rather than positive results alone. Ideally, incentives should be offered at all levels so that institutional, publisher, funder, and professional society are able to affect meaningful change.

Several presenters in other parts of the conference made mention of the relevance of considering rewards and incentives specifically in education and training for reproducibility. For instance, Tracy Teal referred to a study where researchers reported most wanting training from a particular organization, even more than funding; this suggests that in some cases individuals may see intrinsic rewards in education around research practices (Ball, Riegelman, and Teal 2020). In a more specific example, another presenter described an online course on research transparency and reproducibility being coupled with a research transparency badge for those completing the course, which they could apply to their professional profile and/or e-mail signature (Gottlieb 2020). Another set of presenters recruited participants to their various training offerings by emphasizing the benefits to the job market in having reproducible research skills (Lapato and Exner 2020).

3.5 Make It Required

3.5.1 Implementing Top-down Change

Though conference participants emphasized the importance of bottom-up, grassroots efforts to spread change and influence communities, many speakers also noted the importance of top-down efforts by institutions, funding agencies, and journal publishers. For instance, Johnson and Goodman (2020) noted that their workshops were designed to meet and exceed funder mandates for both responsible conduct of research and rigor and reproducibility education. They used the funder mandated education as a pathway to introduce better scientific writing practices and transparent coding skills. Multiple speakers commented on top-down federal mandates as a starting place for their work, though also noted that when educational efforts are construed as compliance or administrative issues, they are less valuable to learners, as they are often too abstract to translate to everyday efforts (Johnson and Goodman 2020; Nosek 2020). Lab-based discussion and reasoning about ethical practice involving authorship and data management policies and training, for instance, are lessons in research ethics closer to everyday concerns (Nosek 2020).

Improving transparency of research reporting through institutional and journal publisher policies was frequently discussed by speakers. Both reporting guidelines (i.e., CONSORT, ARRIVE 2.0, etc) and NIH rigor and reproducibility principles were lauded (Reynolds 2020; Selfe 2020), but speakers made it clear these are not as useful unless enforced by journal editors (Menke et al. 2020). Several presenters examined the transparency of published studies and preprints, which are still often poorly reported or of poor quality (Cooper et al. 2020; Menke et al. 2020). Ideas to implement templates for registered reports and interventions to improve reporting quality using automated messages to authors on how to improve their preprints were proposed (Eckmann et al. 2020; Meghreblian, Chambers, and Tzavella 2020). Finally, speakers also suggested that departments and institutions should develop policies, including tenure and promotion policies, that require transparency and rigor (Ball, Riegelman, and Teal 2020).

4 Discussion

Much reproducibility education has previously been highly discipline-centric, focusing on specific skills, tools, and techniques, but reproducibility is both an interdisciplinary concept and an integral component in all research. By bringing together a widely divergent group of individuals from fields as unique in their research practices as geography and genetics, different levels of education and experience, and multiple countries, we were able to identify commonalities to inspire new opportunities for learning and growth for our diverse research communities.

Through thematic analysis of the conference content, thirteen interdisciplinary approaches and lessons for reproducibility education emerged and are summarized in Table 1. These themes are sometimes in opposition (“cultivating bottom-up change” and “implementing top-down change” the most obvious example), but the overarching message of the conference was that multiple approaches are both necessary to address the complexities of implementing reproducible research and welcomed by researchers, who span disciplines and career stages and are therefore not a monolithic group with identical motivations and needs. Whereas top-down policy changes may be effective to spur institutions and principal investigators to make major, potentially costly changes, bottom-up approaches can engage those who are more curious and flexible in making incremental changes to their practices—and who may band together to shift norms through collective efforts. Other approaches like “just-in-time learning,” “sneak in learning,” and “learn by doing” are more directly complementary to one another and can be used within the same initiatives to scaffold approaches for a wide range of learners.

The planning and execution of this conference also created an opportunity for the organizers, speakers, and attendees to enact and observe these principles in action. Organizing the conference with a focus on interdisciplinary education and later moving it into a virtual venue enabled a wide range of participants that might not otherwise have met, connected, or collaborated. In parallel, the operations of the conference were designed to be as open as possible to engage attendees in a unique, virtual community of open and reproducible practices—resources from the conference remain freely available and open for use, including videos of most presentations (2020a), posters and presentations (2020c), abstracts (2020d), and the collaborative notes from the sessions (2020e). Table 2 includes additional resources that were suggested by conference speakers. We hope that the themes we have identified in this article and these products from the conference will inspire more educators to collaborate to explore effective and innovative methods of reproducibility education.

Table 2 Select resources shared by speakers and attendees.

Name of Resource	Web Address	Audience/Purpose
ReproducibiliTea Journal Clubs	https://reproducibilitea.org/journal-clubs/	Journal club leaders; provides materials for reuse and ideas for content
Berkeley Initiative for Transparency in the Social Sciences	https://www.bitss.org/	Learners, instructors; exercises, workshops and programs, and templates
The Carpentries Curricula	https://carpentries.org/workshops-curricula/	Learners, instructors; provides modular, step by step lessons on data and code-related topics
Center for Open Science	https://www.cos.io	All; platform of open science tools and initiatives, including registered reports
Reproducibility for Everyone	www.repro4everyone.org	Learners, instructors; workshop lessons that can be used and adapted
GitHub Learning Lab	https://lab.github.com/	Learners; tutorials on using GitHub
Frictionless Data for Reproducible Research Fellows Programme	https://fellows.frictionlessdata.io/	Learners; contains lessons, resources, and syllabus for a fellowship program
Data Carpentry for Biologists	https://datacarpentry.org/semester-biology/	Learners, instructors; semester-long syllabus and class materials, lessons
ProjectTIER	https://www.projecttier.org/	Learners, instructors; exercises, workshops and programs, and templates
Psychology’s Credibility Revolution course	https://docs.google.com/presentation/d/e/	Learners; course by Carleton College
	2PACX-1vREwmeADm5WZCBfjAKyAkzYC	faculty member Julia Strand with
	_cQb7xtYv6C7e_wBJx0BUhJ3lFY0WABpI	lectures, exercises, and more
	b0XotJake0Xrh-CC6g1Vwy/pub?start =fal
	se&loop =false&delayms =99999
	999#slide =id.g950fb47fc8_0_16
STAT 545: Data wrangling, exploration, and analysis with R	https://stat545.com/	Learners; course by University of British Columbia faculty member Jennifer Bryan

5 Conclusion

Reproducibility educational needs may vary by discipline, but there are transdisciplinary common threads that can be explored at all levels of learning. The conference’s themes are a snapshot of emergent practices that can be used to embed reproducibility education throughout a lifelong learning process.

Ethics Review

The evaluation of Research Reproducibility 2020 was approved exempt by the University of Florida Institutional Review Board (IRB202000619). All Research Reproducibility 2020 speakers agreed to be filmed and the recordings publicly shared.

Disclosure Statement

Melissa Rethlefsen and Hao Ye are founders of the ReproducibiliTea journal club at the University of Florida. Katherine MacWilkinson is an employee of the University of Florida Conference Department. Hao Ye is a certified Carpentries instructor and trainer and is a member of the Code of Conduct Committee for Repro4Everyone.

Additional information

Funding

Research Reproducibility 2020: Educating for Reproducibility was supported by the Department of Health and Human Services Office of Research Integrity (ORIIR190046). The content is solely the responsibility of the authors and does not necessarily represent the official views of the U.S. Department of Health & Human Services.