A Model for an Undergraduate Research Experience Program in Quantitative Sciences

Abstract

We developed a summer research experience program within a freestanding comprehensive cancer center to cultivate undergraduate students with an interest in and an aptitude for quantitative sciences focused on oncology. This unconventional location for an undergraduate program is an ideal setting for interdisciplinary training in the intersection of oncology, statistics, and epidemiology. This article describes the development and implementation of a hands-on research experience program in this unique environment. Core components of the program include faculty-mentored projects, instructional programs to improve research skills and domain knowledge, and professional development activities. We discuss key considerations such as fostering effective partnership between research and administrative units, recruiting students, and identifying faculty mentors with quantitative projects. We describe evaluation approaches and discuss post-program outcomes and lessons learned. In its initial two years, the program successfully improved the students’ perception of competence gained in research skills and statistical knowledge across several knowledge domains. The majority of students also went on to pursue graduate degrees in a quantitative field or work in oncology-centric academic research roles. Our research-based training model can be adapted by a variety of organizations motivated to develop a summer research experience program in quantitative sciences for undergraduate students. Supplemental files for this article are available online.

Keywords:

• Analysis of biomedical data
• Applied statistics internship
• Computational biology
• Experiential learning
• Mentoring
• Statistical training

1 Introduction

The field of oncology acts on data (DFCI 2020). Data-centric decisions govern all areas of cancer, including screening (Margolies et al. 2015), diagnosis (Manogaran et al. 2018), drug development (Workman, Antolin, and Al-Lazikani 2019), treatment delivery (Ngiam and Khor 2019), follow-up care (Foerster et al. 2020), and identifying novel mutations to understand disease biology (Poulos and Wong 2019), to name a few. Oncology data are large in (i) volume, due to petabytes of information; (ii) variety, due to multimedia data, such as images, audio, video, and texts; (iii) veracity, due to relevant information embedded within a large volume of data; and (iv) velocity, due to urgent calls to action to control the disease (Otte 2021). The oncology research community is increasingly seeking to transform disease management through effective application of statistical and computational tools to these data. Demand for people with quantitative skills is, therefore, rising in the cancer biomedical enterprise.

To address this surge, universities are expanding degree programs in myriad quantitative areas, including statistics, biostatistics, epidemiology, health policy, health economics, bioinformatics, biomathematics, quantitative life science, data science, health informatics, and computational biology. These programs often incorporate niche curricula in regularized regression techniques, statistical machine learning, artificial intelligence, network analysis, Markov models, combinatorial optimization, and data visualization, to name a few areas. As recommended by the Guidelines for Assessment and Instruction in Statistics Education (GAISE) College report, classroom education should engage students in quantitative thinking, conceptual understanding, and active learning through didactic lectures and by integrating real data with a context and purpose (GAISE College Report ASA Revision Committee 2016). These approaches emphasize academic rigor and professional development to adequately prepare students for success in future careers. Summer research experience programs can supplement academic learning by offering students unique professional and personal skills that cannot be gained solely from classroom instruction. In these apprentice-style programs, students gain valuable disciplinary knowledge, increase their intellectual skills in inquiry and analysis, learn the research process, view the culture and practice of science, witness how scientific decisions are made, develop dexterity in working with data, build a professional community, identify career paths, and develop enduring relationships with mentors and peers (Lopatto 2010; Hernandez et al. 2018; Wilson et al. 2018). Summer research experience programs also allow employers to evaluate emerging talent, expose young professionals to their brand, learn the students’ perspectives, and enjoy the benefits of fresh thinking.

Thus, student and employer demand for summer research experience is increasing, especially in the field of quantitative sciences. This is evident from the American Statistical Association’s announcements of summer opportunities for students; the number of employers offering summer research experience programs increased by 57% from 37 in 2015 to 58 in 2019 (Table 1). Summer research experience programs are available for both undergraduate and graduate students from several organizations. Industries offer more graduate than undergraduate summer positions. In contrast, academic, government, and nonprofit research organizations offer more summer positions to undergraduates. Yet, the cumulative numbers of these organizations offering summer positions each year are fewer than industries. Likewise, the cumulative number of summer undergraduate positions across these organizations each year is fewer than the number of summer graduate positions in industry (Table 1).

Table 1 Characteristics of quantitative summer research experience programs announced in the American Statistical Association’s online and print venues between 2015 and 2019 for the following types of organizations: academia, government, nonprofit research, and industry.

Year	Academia			Government			Nonprofit research			Industry
	N	U	G	N	U	G	N	U	G	N	U	G
2015	4	34	2	8	34	14	3	10	10	22	1	76
2016	5	44	2	6	29	12	2	10	3	30	0	132
2017	8	22	22	8	52	5	6	18	6	34	3	116
2018	9	27	46	10	52	7	5	43	5	31	3	122
2019	11	43	3	5	27	5	6	46	10	36	1	138

The NIH-funded Quantitative Sciences Undergraduate Research Experience (QSURE) established at Memorial Sloan Kettering Cancer Center (MSKCC) in 2018, expands opportunities by training undergraduate students in a range of quantitative methods and topics in cancer biomedicine and population sciences for 10 weeks between June and August every year (MSKCC 2021). A distinctive feature of QSURE is its position within a freestanding cancer center, where research activities cover the full spectrum of cancer prevention, diagnosis, treatment, and survivorship, as well as basic science in cancer etiology and control. This offers unique opportunities for undergraduates to gain hands-on experience in oncology study design, technology-driven approaches to data acquisition and analysis, genomic investigations for risk management and drug development, data sources, and methods for studying healthcare delivery, studies of determinants and disparities in health outcomes, effective science communication, and team science interactions.

In a freestanding cancer center, the traditional audience for quantitative academic instruction includes oncology fellows engaged in patient care and post-doctoral researchers conducting pre-clinical laboratory research. Establishing a summer research program for undergraduate students, therefore, requires a nuanced partnership between research and administrative units to identify or create faculty-mentored research projects, relevant instructional programs, and professional development activities throughout the center. This article describes the steps taken to design and implement QSURE in this unique environment. We describe the feasibility of imparting practical quantitative experience to undergraduate students through QSURE and the outcomes observed during the first two years of the program. The approaches described for QSURE should be useful for any organization seeking to develop a summer undergraduate research experience program. Throughout, we refer to the undergraduate students participating in the QSURE program as “fellows.”

2 Methods

2.1 Advertising and Outreach

To recruit QSURE fellows, promotional materials such as the program website and poster were advertised nationally through multiple avenues. One of the most effective channels was direct communication with faculty members in university quantitative science departments, such as biostatistics, statistics, mathematics, computer science, data science, epidemiology, sociology, and public health. The program was also advertised on internship listings coordinated by the American Statistical Association (ASA).

To increase the diversity of our applicant pool, we advertised QSURE through specialized programs and organizations that support students from different socioeconomic and racial backgrounds across the country—for example, the Society for Advancement of Chicanos/Hispanics and Native Americans in Science and Pathways to Science.

2.2 Application and Selection Process

Undergraduate students applied through a secure online portal. The application included basic demographic and schooling information, a resume, a statement of interest, and a letter of recommendation. Instead of a transcript, only a letter of good standing from the school was requested to reflect our dedication to evaluate applicants holistically, without focusing on grades and school prestige.

Completed applications were reviewed by the program steering committee consisting of three faculty members—one biostatistician who collaborates closely with cancer treatment programs, one biostatistician who collaborates closely with population science and cancer genomics programs, and one health outcomes researcher who collaborates closely with health policy and healthcare delivery research programs—to select a cohort of 10 QSURE fellows. An ideal fellow was one who demonstrated:

1. a desire to pursue graduate education or a career in quantitative sciences in health,

2. motivation and aptitude for independent and collaborative research, and

3. an inquisitive mind.

2.3 Development of a Comprehensive Program

We developed a training program that included both mentored research experience and professional development through the core components summarized below.

2.3.1 Mentored Research Experience

For each fellow, the centerpiece of the program was a faculty-mentored research project. Faculty mentors were identified within the Department of Epidemiology and Biostatistics, which includes biostatisticians, epidemiologists, and health services researchers. Following a solicitation from the QSURE leadership team, potential mentors proposed a quantitative research project that could be completed within an eight- to ten-week period. Projects differed depending on the mentor’s area of expertise, but they all required statistical and computational analysis of data that could be performed by an undergraduate with at least one semester of college-level statistics and strong quantitative aptitude. The steering committee matched selected fellows to mentors whose projects best aligned with the student’s skills and area of interest. Mentor input was also sought before arriving at a final decision about the match. For projects that involved nonexempt human subjects research, QSURE fellows completed training in human subjects research and were added to IRB protocols as necessary prior to any interaction with the data. One week before the program started, mentors provided their assigned fellows with reading material to introduce the fellow to the content area of the project.

At the beginning of the program, each mentor-fellow pair completed a Project Work Plan (included in the supplementary materials). This work plan allowed fellows to organize their project and schedule in a structured manner; the mentor-fellow pair then developed a set of learning objectives and identified corresponding project-specific activities. Mentor-fellow pairs were encouraged to revisit the work plan throughout the program and adapt the work plan as necessary. The work plan was not intended to be binding; rather, its purpose was to facilitate discussions between the mentor and fellow to set mutual goals and expectations.

At the end of the program, fellows were expected to present their project findings to an audience of mentors, peers, and invited faculty. In addition, the fellows submitted a structured abstract that summarized major aspects of their projects. Project abstracts and presentations were archived by QSURE staff as a resource for the program.

2.3.2 Statistical Software Training

Prior to program initiation, we surveyed all mentors about the software needs for their fellows’ projects. The primary statistical software used in the QSURE program is the R programming language (R Core Team 2021), a free statistical software widely used within the quantitative sciences community. A multi-session R software training was led by research biostatisticians at the Department of Epidemiology and Biostatistics to cover basic functions and advanced topics listed below:

• R data structures and types

• Overview of tidyverse

• Data manipulation and cleaning

• Data visualization

• Model building

• Writing functions

• Project organization

• RMarkdown

• Coding reproducibility best practices

• Case study in coding

• Introduction to git and GitHub

All fellows attended the software training sessions. As each mentored research project required different statistical programming needs, the research biostatisticians also held office hours to assist fellows with any programming needs. Additionally, we facilitated supplemental training in other statistical software and applications (e.g., SAS software) for fellows or mentors who sought it.

2.3.3 Training in Responsible Conduct of Research

QSURE’s responsible conduct of research (RCR) training involved a weekly facilitated discussion seminar built upon the National Academy of Science’s publication, “On Being A Scientist” (NAS 2009). Seminar topics included:

• research collaboration and mentoring,

• human subjects, privacy, and HIPAA,

• rigor, reproducibility, and responsibility in data analysis,

• data safety, security, and sharing,

• responsible authorship, peer review, and conflicts of interest,

• research misconduct, whistleblowing, and

• social responsibility and contemporary ethical issues in cancer research.

This training was intended to allow for peer-to-peer learning through a critical review of case studies, contemporary media reports, and small group presentations. When possible, readings, and discussions addressed experiences relevant to undergraduate fellows. For example, for the discussion on human subject protections, fellows were asked to find, review, and comment on their own university’s policies for students as research subjects.

2.3.4 Workshop on Effective Scientific Presentation

The fellows received training in the development and delivery of oral scientific presentations, led by an expert in scientific communication. The training prepared fellows to summarize their mentored research projects in the form of a clear and compelling narrative. These are all transferable skills that can be sustained throughout their professional careers.

2.3.5 Quantitative Science Lecture Series

A series of didactic lectures highlighted areas of biomedical cancer research pursued by departmental and institutional faculty to advance cancer diagnosis and care. The objective of the lecture series was to showcase cutting-edge quantitative methods and applications in cancer, with the overarching goal of inspiring the fellows to pursue graduate education or professions in cancer-related quantitative science. Faculty members from the department and from across the institution presented one-hour lectures on various topics (see supplementary material). The lecture series also provided an opportunity for fellows to interact and network with cancer researchers in a professional setting.

2.3.6 Program Symposia

Two symposia offered fellows opportunities for convivial discussions with invited experts engaged in creative uses of quantitative sciences. These sessions promoted introspection and dialogue through which fellows gained invaluable advice as they begin their journeys as quantitative scientists. Symposium topics in the first 2 years of the program included the intersection of data and performing arts, data and journalism, and data and science policy.

2.3.7 Field Trips and Excursions

QSURE fellows also visited unique science facilities in New York City, including MSKCC’s Integrated Genomics Operation Laboratory and Outpatient Surgery Center. Fellows learned about graduate education and graduate student life and professional opportunities from area academic institutions. Interactions during these field trips also allowed the fellows to get to know their peers in a relaxed setting and further develop a sense of camaraderie.

2.4 Program Personnel

Because QSURE is housed within a freestanding cancer center where statistical training does not occur at the level of a traditional academic institution, a formal nonoverlapping three-part structure was put in place to operationalize this program successfully:

1. Program administrators participated in recruitment efforts by distributing program information, liaised with hospital administration for legal aspects of onboarding fellows and access to the computing environment, and ensured that local, state, and institutional requirements were fulfilled in a timely manner.

2. Program directors were responsible for developing and implementing the program components within the institutional boundaries defined by the program administrators.

3. An external advisory committee of six senior researchers from peer institutions with extensive experience in mentoring and training students and junior researchers provided recommendations and constructive critiques to sustain and continually improve the program.

2.5 Evaluations

The short-term goals of QSURE are to ensure that fellows gain increased levels of competence in various statistical, scientific, and professional topics at the end of the program relative to program inception. Two structured surveys and an open-ended mid-program survey were implemented to assess the short-term goals of the program. The baseline and end-of-program structured surveys were designed in a pre-post format to evaluate perceived levels of competence in two skills domains (analytical and communication) and two knowledge domains (statistical and other scientific knowledge) gained through the program (see supplementary materials). In the baseline survey, fellows rated their current (i.e., initial) perceived competency at the start of the program and the desired competency at the end of the program for each domain on a five-point scale defined as follows: 1 = not at all proficient or confident, 2 = low level of proficiency or confidence, 3 = medium level of proficiency or confidence, 4 = high level of proficiency or confidence, and 5 = very proficient or confident. In the end-of-program survey, fellows rated their perceived competency achieved by the end of the program, using the same five-point scale. The fellows did not have access to their responses to the baseline survey when completing the end-of-program survey.

The mid-program survey consisted of open-ended questions for the fellows to communicate areas that worked well and parts that needed improvements. The purpose of this survey was to identify potential problems during the program and readily implement changes as appropriate in order to improve the fellows’ experiences.

The long-term goal of QSURE is to ensure that fellows pursue and make sustained contributions to quantitative professions, possibly focusing on cancer. We followed fellows through email contacts, LinkedIn professional social media, and post-program interactions between individual faculty and their assigned fellows to track higher education and professional careers of QSURE alumni.

2.6 Statistical Analysis

The cohort size per year was 10 fellows. We conducted descriptive analyses of the baseline and end-of-program structured surveys for 20 fellows by combining the 2018 and 2019 cohorts. We categorized fellows’ response to each question as Low (if the response was 1 or 2), Medium (if the response was 3), and High (if the response was 4 or 5). We prepared alluvial plots using the R programming language to examine changes in perceived competency levels between start and end of the program for each question. Ideally, we would expect students to have high perceived levels of competency at the end of the program relative to the start of the program.

3 Results

There were 73 applicants in 2018 and 88 in 2019. Ten fellows were admitted to the QSURE program each year. The majority of applicants self-reported as female (75% in 2018, 80% in 2019) (Table 2). The distribution of race across applicants was similar in each year, with 56% of applicants identifying as non-White in 2018 and 2019. Most students were attending schools in the Northeast region (68% in 2018, 62% in 2019).

Table 2 Characteristics of selected fellows for the 2018 and 2019 programs.

		2018		2019
		All applicants (N = 73)	Admitted fellows (N = 10)	All applicants (N = 88)	Admitted fellows (N = 10)
Sex	Female	55 (75%)	6	70 (80%)	7
	Male	18 (25%)	4	18 (20%)	3
Race	White	32 (44%)	4	39 (44%)	3
	Black	6 (8%)	1	6 (7%)	0
	Asian/Pacific Islander	24 (33%)	5	36 (41%)	6
	More than one race	6 (8%)	0	5 (6%)	1
	Other/Did not disclose	5 (7%)	0	2 (2%)	0
Ethnicity	Hispanic	3 (4%)	0	8 (9%)	1
School year completed	Freshman	6 (8%)	0	7 (8%)	0
	Sophomore	20 (27%)	1	35 (40%)	1
	Junior	42 (58%)	7	45 (51%)	8
	Senior	5 (7%)	2	1 (1%)	1
School region${}^{\ast }$	Northeast	50 (68%)	6	55 (62%)	7
	Midwest	7 (10%)	1	11 (12%)	0
	South	13 (18%)	3	20 (23%)	3
	West	3 (4%)	0	2 (2%)	0

${}^{\ast }$Northeast = Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island, Vermont, New Jersey, New York, Pennsylvania.

Midwest = Illinois, Indiana, Michigan, Ohio, Wisconsin, Iowa, Kansas, Minnesota, Missouri, Nebraska, North Dakota, South Dakota.

South = Delaware, Florida, Georgia, Maryland, North Carolina, South Carolina, Virginia, District of Columbia, West Virginia, Alabama, Kentucky, Mississippi, Tennessee, Arkansas, Louisiana, Oklahoma, Texas.

West = Arizona, Colorado, Idaho, Montana, Nevada, New Mexico, Utah, Wyoming, Alaska, California, Hawaii, Oregon, Washington.

Research projects conducted by the admitted fellows in 2018 and 2019 are summarized in the supplementary material. Among the 20 projects across both years, eight (40%) focused on biostatistics (e.g., statistical methodology, prediction modeling, clinical trials), five (25%) focused on epidemiology, five (25%) focused on computational oncology, and two focused on health outcomes and policy.

3.1 Pre- and Post-program Evaluation

Figures 1–4 show alluvial plots demonstrating changes in perceived competency in various areas between start and end of the program. In general, the fellows reported a higher level of perceived final than initial competency in all domains. All 20 (100%) fellows reported high perceived competency in data analysis and coding skills by the end of the program. Only one (5%) fellow reported high perceived initial competency in health services research knowledge and two (10%) fellows reported high perceived initial competency in cancer knowledge and genetics/genomics knowledge. For most fellows, these perceptions improved over the course of the program, with a total of nine (45%), eleven (55%), and eight (40%) fellows reporting high perceived final competency in health services knowledge, cancer knowledge, and genetics/genomics knowledge, respectively.

Fig. 1 Alluvial plots of survey responses related to the “Analytical Skills” domain given by 20 QSURE fellows for initial perceived level of competency (from baseline survey) and final perceived level of competency (from end-of-program survey). Survey responses were categorized as Low (if the response was 1 or 2), Medium (if the response was 3), and High (if the response was 4 or 5). The number of fellows in each category is presented in brackets; the width of the stream reflects the proportion of fellows in each pre-post pair of response categories.

Fig. 2 Alluvial plots of survey responses related to the “Communication Skills” domain given by 20 QSURE fellows for initial perceived level of competency (from baseline survey) and final perceived level of competency (from end-of-program survey).

Fig. 3 Alluvial plots of survey responses related to the “Statistical Knowledge” domain given by 20 QSURE fellows for initial perceived level of competency (from baseline survey) and final perceived level of competency (from end-of-program survey). Two students did not provide perceived level of competency for prediction modeling.

Fig. 4 Alluvial plots of survey responses related to the “Other Scientific Knowledge” domain given by 20 QSURE fellows for initial perceived level of competency (from baseline survey) and final perceived level of competency (from end-of-program survey). One student did not provide perceived level of competency for cancer knowledge.

For survival analysis, 13 (65%) fellows reported low perceived initial competency, of which five (38% of 13) reported high and three (23% of 13) reported medium perceived final competency. Few fellows reported reduced perceived final relative to initial competency levels for certain domains. Notably, for regression modeling, 15 (75%) fellows reported high perceived initial competency, of which two (13% of 15) reported medium perceived final competency. Further, of four (20%) fellows reporting high perceived initial competency for epidemiology knowledge domain, two (50% of four) reported medium perceived final competence, and of five (25%) fellows reporting medium perceived initial competency for genetics and genomics knowledge, one (20% of five) reported low perceived final competency.

3.2 Mid-program Evaluation

Mid-program evaluations were constructive and positive (Table 3). Overall, fellows indicated enthusiasm for the program components. There were no major negative comments or concerns warranting any changes to the program.

Table 3 Selection of student open-ended feedback from mid-program evaluations.

Topics	Positive feedback	Constructive feedback
Responsible conduct of research workshop	“I think the weekly topics and discussions are interesting and important. I have learned a lot about the scientific community and process.” “The RCR seminars have been great. I enjoy the discussions on topics that are not commonly touched upon in school/research settings. I also appreciate the readings that are assigned to us every week.” “I learned a lot from the discussion and real-world cases. The seminars make me more aware of the ethical issues in my daily life and research.” “This is by far my most favorite part of the internship! While statistics is thought of as a technical subject, these seminars are telling us something new. That, it is not just the math, that is, important but also policies and other socioeconomic factors.”	“Sometimes, it was not easy to finish all the readings when we were busy working on our projects, but most often, the reading level was appropriate.” “Can be a little bit boring, maybe can do something more interactive other than discussions. Reading can be a bit heavy and repetitive.”
R programming tutorials	“I have worked with R a lot and I still learned a lot of new techniques/packages in this seminar that I hadn’t used before.” “One aspect that stood out was the instructor’s use of real-time example. As a result, we could easily follow the presentation while at the same time experiencing how different functions come together during statistical analysis.” “These tutorial series have been helpful in seeing new ways to code, specifically with the tidyverse package, which I have already started using on my own. Further, it was nice to see some of the good practices in coding from someone who does it on a daily basis and has to be organized in how they communicate the information to others.” “Even for the topics I was familiar with, the tutorials were really well presented and taught. I learned a lot about other practical aspects of coding, such as with markdown and projects.”	“Perhaps allocating three tutorials each two hours each would help provide a more comprehensive overview without feeling like we had to get everything in a condensed amount of time.” “Much information but at times can be a little bit hard to follow for some but too easy for others. However, this is understandable considering the different background everyone comes from. It is more helpful to either look up documents online or ask [the instructors] for individual help instead of a tutorial for everyone.”
Workshop on how to give scientific presentations	“Such a great opportunity that changed the way I looked at making PowerPoints and how best to present scientific information. Truly one of those experiences that becomes a ‘before’ and ‘after”’ “It was really useful although it is hard to shift the style at the beginning. After the presentation, I think I start to think more from the audience’s perspective when I prepared my presentation.” “The workshop definitely changed how I viewed giving presentations. I was able to incorporate a few key points of her ideas into my own presentation. I’m glad we got this experience.” “Very engaging and probably one of the more practical workshops that we’ve had so far because it taught us how to develop our presentation skills. Should definitely keep this workshop for the future.”

NOTE: Responses were presented for the combined 2018 and 2019 QSURE cohorts (N = 20 fellows, with 10 fellows per cohort).

Fellows appreciated the use of real-life examples based on studies from the cancer center in the statistical software training. Some fellows felt that they learned new programming techniques even though they had previously worked with the R programming language. Fellows were enthusiastic about the RCR workshop, acknowledging that they learned a lot about the scientific community and processes, including topics that are rarely addressed in school, through real-world cases. They also felt that the RCR workshop made them more aware of ethical issues in daily life and research. Fellows also found the workshop on scientific presentations as a great opportunity that changed the way they prepare their presentations.

Despite the overall enthusiasm, two fellows who were already well-versed with the R programming language felt that they did not gain new information from QSURE’s statistical software training. One fellow felt that the training was packed with information and that it was more beneficial to seek individual help from the training instructor or to look up online. One fellow felt that it was not easy to complete all the required readings for the RCR discussions while working on their project, although they felt that the reading level was appropriate.

3.3 Post-program Fellow Outcomes

As of the time of writing, 15 of 20 (75%) fellows went on to pursue graduate degrees in biostatistics, epidemiology, or computational genomics (six at the PhD level) after graduating from college/university; the remaining fellows are completing undergraduate studies, pursuing medical degrees, or working in quantitative fields (Table 4). One fellow successfully wrote and published the findings of their QSURE project (Latour et al. 2020). One student presented their summer project at the Joint Mathematics Meeting, in Baltimore, MD, and another student presented at their university’s mathematics department seminar.

Table 4 Post-program outcomes of fellows for the combined 2018 and 2019 QSURE cohorts (N = 20 fellows).

Post-program activity	N (%)
Undergraduate program	1 (5%)
Graduate program (MS/PHD)
Biostatistics	12 (60%)
Epidemiology	2 (10%)
Computational genomics	1 (5%)
Medical program	2 (10%)
Professional career	2 (10%)

NOTE: In total, 75% of program participants enrolled in graduate programs.

3.4 Alumni Engagement

Post-program, QSURE continues to engage with and track the alumni through LinkedIn and a QSURE newsletter. QSURE has its own public LinkedIn page as well as a private group limited to alumni and program leadership that is an invaluable resource for both parties (QSURE 2021). Additionally, QSURE alumni are also invited to give talks to the new cohort about educational and professional opportunities and experiences. QSURE leadership and mentors also continue to provide professional guidance to alumni on application to graduate schools, job applications, subsequent opportunities in the research field, and work-life balance strategies, especially during the COVID-19 pandemic.

3.5 Program Outcomes and Dissemination

To engage with the education community, the QSURE leadership team participated in various academic activities to bring awareness to the program and to make the program visible in the scientific community. QSURE disseminated these components in the form of invited oral and poster presentations at the International Cancer Education Conference (ICEC). QSURE also participated in the NCI-sponsored workshop at ICEC to engage with other R25 programs to exchange best practices and future plans. Members of the leadership team also served as chairs of oral presentation sessions and judges of poster presentations.

4 Discussion

We summarize the lessons learned from developing and implementing QSURE during 2018 and 2019. QSURE is focused on training undergraduate students in quantitative sciences within a nontraditional academic setting of a free-standing cancer center. Program administrators established a seamless onboarding process, including timely access to the cancer center’s computing environment. Department faculty enthusiastically participated as mentors by engaging fellows in several ongoing projects. Faculty and staff generously contributed their time to deliver lectures on quantitative topics.

The pre- and post-program evaluation through structured surveys illustrate that, overall, fellows’ perceived competency levels increased by the end of the program. Survival analysis skill had the maximum number of students (5 or 25%) reporting low perceived final competency. This could be because few undergraduate programs include advanced training in survival analysis, which is a specialized topic, and few QSURE projects involved survival analysis, thus providing limited opportunity for certain fellows to advance their understanding of this topic. Given the pivotal role of survival analysis in cancer and other areas of health science, future QSURE programs should include opportunities for fellows to gain additional exposure in this specialized topic. For example, QSURE fellows could be observers at meetings where they would accompany a faculty member (possibly other than their project mentor) to research study meetings where a research team of statisticians and clinical researchers would be planning a research question involving survival analysis, or reviewing the results of a survival analysis, or planning the specific aims of a potential grant involving survival analysis. In addition, the lecture on survival analysis could also be supplemented with hands-on activities involving survival data analysis using innovative engagement approaches such as “The Kaplan Meier Theater” (Gerds 2016). Research biostatisticians leading the statistical software training sessions could be encouraged to distribute a sample survival dataset from one of the cancer center’s studies and have the fellows work through related survival analysis as part of software training.

Fewer than 10 students reported high perceived final competency level for two skills: eight (40%) for genetics/genomics knowledge and nine (45%) for health services knowledge. This could be because the level of genetics, genomics, and health services topics used in oncology settings may not be covered at all or not covered in depth in undergraduate statistics programs. Future QSURE programs should include lectures by oncology researchers on genetics and genomics of cancer and related computational and statistical research works that are underway across the cancer center, and opportunities to spend time at the Integrative Genomics Operation Laboratory and other laboratories at the cancer center to observe research discussions surrounding genetics and genomics. Future programs should also include lectures in health services research topics by members of the cancer center’s Strategic Innovations Team where the hospital’s health services needs are assessed, and related decisions are made. Opportunities can also be explored and pursued for fellows to be observers at meetings of the Strategic Innovations Team.

A few fellows reported reduced perceived final relative to initial competency in regression modeling, epidemiology knowledge and genetics and genomics knowledge. Whether this is due to self-reflection by these fellows in recognizing that they have much more to learn or whether these fellows’ understanding indeed reduced during the course of the program is an important question that cannot be addressed at this time. Future QSURE program could incorporate confidential exit interviews for fellows reporting reduced perceived final competency in any skill level. This exit interview can be conducted by the program directors who can use insights gained from the interview to identify intervention strategies to improve program and mentoring experiences in subsequent years of the program.

While QSURE has successfully trained and mentored 20 fellows to date in cancer-related quantitative sciences, there are limitations in program assessment. The pre-post skills assessment admittedly captures self-reported perceived initial and final skill levels of fellows in various topics. As such, this is not an assessment of program effectiveness or success. As QSURE continues to train more undergraduate students, we will continue to collect information on higher education and career paths of the fellows as measures of program effectiveness and success. We will also develop and implement additional student evaluations of internship experience such as the evaluation approach used by the Georgetown College Internship program (Georgetown College 2021) and the Environmental Protection Agency’s (EPA) internship evaluation approach (Industrial Economics 2003) to assess the effectiveness of the program each year.

Finally, a dedicated mentor is crucial to the research experience of the fellow. Our faculty recognizes that their commitment and time are investments in the next generation of cancer quantitative scientists. To date, QSURE has enjoyed enthusiastic participation from faculty members as mentors. Moving forward, we will measure sustained enthusiasm, that is, critical for programmatic success by tracking mentor participation and retention rates.

5 Conclusion

The successful recruitment and training of 20 QSURE fellows demonstrate the merits of the program. In its initial two years, this program has established the core components of a successful research experience program for undergraduate students in quantitative sciences. In addition, the success of the program is reflected by the proportion of students who went on to pursue graduate degrees in a quantitative sciences field or work in academic research roles. As we continue QSURE in the coming years and improve specific components of the program based on lessons learned, we are enthusiastic and confident that this program can serve as a model that could be implemented at other cancer centers and health service organizations to successfully train the next generation of researchers in health-related quantitative sciences.

Acknowledgments

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We thank Mr. Richard Koppenaal and Ms. Shireen Lewis for their administrative support for the QSURE program, and Ms. Yessenia Werner for her assistance in compiling the database of all internship listings posted on the American Statistical Association website. We would like to thank Dr. William Trochim for helping develop the pre-post evaluation.

Supplementary material

The list of lecture series titles, student research projects and templates for student workplan and program assessments have been provided as supplementary material.

Additional information

Funding

The QSURE program is supported by the National Cancer Institute of the National Health Institutes (R25 CA214255) and by the Memorial Sloan Kettering Center Grant (CA008748).