10,472
Views
13
CrossRef citations to date
0
Altmetric
Special Topics: Statistics Education in the Health Sciences

The Importance of Statistical Competencies for Medical Research Learners

&

ABSTRACT

It is very important for medical professionals and medical researchers to be literate in statistics. However, we have found that the degree of literacy that is required should not be identical for every statistical competency or even for every learner. We first begin by describing why the development, teaching, and assessment of statistical competencies for medical professionals and medical researchers are critical tasks. We next review our three substantial efforts at developing a comprehensive list of statistical competencies that can be used as a guide for what medical research learners should know about statistics, for curricular development, and for assessment of statistical education. We then summarize the origin and the inclusion of the statistical competency items. We follow this with a description of potential uses and applications of the statistical competencies to improve targeted learning for medical research learners. Finally, we discuss implications of the statistical competencies for undergraduate statistics education.

1. Introduction

In considering statistics education at the undergraduate and graduate level, it may be informative to consider how that education might be used by students who go on to become physicians or other health scientists. In this article, our goal is to inform the readers of this journal about recent developments in the field of statistics education for medical researchers and health scientists.

It is very important for medical researchers and health science learners to be literate in biostatistics, as biostatistics is frequently used to assist in the design of medical research studies and to summarize, analyze, and report on data obtained from these studies. In recent years, results obtained from statistical analyses have been reported in the medical literature on an increasing basis (Altman et al. Citation2002; Horton and Switzer Citation2005; Strasak et al. Citation2007), and so an increased knowledge of biostatistics may be needed in order to fully understand results that are being reported and translate those results to research or practice. Additional familiarity in biostatistics is also needed in order to understand when statistical methods have been incorrectly applied or when statistical results have been incorrectly reported; it has been noted that without the benefit of statistical expertise, statistical mistakes are common in manuscripts that have been submitted to or already published in peer-reviewed journals (Harris et al. Citation2009; Mazumdar et al. Citation2010; Fernandes-Taylor et al. Citation2011).

The development, teaching, and assessment of statistical competencies for medical research are essential tasks. Competency has been defined as the ability or skill to do something successfully or efficiently (Oxford Dictionary Citation2018). Statistical competencies are used to define statistical education in terms of the topics that should be taught and more generally for designing curricula for learners. Our previous and current work in this area is oriented toward the statistical education of medical research learners. In the remainder of this article, we will review our previous efforts, describe our current efforts, summarize the origin and the inclusion of statistical competency items, and discuss possible future directions toward the need for and development, assessment, implementation of statistical competencies for medical research learners, and implications of this work for undergraduate statistics education.

2. Need for A New Assessment Tool to Assess Statistical Competencies

A set of competencies was developed for Master's degree-level training in clinical and translational science (CTS) by the Education Key Function Committee of the Clinical and Translational Science Award's (CTSA) National Consortium (CTSA Education Core Competency Workgroup 2011). This set of competencies included ten statistical competencies in the core thematic area of statistical approaches. Separate from this effort, a core competency model for the Master of Public Health (PH) degree was developed (Calhoun et al. Citation2008). This model included ten statistical competencies in the biostatistics core domain.

Graduate-level statistical competencies and existing instruments, including those mentioned above, were reviewed by Enders (Citation2011). Regarding the application to medical research learners, these instruments were intended to assess (1) physicians’ ability to read the medical literature and (2) for undergraduate statistics, the alignment with core competencies necessary for the successful use of statistics. Included in this review were widely recognized guidelines for clinical research that are inherently based on statistical concepts; the specific guidelines are the Consolidated Standards of Reporting Trials (CONSORT) document (Moher et al. Citation2010) which describes fundamental statistical methods for the design, analysis, and reporting of clinical research and lists criteria for writing manuscripts from randomized clinical trials; and the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) and Strengthening the Reporting of OBservational studies in Epidemiology (STROBE) manuscript guidelines (Des Jarlais et al. Citation2004; Vandenbroucke et al. Citation2007; von Elm et al. Citation2007) which describe criteria similar to those in the CONSORT document for observational studies.

Statistical competencies from the CTS and PH competency documents were combined in order to obtain a full set of competencies for graduate-level biostatistics. The set of statistical competencies was then compared with the set of statistical methods needed to write manuscripts that are listed in the CONSORT, TREND, and STROBE guidelines. The instrument validation included an examination of content validity and criterion validity, which itself included concurrent validity and responsiveness. A comprehensive set of 17 statistical competencies was obtained from the CTS and PH competency documents and the three sets of manuscript guidelines. Eight other statistical assessment instruments, none of which were designed specifically for a population of medical researchers, were then evaluated. The number and percent of items associated with 24 unique statistical topic areas for the various assessment instruments was then determined. Statistical topic areas included in most of these instruments included the following: “selecting assumptions and selecting an appropriate method, unadjusted methods for independent continuous and binary data, significance testing and p-values, confidence intervals, and clinical relevance versus statistical significance” (Enders Citation2011). Finally, a comparison and gap analysis of the 24 statistical topics versus the 17 statistical competency areas was presented. Additional statistical topics that were needed, none of which were included in any of the instruments that were assessed, were also discussed.

The assessment of these competencies and instruments demonstrated that there was a need for a new instrument to assess biostatistical competencies for medical researchers. It was determined that the existing instruments, which included the CTS and PH documents, the CONSORT, TREND, and STROBE guidelines, and eight other statistical assessments, did not sufficiently address the competencies required for medical research using statistics. It was also determined that the set of questions taken across these instruments did not sufficiently address these competencies. In addition, it was determined that a new assessment instrument should be developed and validated by a group of interinstitutional experts in biostatistics as used in clinical research. It was thought that such an instrument could become a definitive list of statistical competencies that are required by medical research learners.

3. Development of A New Assessment Tool to Assess Statistical Competencies

The development of the new assessment tool (Oster et al. Citation2015) began in 2012 through the national CTSA Biostatistics, Epidemiology, and Research Design (BERD) Key Function Committee (KFC). This effort, which continued through 2014, took place via annual Face-to-Face Meetings of the BERD KFC and conference calls. Many of the individuals involved in this effort were members of the BERD KFC Education Working Group and its parent committee, the Online Resources Task Force, although any member of the BERD KFC was welcome to participate.

The new assessment tool was developed around the previously published set of 17 statistical competencies (Enders Citation2011) described in the previous section. Four new competencies reflecting recent developments in statistical practice were added to this list, bringing the total number of competencies to 21. The goal of this research was to address the following question: “What depth of statistical knowledge do different CTS learners require?”

For three types of CTS learners (principal investigator, coinvestigator, informed reader of the literature), each with different backgrounds in research (no previous research experience, reader of the research literature, previous research experience), 18 experts in biostatistics, epidemiology, and research design proposed levels for the 21 statistical competencies. Each expert was asked for each topic what level of learner competency was needed (none, some, and high) given the type of CTS learner and the research background. Statistical competencies were categorized as fundamental, intermediate, or specialized based on the responses of the experts and according to the previously described criteria for grouping the competencies (Oster et al. Citation2015).

Statistical competencies that were assessed by the experts appear in under “Oster et al. (Citation2015), unless otherwise specified.” Of the 21 statistical competencies that were assessed, nine (43%) were classified as fundamental, six (29%) were classified as intermediate, and six (29%) were classified as specialized. The rank assigned to each of these competencies is based on a subsequent evaluation of these and additional statistical competencies (Enders et al. Citation2017); this evaluation is described later in Section 4.

Table 1. Original and revised wording for statistical competencies with results of surveys from Oster et al. (Citation2015) and Enders et al. (Citation2017). Modified wording is shown in bold text.

Based on the criteria for grouping the competencies, which were then based on the responses about what level of learner competency was needed, it became clear to us that CTS learners who intend to become independent principal investigators require more specialized training, while those intending to become informed consumers of the medical literature require more fundamental education. In addition, we noticed that for most competencies, less training was proposed for those with a higher-level background in research. This led us to conclude that not all of the statistical competencies are appropriate for the goals and perspectives of all learners. We also noted that rather that providing a one-size-fits-all curriculum for statistical learning, institutions should account for the multifaceted needs of different types of learners. Our advice to readers was that tailoring education to baseline knowledge, learner background, and future goals increases learning potential while minimizing classroom time.

We also proposed further changes to our list of statistical competencies. After we had already obtained our results, we noted that some of the fundamental and intermediate competencies had verbs associated with higher cognitive function. After a detailed examination of these competencies, we concluded that some of them would be better phrased using a different verb, which would imply that the goal of the competency would then be different, i.e., to reduce the required competency level for the specific type of learner or to increase the required competency level for a specific type of learner. We then described our specific proposed changes to the wording of several of the competencies (Oster et al. Citation2015).

4. Refinement of the Tool to Assess Statistical Competencies

After a presentation on the previously described statistical competencies to members of the Association for Clinical and Translational Statisticians (ACTStat) and a follow-up discussion with several members of this organization, it became clear that the terminology used to describe the statistical competencies was the basis of the consideration of ACTStat discussants as fundamental to the needs of a medical research learner. These discussants suggested that the competencies should be phrased in a manner that would reflect the fundamental needs of medical research learners. They also suggested that some specific competency areas may be required for some learners. Accounting for this and for the changes that we already proposed to our list of statistical competencies (Oster et al. Citation2015), a working group was formed in order to revise the competencies so that they would reflect the role of the medical research learner within the research team.

We then evaluated the refined set of statistical competencies for medical research learners (Enders et al. Citation2017). This revised set now included a total of 24 statistical competencies. The goal of the study was to determine the degree to which each statistical competency is fundamental, i.e., necessary for all learners, or specialized, i.e., necessary for only some learners. Respondents were all doctorally trained biostatisticians and epidemiologists who had taught statistics to medical research learners during the previous 5 years (from 2010 to 2015). These respondents rated the 24 statistical competencies on a semiquantitative five-point Likert scale anchored by “fundamental” and “specialized.” There were 112 responses.

Statistical competencies that were evaluated appear in under “Enders et al. (Citation2017).” Of the 24 statistical competencies that were assessed, 19 (79%) were classified as fundamental and five (21%) were classified as not fundamental. The rank assigned to each of these competencies is based on the percentage of respondents who stated that the competency is fundamental, from highest percentage to lowest percentage. The wording that was modified from the previous assessment (Oster et al. Citation2015) to this evaluation (Enders et al. Citation2017) is displayed in bold text.

This research identified a set of 19 statistical competencies that are needed by all medical researchers and five statistical competencies that are needed by only some medical researchers. The different groups of educators from many outstanding academic institutions that were included in the survey all offered a similar perspective, indicating that these results may be viewed as comprehensive for medical research learners. Medical research learners who master the set of fundamental competencies and who engage in team science should have the appropriate statistical skills to be successful in collaborative research. We suggest that these competencies be considered when designing statistical curricula for medical research learners. They should also be used by statistical educators when deciding on which topics will be taught in graduate programs and medical courses where learners need to read and understand the medical research literature.

5. The Origin and the Inclusion of Statistical Competency Items

The origin of the statistical competency items and the inclusion of these items into the publications that are cited appear in . The competencies identified in Enders (Citation2011) from the CONSORT, STROBE, and TREND guidelines appear in gray. Regarding the other competencies, those included in Calhoun et al. (Citation2008) appear in yellow; those included in the CTSA Education Core Competency Workgroup report (2011) appear in red; those included in the publication guidelines (Enders Citation2011) appear in purple; those included in the first survey (Oster et al. Citation2015) appear in blue; and those included in the second survey (Enders et al. Citation2017) appear in green. The rank of the competency is as described in the previous section.

Figure 1. Origin and inclusion of statistical competency items. In gray are competencies identified in Enders (Citation2011) from the publication guidelines (CONSORT, TREND, and STROBE). In color are competencies included for each subsequent publication. For Enders et al. (Citation2017), competencies in light green are those identified as Not Fundamental.

Figure 1. Origin and inclusion of statistical competency items. In gray are competencies identified in Enders (Citation2011) from the publication guidelines (CONSORT, TREND, and STROBE). In color are competencies included for each subsequent publication. For Enders et al. (Citation2017), competencies in light green are those identified as Not Fundamental.

In Enders et al. (Citation2017), the 19 statistical competencies that were identified as fundamental, i.e., those that are needed by all medical research learners, appear in dark green, and the five statistical competencies that were identified as not fundamental, i.e., those that are needed by only some medical research learners, appear in light green. This is also the only publication that includes an assessment of all of the 24 statistical competencies.

6. Potential Uses and Applications of the Statistical Competencies Within Statistics for the Health Sciences

Once we have an understanding of both which statistical competencies are taught to medical research learners and which are not well addressed, there may be opportunities to develop educational modules to fill the gaps. Strong content is resource-intensive to develop and implement; in the era of online learning, such resources need not be spent at each institution. In order to facilitate this, content would need to be modularized and available on an on-demand basis in order to fit in with scheduling at each site. Content would ideally also be available on a tiered basis to provide different levels of intensity to different learners.

Our work has shown that learners who will be principal investigators require a higher level of understanding than those who will be coinvestigators. Institutions should consider assessing learning options to ensure that content is available at different levels, rather than providing a one size fits all approach. In order to do this without expanding the number of courses, learners could access additional course content within existing courses. Such an approach could tie course content to the intended level of each learner determined during the program application phase.

Another way to utilize our work would be to develop assessment tools to assess the statistical competency of learners at matriculation and prior to program completion. The statistical competencies are specific and can be used to develop such assessments that would then be validated prior to use. In practice, some learners have knowledge at the start of their training program and might use such assessments to test out of courses or portions of content, which would constitute a valuable asset in programs in which the majority of learners are practicing clinicians. Assessments could also be used to ensure learner-appropriate competency prior to program completion.

7. Implications for Undergraduate Statistics Education

Undergraduate statistics students who are in courses designed for nonmajors may become medical researchers or other health scientists. For this group, the competency work above is begun during their undergraduate training. The revised Guidelines for Assessment and Instruction in Statistics Education College Report (GAISE) (GAISE College Report ASA Revision Committee 2016) is well aligned with the competency work above and now presents an even greater emphasis on providing students with experience with collecting real data. This is well aligned with one of our most fundamental competencies, the need to “assess sources of bias and variation.” While we believe that variation is already well addressed with the GAISE, we believe bias may be underaddressed. Within both flavors of undergraduate courses (statistical literacy and statistical methods), instructors could build upon the GAISE recommendations by introducing the concept of bias especially when students are working with real data. Within the revised college-level GAISE report, bias is included within Recommendation 2 (Focus on conceptual understanding); however, this concept is lacking in Recommendation 3 (Integrate real data with a context and a purpose). We hope that undergraduate instructors will include the concept of bias whenever possible because it is so integral to future understanding of statistical concepts.

8. Conclusion

Our work on this topic has progressed from identifying a set of statistical competencies for learners in academic medicine to developing and validating the competencies themselves. Our plan is now to put the competencies to use by assessing the degree to which each is taught in required statistical coursework for learners of different types. We believe this work will help strengthen statistical training to medical research learners and can help enhance undergraduate statistics education.

Funding

This work was funded in part by the National Institutes of Health (NIH) Clinical and Translational Science Award Program in collaboration with the University of Alabama at Birmingham (UL1 TR001417) and the Mayo Clinic (UL1 TR000135). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

References

  • Altman, D. G., Goodman, S. N., and Schroter, S. (2002), “How Statistical Expertise is Used in Medical Research,” Journal of the American Medical Association, 287, 21, 2817–2820.
  • Calhoun, J. G., Ramiah, K., Weist, E. M., and Shortell, S. M. (2008), “Development of a Core Competency Model for the Master of Public Health Degree,” American Journal of Public Health, 98, 9, 1598–1607.
  • CTSA Education Core Competency Workgroup. (2011), Core Competencies in Clinical and Translational Science for Master's Candidates. Revised 2011 [Internet]. [cited May 28, 2018] Available at https://clic-ctsa.org/sites/default/files/CTSA_Core_Competencies_final_2011.pdf.
  • Des Jarlais, D., Lyles, C., Crepaz, N., and Group, T. (2004), “Improving the Reporting Quality of Nonrandomized Evaluations of Behavioral and Public Health Interventions: The TREND Statement,” American Journal of Public Health, 94, 3, 361–366.
  • Enders, F. T. (2011), “Evaluating Mastery of Biostatistics for Medical Researchers: Need for a New Assessment Tool,” Clinical and Translational Science, 4, 6, 448–454.
  • Enders, F. T., Lindsell, C. J., Welty, L. J., Benn, E. K. T., Perkins, S. M., Mayo, M. S., Rahbar, M. H., Kidwell, K. M., Thurston, S. W., Spratt, H., Grambow, S. C., Larson, J., Carter, R. E., Pollock, B. H., and Oster, R. A. (2017), “Statistical Competencies for Medical Research Learners: What is Fundamental?” Journal of Clinical and Translational Science, 1, 3, 146–152.
  • Fernandes-Taylor, S., Hyun, J. K., Reeder, R. N., and Harris, A. H. S. (2011), “Common Statistical and Research Design Problem in Manuscripts Submitted to High-Impact Journals,” BMC Research Notes, 4, 304.
  • GAISE College Report ASA Revision Committee. (2016), “Guidelines for Assessment and Instruction in Statistics Education College Report 2016,” Available at http://www.amstat.org/education/gaise.
  • Harris, A. H. S., Reeder, R., and Hyun, J. K. (2009), “Common Statistical and Research Design Problem in Manuscripts Submitted to High-Impact Psychiatry Journals: What Editors and Reviewers Want Authors to Know,” Journal of Psychiatric Research, J43, 15, 1231–1234.
  • Horton, N. J., and Switzer, S. S. (2005), “Statistical Methods in the Journal,” New England Journal of Medicine, 353, 18, 1977–1979.
  • Mazumdar, M., Banerjee, S., and Van Epps, H. L. (2010), “Improved Reporting of Statistical Design and Analysis: Guidelines, Education, and Editorial Policies,” Methods of Molecular Biology, 620, 563–598.
  • Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P., Devereaux, P. J., Elbourne, D., Egger, M., and Altman, D. G. (2010), “CONSORT 2010 Explanation and Elaboration: Updated Guidelines for Reporting Parallel Group Randomised Trials,” BMJ, 340, c869, 1–28.
  • Oster, R. A., Lindsell, C. J., Welty, L. J., Mazumdar, M., Thurston, S. W., Rahbar, M. H., Carter, R. E., Pollock, B. H., Cucchiara, A. J., Kopras, E. J., Jovanovic, B. D., and Enders, F. T. (2015), “Assessing Statistical Competencies in Clinical and Translational Science Education: One Size Does not Fit All,” Clinical and Translational Science, 8, 1, 32–42.
  • Oxford Dictionary [Internet]. Available at http://www.oxforddictionaries.com/us/definition/american_english/competence. Last accessed May 28, 2018.
  • Strasak, A. M., Zarnan, Q., Marinell, G., Pfieffer, K. P., and Ulmer, H. (2007), “The Use of Statistics in Medical Research: A Comparison of The New England Journal of Medicine and Nature Medicine,” The American Statistician, 61, 1, 47–55.
  • Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gotzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., and Egger, M. (2007), “Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration,” PLoS Medicine, 4, 10, 1628–1654.
  • von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gotzsche, P. C., and Vandenbroucke, J. P. (2007), “The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies,” Bulletin of the World Health Organization, 85, 11, 867–872.