Good Data Science Practice: Moving Toward a Code of Practice for Drug Development

References

• Akacha, M., Bretz, F., and Ruberg, S. (2017), “Estimands in Clinical Trials – Broadening the Perspective,” Statistics in Medicine, 36, 5–19. DOI: 10.1002/sim.7033.
   PubMed Web of Science ®Google Scholar
• Altman, D. G. (1994), “The Scandal of Poor Medical Research,” BMJ, 308, 283–384. DOI: 10.1136/bmj.308.6924.283.
   PubMed Web of Science ®Google Scholar
• Altman, D. G., and Bland, J. M. (1983), “Measurement in Medicine: The Analysis of Method Comparison Studies,” The Statistician, 32, 307–317. DOI: 10.2307/2987937.
   Google Scholar
• Amrhein, V., Greenland, S., and McShane, B. (2019), “Scientists Rise up against Statistical Significance.” Nature, 567, 305–307. DOI: 10.1038/d41586-019-00857-9.
   PubMed Web of Science ®Google Scholar
• Arnold, B., Bowler, L., Gibson, S., Herterich, P., Higman, R., Krystalli, A., Morley, A., O’Reilly, M., and Whitaker, K. (2019), The Turing Way: A Handbook for Reproducible Data Science (Version v0.0.4). Zenodo, March 25. DOI: 10.5281/zenodo.3233986.
   Google Scholar
• Arnold, K. F., Davies, V., de Kamps, M., Tennant, P. W. G., Mbotwa, J., and Gilthorpe, M. S. (2020), “Reflections on Modern Methods: Generalized Linear Models for Prognosis and Intervention-Theory, Practice and Implications for Machine Learning,” International Journal of Epidemiology, 49, 2074–2082. DOI: 10.1093/ije/dyaa049.
   Web of Science ®Google Scholar
• Baker, M. (2016), “1,500 Scientists Lift the Lid on Reproducibility,” Nature, 533, 452–454. DOI: 10.1038/533452a.
   PubMed Web of Science ®Google Scholar
• Blei, D. M., and Smyth, P. (2017), “Science and Data Science,” Proceedings of the National Academy of Sciences of the United States of America, 114, 8689–8692. DOI: 10.1073/pnas.1702076114.
   PubMed Web of Science ®Google Scholar
• Boulesteix, A.-L., Groenwold, R. H., Abrahamowicz, M., Binder, H., Briel, M., Hornung, R., Morris, T. P., Rahnenführer, J., and Sauerbrei, W. (2020), “Introduction to Statistical Simulations in Health Research,” BMJ Open, 10, e039921. DOI: 10.1136/bmjopen-2020-039921.
   PubMed Web of Science ®Google Scholar
• Bourne, P. E. (2021), “Is ‘Bioinformatics’ Dead?,” PloS Biology, 19, e3001165. DOI: 10.1371/journal.pbio.3001165.
   PubMed Web of Science ®Google Scholar
• Cao, L. (2018), “Data Science Thinking,” in Data Science Thinking: The Next Scientific, Technological and Economic Revolution, ed. L. Cao, pp. 59–90, Cham: Springer.
   Google Scholar
• Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., and Elhadad, N. (2015), “Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-Day Readmission,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721–30. KDD ’15, New York: Association for Computing Machinery. DOI: 10.1145/2783258.2788613.
   Google Scholar
• Chambers, J. M. (1993), “Greater or Lesser Statistics: A Choice for Future Research,” Statistics and Computing, 3, 182–184. DOI: 10.1007/BF00141776.
   Web of Science ®Google Scholar
• Chatfield, C. (1991), “Avoiding Statistical Pitfalls,” Schweizerische Monatsschrift Fur Zahnheilkunde = Revue Mensuelle Suisse D’odonto-Stomatologie/SSO, 6, 240–252.
   Google Scholar
• Chatfield, C. (2002), “Confessions of a Pragmatic Statistician,” Journal of the Royal Statistical Society, Series D, 51, 1–20. DOI: 10.1111/1467-9884.00294.
   Google Scholar
• Cleveland, W. S. (2001), “Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics,” International Statistical Review = Revue Internationale de Statistique, 69, 21–26.
   Web of Science ®Google Scholar
• Cox, D. R. (2017), “Statistical Science: A Grammar for Research,” European Journal of Epidemiology, 32, 465–471. DOI: 10.1007/s10654-017-0288-1.
   PubMed Web of Science ®Google Scholar
• Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., and Moons, K. G. M. (2017), “A Guide to Systematic Review and Meta-Analysis of Prediction Model Performance,” BMJ (Clinical Research ed.), 356, i6460.
   PubMedGoogle Scholar
• Devezer, B., Navarro, D. J., Vandekerckhove, J., and Ozge Buzbas, E. (2021), “The Case for Formal Methodology in Scientific Reform,” Royal Society Open Science, 8, 200805. DOI: 10.1098/rsos.200805.
   PubMed Web of Science ®Google Scholar
• Diggle, P. J. (2015), “Statistics: A Data Science for the 21st Century,” Journal of the Royal Statistical Society, Series A 178, 793–813. DOI: 10.1111/rssa.12132.
   Web of Science ®Google Scholar
• “Do No Harm: An Ethical Data Life Cycle.” n.d. Accessed April 14, 2021, available at https://www.aaaspolicyfellowships.org/blog/do-no-harm-ethical-data-life-cycle.
   Google Scholar
• Donoho, D. (2017), “50 Years of Data Science,” Journal of Computational and Graphical Statistics, 26, 745–766. DOI: 10.1080/10618600.2017.1384734.
   Web of Science ®Google Scholar
• “E9 (R1) Step 5 Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials,” n.d. Available at https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e9-r1-addendum-estimands-sensitivity-analysis-clinical-trials-guideline-statistical-principles_en.pdf.
   Google Scholar
• Efron, B., and Hastie, T. (2016), Computer Age Statistical Inference, Cambridge: Cambridge University Press.
   Google Scholar
• EMBL. n.d. “[No Title].” Accessed June 2, 2021. Available at https://www.embl.org/.
   Google Scholar
• Ferrero, E., Brachat, S., Jenkins, J. L., Marc, P., Skewes-Cox, P., Altshuler, R. C., Gubser Keller, C., Kauffmann, A., Sassaman, E. K., Laramie, J. M., Schoeberl, B., Borowsky, M. L., and Stiefl, N. (2020), “Ten Simple Rules to Power Drug Discovery with Data Science,” PloS Computational Biology, 16, e1008126. DOI: 10.1371/journal.pcbi.1008126.
   PubMedGoogle Scholar
• Fineberg, H., Stodden, V., and Meng, X.-L. (2020), “Highlights of the US National Academies Report on ‘Reproducibility and Replicability in Science”’ Harvard Data Science Review, 2, 1–11. DOI: 10.1162/99608f92.cb310198.
   Google Scholar
• Fleming, T. R., DeMets, D. L., Roe, M. T., Wittes, J., Calis, K. A., Vora, A. N., Meisel, A., Bain, R. P., Konstam, M. A., Pencina, M. J., Gordon, D. J., Mahaffey, K. W., Hennekens, C. H., Neaton, J. D., Pearson, G. D., Andersson, T. L., Pfeffer, M. A., and Ellenberg, S. S. (2017), “Data Monitoring Committees: Promoting Best Practices to Address Emerging Challenges,” Clinical Trials (London, England), 14, 115–123. DOI: 10.1177/1740774516688915.
   PubMed Web of Science ®Google Scholar
• Gawande, A. (2009), The Checklist Manifesto: How to Get Things Right, New York: Metropolitan Books.
   Google Scholar
• Gelman, A., and Loken, E. (2014), “The Statistical Crisis in Science: Data-Dependent Analysis–a ‘Garden of Forking Paths’–Explains Why Many Statistically Significant Comparisons Don’t Hold Up,” American Scientist, 102, 460–466.
   Web of Science ®Google Scholar
• Gelman, A., and Vehtari, A. (2020), “What Are the Most Important Statistical Ideas of the Past 50 Years?” arXiv [stat.ME]. arXiv. http://arxiv.org/abs/2012.00174.
   Google Scholar
• Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P.-C., and Modrák, M. (2020) “Bayesian Workflow.” arXiv [stat.ME]. arXiv. http://arxiv.org/abs/2011.01808.
   Google Scholar
• Gibson, E. W. (2019), “Leadership in Statistics: Increasing Our Value and Visibility,” The American Statistician, 73, 109–116. DOI: 10.1080/00031305.2017.1336484.
   Web of Science ®Google Scholar
• Glass, D. J. (2010), “A Critique of the Hypothesis, and a Defense of the Question, as a Framework for Experimentation,” Clinical Chemistry, 56, 1080–1085. DOI: 10.1373/clinchem.2010.144477.
   PubMed Web of Science ®Google Scholar
• Greenhouse, J. B., and Seltman, H. J. (2018), “On Teaching Statistical Practice: From Novice to Expert,” The American Statistician, 72, 147–154. DOI: 10.1080/00031305.2016.1270230.
   Web of Science ®Google Scholar
• Hand, D. J. (1994), “Deconstructing Statistical Questions,” Journal of the Royal Statistical Society, Series A, 157, 317–356. DOI: 10.2307/2983526.
   Web of Science ®Google Scholar
• Hand, D. J. (2006), “Classifier Technology and the Illusion of Progress,” Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 21, 1–14.
   PubMed Web of Science ®Google Scholar
• Hernán, M. A., and Robins, J. M. (2016), “Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available,” American Journal of Epidemiology, 183, 758–764. DOI: 10.1093/aje/kwv254.
   PubMed Web of Science ®Google Scholar
• Hernan, M. A., and Robins, J. M. (2019), Causal Inference, Boca. Raton, FL: Taylor & Francis.
   Google Scholar
• Hernán, M. A., Hsu, J., and Healy, B. (2019), “A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks,” Chance, 32, 42–49. DOI: 10.1080/09332480.2019.1579578.
   Google Scholar
• Huebner, M., Vach, W., and Le Cessie, S. (2016), “A Systematic Approach to Initial Data Analysis Is Good Research Practice,” The Journal of Thoracic and Cardiovascular Surgery, 151, 25–27. DOI: 10.1016/j.jtcvs.2015.09.085.
   PubMed Web of Science ®Google Scholar
• “ICH Official Web Site: ICH.” n.d. Accessed June 3, 2021. Available at https://www.ich.org/.
   Google Scholar
• International Conference on Harmonisation (ICH). (2018a), “ICH E8 General Considerations for Clinical Studies,” September 17, 2018. https://www.ema.europa.eu/en/ich-e8-general-considerations-clinical-studies.
   Google Scholar
• International Conference on Harmonisation (ICH). “ICH E9 Statistical Principles for Clinical Trials.” September 17, 2018, available at https://www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials.
   Google Scholar
• Keller, S. A., Shipp, S. S., Schroeder, A. D., and Korkmaz, G. (2020), “Doing Data Science: A Framework and Case Study,” 2.1.
   Google Scholar
• Kenett, R. S., and Shmueli, G. (2015), “Clarifying the Terminology That Describes Scientific Reproducibility,” Nature Methods, 12, 699. DOI: 10.1038/nmeth.3489.
   PubMed Web of Science ®Google Scholar
• Krieger, N., and Davey Smith, G. (2016), “The Tale Wagged by the DAG: Broadening the Scope of Causal Inference and Explanation for Epidemiology,” International Journal of Epidemiology, 45, 1787–1808. DOI: 10.1093/ije/dyw114.
   PubMed Web of Science ®Google Scholar
• Kuemmel, C., Yang, Y., Zhang, X., Florian, J., Zhu, H., Tegenge, M., Huang, S-M., Wang, Y., Morrison, T., and Zineh, I. (2020), “Consideration of a Credibility Assessment Framework in Model-Informed Drug Development: Potential Application to Physiologically-Based Pharmacokinetic Modeling and Simulation,” CPT: Pharmacometrics & Systems Pharmacology, 9, 21–28.
   PubMedGoogle Scholar
• Leek, J. T., and Peng, R. D. (2015), “Statistics: P Values Are Just the Tip of the Iceberg,” Nature 520, 612. DOI: 10.1038/520612a.
   PubMed Web of Science ®Google Scholar
• Liu, G., Lu, D., and Lu, J. (2021), “Pharm-AutoML: An Open-Source, End-to-End Automated Machine Learning Package for Clinical Outcome Prediction,” CPT: Pharmacometrics & Systems Pharmacology, 10, 478–488.
   PubMedGoogle Scholar
• Mallows, C. (1998), “The Zeroth Problem,” The American Statistician, 52, 1–9.
   Web of Science ®Google Scholar
• McElreath, R. (2020), Statistical Rethinking: A Bayesian Course with Examples in R and Stan, Boca Raton, FL: CRC Press.
   Google Scholar
• Meyer, E. L., Mesenbrink, P., Dunger-Baldauf, C., Fülle, H.-J., Glimm, E., Li, Y., Posch, M., and König, F. (2020), “The Evolution of Master Protocol Clinical Trial Designs: A Systematic Literature Review,” Clinical Therapeutics, 42, 1330–1360. DOI: 10.1016/j.clinthera.2020.05.010.
   PubMed Web of Science ®Google Scholar
• Morris, T. P., White, I. R., and Crowther, M. J. (2019), “Using Simulation Studies to Evaluate Statistical Methods,” Statistics in Medicine, 38, 2074–2102. DOI: 10.1002/sim.8086.
   PubMed Web of Science ®Google Scholar
• Mukherjee, S., and Richardson, S. (2020), “The Future of Data Science,” Harvard Data Science Review, 2, 1–18.
   Google Scholar
• Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Du Sert, N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., and Ioannidis, J. P. A. (2017), “A Manifesto for Reproducible Science,” Nature Human Behaviour, 1, 0021. DOI: 10.1038/s41562-016-0021.
   PubMed Web of Science ®Google Scholar
• Neff, G., Tanweer, A., Fiore-Gartland, B., and Osburn, L. (2017), “Critique and Contribute: A Practice-Based Framework for Improving Critical Data Studies and Data Science,” Big Data, 5, 85–97. DOI: 10.1089/big.2016.0050.
   PubMed Web of Science ®Google Scholar
• Nelder, J. A. (1986), “Statistics, Science and Technology,” Journal of the Royal Statistical Society, Series A, 149, 109–121. DOI: 10.2307/2981525.
   Web of Science ®Google Scholar
• Nosek, B. n.d. “Strategy for Culture Change.” Accessed April 6, 2021, available at https://www.cos.io/blog/strategy-for-culture-change.
   Google Scholar
• “OHDSI – Observational Health Data Sciences and Informatics.” n.d. Accessed June 2, 2021. https://ohdsi.org/.
   Google Scholar
• “Open Science – OECD.” n.d. Accessed June 2, 2021, available at https://www.oecd.org/sti/inno/open-science.htm.
   Google Scholar
• Pajouheshnia, R., Groenwold, R. H. H., Peelen, L. M., Reitsma, J. B., and Moons, K. G. M. (2019), “When and How to Use Data from Randomised Trials to Develop or Validate Prognostic Models,” BMJ (Clinical Research ed.), 365, l2154.
   PubMedGoogle Scholar
• Parker, H. (2017), “Opinionated Analysis Development.” e3210v1. PeerJ Preprints. Available at DOI: 10.7287/peerj.preprints.3210v1.
   Google Scholar
• Picard, R. R., and Cook, R. D. (1984), “Cross-Validation of Regression Models,” Journal of the American Statistical Association 79, 575–583. DOI: 10.1080/01621459.1984.10478083.
   Web of Science ®Google Scholar
• “PRISMA.” n.d. Accessed December 21, 2020, available at http://www.prisma-statement.org/.
   Google Scholar
• Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., Sundberg, P., Yee, H., Zhang, K., Zhang, Y., Flores, G., Duggan, G. E., Irvine, J., Le, Q., Litsch, K., Mossin, A., Tansuwan, J., Wang, D., Wexler, J., Wilson, J., Ludwig, D., Volchenboum, S. L., Chou, K., Pearson, M., Madabushi, S., Shah, N. H., Butte, A. J., Howell, M. D., Cui, C., Corrado, G. S., and Dean, J. (2018), “Scalable and Accurate Deep Learning with Electronic Health Records,” NPJ Digital Medicine, 1, 18. DOI: 10.1038/s41746-018-0029-1.
   PubMed Web of Science ®Google Scholar
• “Rss-Report-Opps-and-Ethics-of-Big-Data-Feb-2016.pdf.” n.d.
   Google Scholar
• Rubinstein, A., and Chor, B. (2014), “Computational Thinking in Life Science Education,” PloS Computational Biology, 10, e1003897. DOI: 10.1371/journal.pcbi.1003897.
   PubMed Web of Science ®Google Scholar
• Rudin, C., J. and Radin, (2019), “Why Are We Using Black Box Models in AI When We Don’t Need to? A Lesson from an Explainable AI Competition.” 1.2 1 (2).
   Google Scholar
• Saltelli, A. (2020), “Ethics of Quantification or Quantification of Ethics?” Futures, 116, 102509. DOI: 10.1016/j.futures.2019.102509.
   Web of Science ®Google Scholar
• Schuemie, M. J., Ryan, P. B., Pratt, N., Chen, R., You, S. C., Krumholz, H. M., Madigan, D., Hripcsak, G., and Suchard, M. A. (2020), “Principles of Large-Scale Evidence Generation and Evaluation across a Network of Databases (LEGEND),” Journal of the American Medical Informatics Association: JAMIA, 27, 1331–1337. DOI: 10.1093/jamia/ocaa103.
   PubMed Web of Science ®Google Scholar
• Senge, P. M. (2006), The Fifth Discipline: The Art and Practice of the Learning Organization, New York: Currency.
   Google Scholar
• Sheiner, L. B. (1997), “Learning versus Confirming in Clinical Drug Development.” Clinical Pharmacology and Therapeutics, 61, 275291. DOI: 10.1016/S0009-9236(97)90160-0.
   PubMed Web of Science ®Google Scholar
• Shmueli, G. (2010), “To Explain or to Predict?” Statistical Science, 25, 289–310. DOI: 10.1214/10-STS330.
   Web of Science ®Google Scholar
• Stan Development Team. n.d. “Stan User’s Guide.” Accessed June 2, 2021. Available at https://mc-stan.org/docs/2_26/stan-users-guide/index.html.
   Google Scholar
• Stark, P. B. (2018), “Before Reproducibility Must Come Preproducibility,” Nature, 557, 613. DOI: 10.1038/d41586-018-05256-0.
   PubMed Web of Science ®Google Scholar
• Stark, P. B., and Saltelli, A. (2018), “Cargo-Cult Statistics and Scientific Crisis,” Significance (Oxford, England), 15, 40–43. DOI: 10.1111/j.1740-9713.2018.01174.x.
   Google Scholar
• Steuer, D. (2020), “Time for Data Science to Professionalise,” Significance (Oxford, England), 17, 44–45. DOI: 10.1111/1740-9713.01430.
   Google Scholar
• Stodden, V. (2020), “The Data Science Life Cycle: A Disciplined Approach to Advancing Data Science as a Science,” Communications of the ACM, 63, 58–66. DOI: 10.1145/3360646.
   Web of Science ®Google Scholar
• Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., Liu, B., Matthews, P., Ong, G., Pell, J., Silman, A., Young, A., Sprosen, T., Peakman, T., and Collins, R. (2015), “UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age,” PloS Medicine, 12, e1001779. DOI: 10.1371/journal.pmed.1001779.
   PubMed Web of Science ®Google Scholar
• Tansley, S., and Tolle, K. M. (2009), Jim Gray on eScience: A Transformed Scientific Method, available at http://www.academia.edu/download/53717445/Fourth_Paradigm.pdf#page=19.
   Google Scholar
• “The EQUATOR Network.” n.d. Accessed December 21, 2020, available at https://www.equator-network.org/.
   Google Scholar
• The Royal Statistical Society (RSS) and the Institute and Faculty of Actuaries (IFoA). (2019), “A Guide for Ethical Data Science: A Collaboration between the Royal Statistical Society (RSS) and the Institute and Faculty of Actuaries (IFoA),” available at https://rss.org.uk/RSS/media/News-and-publications/Publications/Reports%20and%20guides/A-Guide-for-Ethical-Data-Science-Final-Oct-2019.pdf.
   Google Scholar
• Tong, C. (2019), “Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science,” The American Statistician, 73, 246–261. DOI: 10.1080/00031305.2018.1518264.
   Web of Science ®Google Scholar
• Tractenberg, R. E. (2020a), “Concordance of Professional Ethical Practice Standards for the Domain of Data Science: A White Paper.” DOI: 10.31235/osf.io/p7rj2.
   Google Scholar
• Tractenberg, R. E. (2020b), “Ten Simple Rules for Integrating Ethics into Statistics and Data Science Instruction.” DOI: 10.31235/osf.io/z9uej.
   Google Scholar
• Tukey, J. W. (1962), “The Future of Data Analysis,” The Annals of Mathematical Statistics, 33, 1–67. DOI: 10.1214/aoms/1177704711.
   Google Scholar
• Tukey, J. W. (1977), Exploratory Data Analysis. (Vol. 2). Reading, MA.
   Google Scholar
• Tukey, J. W. (1986), “Sunset Salvo,” The American Statistician, 40, 72–76.
   Web of Science ®Google Scholar
• Turner, R. M., Spiegelhalter, D. J., Smith, G. C. S., and Thompson, S. G. (2009), “Bias Modelling in Evidence Synthesis,” Journal of the Royal Statistical Society: Series A (Statistics in Society), 172, 21–47. DOI: 10.1111/j.1467-985X.2008.00547.x.
   PubMed Web of Science ®Google Scholar
• “UK Biobank – UK Biobank.” n.d. Accessed June 2 2021, available at https://www.ukbiobank.ac.uk/.
   Google Scholar
• Van Calster, B., Wynants, L., Timmerman, D., Steyerberg, E. W., and Collins, G. S. (2019), “Predictive Analytics in Health Care: How Can We Know It Works?” Journal of the American Medical Informatics Association: JAMIA, 26, 1651–1654. DOI: 10.1093/jamia/ocz130.
   PubMed Web of Science ®Google Scholar
• Vickers, A. (2005), “Interpreting Data from Randomized Trials: The Scandinavian Prostatectomy Study Illustrates Two Common Errors,” Nature Clinical Practice. Urology, 2, 404–405. DOI: 10.1038/ncpuro0294.
   PubMedGoogle Scholar
• Weber, L. M., Saelens, W., Cannoodt, R., Soneson, C., Hapfelmeier, A., Gardner, P. P., Boulesteix, A.-L., Saeys, Y., and Robinson, M. D. (2019), “Essential Guidelines for Computational Method Benchmarking,” Genome Biology, 20, 125. DOI: 10.1186/s13059-019-1738-8.
   PubMed Web of Science ®Google Scholar
• Weihs, C., and Ickstadt, K. (2018), “Data Science: The Impact of Statistics,” International Journal of Data Science and Analytics, 6, 189–194. DOI: 10.1007/s41060-018-0102-5.
   Google Scholar
• Welton, N. J., Sutton, A. J., Cooper, N., Abrams, K. R., and Ades, A. E. (2012), Evidence Synthesis for Decision Making in Healthcare (Vol. 132), Chichester: Wiley.
   Google Scholar
• Wickham, H. (2014), “Tidy Data,” Journal of Statistical Software, 59, 1–23. DOI: 10.18637/jss.v059.i10.
   PubMed Web of Science ®Google Scholar
• Wild, C. J., and Pfannkuch, M. (1999), “Statistical Thinking in Empirical Enquiry,” International Statistical Review = Revue Internationale de Statistique, 67, 223–248.
   Web of Science ®Google Scholar
• Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., Gonzalez-Beltran, A., Gray, A. J. G., Groth, P., Goble, C., Grethe, J. S., Heringa, J., ’t Hoen, P. A. C., Hooft, R., Kuhn, T., Kok, R., Kok, J., Lusher, S. J., Martone, M. E., Mons, A., Packer, A. L., Persson, B., Rocca-Serra, P., Roos, M., van Schaik, R., Sansone, S.-A., Schultes, E., Sengstag, T., Slater, T., Strawn, G., Swertz, M. A., Thompson, M., van der Lei, J., van Mulligen, E., Velterop, J., Waagmeester, A., Wittenburg, P., Wolstencroft, K., Zhao, J., and Mons, B. (2016), “The FAIR Guiding Principles for Scientific Data Management and Stewardship,” Scientific Data, 3, 160018. DOI: 10.1038/sdata.2016.18.
   PubMed Web of Science ®Google Scholar
• Willis, C., and Stodden, V. (2020), “Trust but Verify: How to Leverage Policies, Workflows, and Infrastructure to Ensure Computational Reproducibility in Publication,” Harvard Data Science Review, 2, 1–60. DOI: 10.1162/99608f92.25982dcf.
   Google Scholar
• Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., and Teal, T. K. (2017), “Good Enough Practices in Scientific Computing,” PloS Computational Biology, 13, e1005510. DOI: 10.1371/journal.pcbi.1005510.
   PubMed Web of Science ®Google Scholar
• Wing, J. M. (2006), “Computational Thinking,” Communications of the ACM, 49, 33–35. DOI: 10.1145/1118178.1118215.
   Web of Science ®Google Scholar
• Wing, J. M. (2019), “The Data Life Cycle.” Issue 1, June. DOI: 10.1162/99608f92.e26845b4.
   Google Scholar
• Wing, J. M., D. and Banks, (2019), “Highlights of the Inaugural Data Science Leadership Summit.” 1.2, November, available at DOI: 10.1162/99608f92.e45fcb79.
   Google Scholar
• Winkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., and Haenssle, H. A. (2019), “Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition,” JAMA Dermatology, 155, 1135–1141. DOI: 10.1001/jamadermatol.2019.1735.
   PubMed Web of Science ®Google Scholar
• Zame, W. R., Bica, I., Shen, C., Curth, A., Lee, H.-S., Bailey, S., Weatherall, J., Wright, D., Bretz, F., and van der Schaar, M. (2020), “Machine Learning for Clinical Trials in the Era of COVID-19,” Statistics in Biopharmaceutical Research, 12, 506–517. DOI: 10.1080/19466315.2020.1797867.
   PubMed Web of Science ®Google Scholar