A Statistical Framework for Hypothesis Testing in Real Data Comparison Studies

REFERENCES

• Alon, U., Barkai, N., Notterman, D.A., Gish, K., Ybarra, S., Mack, D., and Levine, A.J. (1999), “Broad Patterns of Gene Expression Revealed by Clustering Analysis of Tumor and Normal Colon Tissues Probed by Oligonucleotide Arrays,” Proceedings of the National Academy of Sciences, 96, 6745–6750.
   PubMed Web of Science ®Google Scholar
• Billingsley, P. (1986), Probability and Measure (2nd ed.), New York: Wiley.
   Google Scholar
• Bock, J. (1998), Bestimmung des Stichprobenumfangs, München/Wien: Oldenbourg.
   Google Scholar
• Boulesteix, A.-L. (2013), “On Representative and Illustrative Comparisons With Real Data in Bioinformatics: Response to the Letter to the Editor by Smith et al. Bioinformatics, 29, 2664–2666.
   PubMed Web of Science ®Google Scholar
• Boulesteix, A.L., Lauer, S., and Eugster, M. J.E. (2013), “A Plea for Neutral Comparison Studies in Computational Sciences,” PLoS One, 8, e61562.
   PubMed Web of Science ®Google Scholar
• Boulesteix, A.L., Strobl, C., Augustin, T., and Daumer, M. (2008), “Evaluating Microarray-Based Classifiers: An Overview,” Cancer Informatics, 6, 77–97.
   PubMedGoogle Scholar
• Braga-Neto, U., and Dougherty, E. (2005), “Exact Performance of Error Estimators for Discrete Classifiers,” Pattern Recognition, 38, 1799–1814.
   Web of Science ®Google Scholar
• Dalton, L., and Dougherty, E. (2012a), “Exact Sample Conditioned MSE Performance of the Bayesian MMSE Estimator for Classification Error—Part I: Representation,” IEEE Transactions on Signal Processing, 60, 2575–2587.
   Web of Science ®Google Scholar
• Dalton, L. (2012b), “Exact Sample Conditioned MSE Performance of the Bayesian MMSE Estimator for Classification Error—Part II: Consistency and Performance Analysis,” IEEE Transactions on Signal Processing, 60, 2588–2603.
   Web of Science ®Google Scholar
• Demšar, J. (2006), “Statistical Comparisons of Classifiers Over Multiple Data Sets,” Journal of Machine Learning Research, 7, 1–30.
   Web of Science ®Google Scholar
• de Souza, B., de Carvalho, A., and Soares, C. (2010), “A Comprehensive Comparison of ML Algorithms for Gene Expression Data Classification,” in The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–8.
   Google Scholar
• Díaz-Uriarte, R., and De Andres, S.A. (2006), “Gene Selection and Classification of Microarray Data Using Random Forest,” BMC Bioinformatics, 7, 3.
   PubMed Web of Science ®Google Scholar
• Dietterich, T. (1998), “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms,” Neural Computation, 10, 1895–1923.
   PubMed Web of Science ®Google Scholar
• Dougherty, E., Zollanvari, A., and Braga-Neto, U. (2011), “The Illusion of Distribution-Free Small-Sample Classification in Genomics,” Current Genomics, 12, 333–341.
   PubMed Web of Science ®Google Scholar
• Dudoit, S., Fridlyand, J., and Speed, T.P. (2002), “Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data,” Journal of the American Statistical Association, 97, 77–87.
   Web of Science ®Google Scholar
• Efron, B., and Tibshirani, R. (1997), “Improvements on Cross-Validation: the 632+ Bootstrap Method,” Journal of the American Statistical Association, 92, 548–560.
   Web of Science ®Google Scholar
• Eugster, M., Hothorn, T., and Leisch, F. (2012), “Domain-Based Benchmark Experiments: Exploratory and Inferential Analysis,” Austrian Journal of Statistics, 41, 5–26.
   Google Scholar
• Eugster, M.J., Leisch, F., and Strobl, C. (2014), “(Psycho-) Analysis of Benchmark Experiments: A Formal Framework for Investigating the Relationship Between Data Sets and Learning Algorithms,” Computational Statistics & Data Analysis, 71, 986–1000.
   Web of Science ®Google Scholar
• Garcia, S., Fernandez, A., Luengo, J., and Herrera, F. (2010), “Advanced Nonparametric Tests for Multiple Comparisons in the Design of Experiments in Computational Intelligence and Data Mining: Experimental Analysis of Power,” Information Sciences, 180, 2044–2064.
   Web of Science ®Google Scholar
• Garcia, S., and Herrera, F. (2008), “An Extension on Statistical Comparisons of Classifiers Over Multiple Data Sets for All Pairwise Comparisons,” Journal of Machine Learning Research, 9, 2677–2694.
   Web of Science ®Google Scholar
• Golub, T., Slonim, D., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J., Coller, H., Loh, M., Downing, J., and Caligiuri, M., Bloomfield, C. D., Lander, E. S. (1999), “Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring,” Science, 286, 531–537.
   PubMed Web of Science ®Google Scholar
• Hanczar, B., and Dougherty, E. (2010), “On the Comparison of Classifiers for Microarray Data,” Current Bioinformatics, 5, 29–39.
   Web of Science ®Google Scholar
• Hand, D.J. (2006), “Classifier Technology and the Illusion of Progress,” Statistical Science, 21, 1–14.
   PubMed Web of Science ®Google Scholar
• Hastie, T., Tibshirani, R., Friedman, J., and Franklin, J. (2009), The Elements of Statistical Learning: Data Mining, Inference and Prediction, New York: Springer.
   Google Scholar
• Hoeffding, W., and Wolfowitz, J. (1958), “Distinguishability of Sets of Distributions. (The Case of Independent and Identically Distributed Chance Variables),” Annals of Mathematical Statistics, 29, 700–718.
   Google Scholar
• Hothorn, T., Leisch, F., Zeileis, A., and Hornik, K. (2005), “The Design and Analysis of Benchmark Experiments,” Journal of Computational and Graphical Statistics, 14, 675–699.
   Web of Science ®Google Scholar
• Jelizarow, M., Guillemot, V., Tenenhaus, A., Strimmer, K., and Boulesteix, A.L. (2010), “Over-Optimism in Bioinformatics: An Illustration,” Bioinformatics, 26, 1990–1998.
   PubMed Web of Science ®Google Scholar
• Lai, C., Reinders, M., Van’t Veer, L., and Wessels, L. (2006), “A Comparison of Univariate and Multivariate Gene Selection Techniques for Classification of Cancer Datasets,” BMC Bioinformatics, 7, 235.
   PubMed Web of Science ®Google Scholar
• Lee, J., Lee, J., Park, M., and Song, S. (2005), “An Extensive Comparison of Recent Classification Tools Applied to Microarray Data,” Computational Statistics & Data Analysis, 48, 869–885.
   Web of Science ®Google Scholar
• Nadeau, C., and Bengio, Y. (2003), “Inference for the Generalization Error,” Machine Learning, 52, 239–281.
   Web of Science ®Google Scholar
• Ripley, B.D. (2008), Pattern Recognition and Neural Networks, Cambridge, UK: Cambridge University Press.
   Google Scholar
• Slawski, M., Daumer, M., and Boulesteix, A.L. (2008), “CMA—A Comprehensive Bioconductor Package for Supervised Classification With High Dimensional Data,” BMC Bioinformatics, 9, 439.
   PubMed Web of Science ®Google Scholar
• Statnikov, A., Aliferis, C., Tsamardinos, I., Hardin, D., and Levy, S. (2005), “A Comprehensive Evaluation of Multicategory Classification Methods for Microarray Gene Expression Cancer Diagnosis,” Bioinformatics, 21, 631–643.
   PubMed Web of Science ®Google Scholar
• Webb, G. (2000), “Multiboosting: A Technique for Combining Boosting and Wagging,” Machine Learning, 40, 159–196.
   Web of Science ®Google Scholar
• Wolpert, D (2001), “The Supervised Learning No-Free-Lunch Theorems,” in Proceedings of the 6th Online World Conference on Soft Computing in Industrial Applications, pp. 10–24.
   Google Scholar
• Yousefi, M., Hua, J., Sima, C., and Dougherty, E. (2010), “Reporting Bias When Using Real Data Sets to Analyze Classification Performance,” Bioinformatics, 26, 68–76.
   PubMed Web of Science ®Google Scholar
• Zollanvari, A., Braga-Neto, U., and Dougherty, E. (2009), “On the Sampling Distribution of Resubstitution and Leave-One-Out Error Estimators for Linear Classifiers,” Pattern Recognition, 42, 2705–2723.
   Web of Science ®Google Scholar
• Zollanvari, A., Braga-Neto, U., and Dougherty, E. (2011), “Analytic Study of Performance of Error Estimators for Linear Discriminant Analysis,” IEEE Transactions on Signal Processing, 59, 4238–4255.
   Web of Science ®Google Scholar