Improving Meta-learning for Algorithm Selection by Using Multi-label Classification: A Case of Study with Educational Data Sets

References

• J. Han and M. Kamber, Data Mining: Concepts and Techniques. Morgan Kauffman, 2006.
   Google Scholar
• J. W. Lee and C. Giraud-Carrier, “Automatic selection of classification learning algorithms for data mining practitioners,” Intelligent Data Analysis, vol. 17, no. 4, pp. 665–678, 2013.
   Web of Science ®Google Scholar
• D. H. Wolpert and W. G. Macready, “No free lunch theorems for optimization,” Evolutionary Computation, IEEE Transactions on, vol. 1, no. 1, pp. 67–82, 1997.
   Google Scholar
• W. Klosgen and J. Zytkow, Handbook of data mining and knowledge discovery. Oxford University Press, 2002.
   Google Scholar
• A. A. Freitas, “Comprehensible classification models: A position paper,” SIGKDD Exploration Newsletter, vol. 15, no. 1, pp. 1–10, 2014.
   Google Scholar
• C. Romero, P. G. Espejo, A. Zafra, J. R. Romero, and S. Ventura, “Web usage mining for predicting final marks of students that use Moodle courses,” Computer Applications in Engineering Education, vol. 21, no. 1, pp. 135–146, 2013.
   Web of Science ®Google Scholar
• A. A. Kardan and H. Sadeghi, “A decision support system for course offering in online higher education institutes,” International Journal of Computational Intelligence Systems, vol. 6, no. 5, pp. 928–942, 2013.
   Web of Science ®Google Scholar
• C. Romero and S. Ventura, “Educational data mining: A survey from 1995 to 2005,” Expert Systems with Applications, vol. 33, no. 1, pp. 135 – 146, 2007.
   Google Scholar
• M. Jovanovic, M. Vukicevic, M. Milovanovic, and M. Minovic, “Using data mining on student behavior and cognitive style data for improving e-learning systems: a case study,” International Journal of Computational Intelligence Systems, vol. 5, no. 3, pp. 597–610, 2012.
   Web of Science ®Google Scholar
• N. Bhatt, A. Thakkar, and A. Ganatra, “A survey & current research challenges in meta learning approaches based on dataset characteristics,” International Journal of Soft Computing and Engineering, vol. 2, no. 1, pp. 239–247, 2012.
   Google Scholar
• E. Gibaja and S. Ventura, “Multi-label learning: a review of the state of the art and ongoing research,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 4, no. 6, pp. 411–444, 2014.
   Web of Science ®Google Scholar
• E. Gibaja and S. Ventura, “A tutorial on multilabel learning,” ACM Computing Surveys, vol. 47, no. 3, pp. 52:1–52:38, 2015.
   Google Scholar
• R. E. Schapire and Y. Singer, “BoosTexter: A Boosting-based System for Text Categorization,” Machine Learning, vol. 39, no. 2/3, pp. 135–168, 2000.
   Web of Science ®Google Scholar
• K. Trohidis, G. Tsoumakas, G. Kalliris, and I. Vlahavas, “Multi-label Classification of Music into Emotions,” in International Conference on Music Information Retrieval (ISMIR), 2008.
   Google Scholar
• M.-L. Zhang and Z.-H. Zhou, “Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization,” Knowledge and Data Engineering, IEEE Transactions on, vol. 18, no. 10, pp. 1338–1351, 2006.
   Web of Science ®Google Scholar
• K. Kawai and Y. Takahashi, “Identification of the Dual Action Antihypertensive Drugs Using TFS-Based Support Vector Machines,” Chem-Bio Informatics Journal, vol. 4, pp. 44–51, 2009.
   Google Scholar
• T. Sobol-Shikler and P. Robinson, “Classification of Complex Information: Inference of Co-Occurring Affective States from Their Expressions in Speech,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 32, no. 7, pp. 1284–1297, 2010.
   Web of Science ®Google Scholar
• A. Krohn-Grimberghe, L. Drumond, C. Freudenthaler, and L. Schmidt-Thieme, “Multi-relational matrix factorization using bayesian personalized ranking for social network data,” in ACM international conference on Web Search and Data Mining (ACM WSDM), pp. 173–182, ACM, 2012.
   Google Scholar
• Y. Zhang, S. Burer, W. N. Street, K. Bennett, and E. Parrado-hern, “Ensemble Pruning Via Semi-definite Programming,” Journal of Machine Learning Research, vol. 7, pp. 1315–1338, 2006.
   Web of Science ®Google Scholar
• A. Pen˜a-Ayala, “Educational data mining: A survey and a data mining-based analysis of recent works,” Expert Systems with Applications, vol. 41, no. 4, Part 1, pp. 1432–1462, 2014.
   Web of Science ®Google Scholar
• V. F. Lo´pez, F. de la Prieta, M. Ogihara, and D. D. Wong, “A model for multi-label classification and ranking of learning objects,” Expert Systems with Applications, vol. 39, no. 10, pp. 8878–8884, 2012.
   Web of Science ®Google Scholar
• E. O¨zpolat and G. B. Akar, “Automatic detection of learning styles for an e-learning system,” Comput. Educ., vol. 53, pp. 355–367, 2009.
   Web of Science ®Google Scholar
• P. B. Brazdil, C. Soares, and J. P. da Costa, “Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results,” Machine Learning, vol. 50, 2003.
   Web of Science ®Google Scholar
• L. Chekina, L. Rokach, and B. Shapira, “Meta-learning for selecting a multi-label classification algorithm,” in IEEE International Conference on Data Mining Workshops (ICDMW), pp. 220–227, 2011.
   Google Scholar
• S. Y. Sohn, “Meta analysis of classification algorithms for pattern recognition,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 21, no. 11, pp. 1137–1144, 1999.
   Web of Science ®Google Scholar
• R. Vilalta, C. Giraud-Carrier, P. Brazdil, and C. Soares, “Using meta-learning to support data mining,” International Journal of Computer Science & Applications, vol. 1, no. 1, pp. 31–45, 2004.
   Google Scholar
• J. Kanda, A. Carvalho, E. Hruschka, and C. Soares, “Selection of algorithms to solve traveling salesman problems using metalearning,” International Journal of Hybrid Intelligent Systems, vol. 8, 2011.
   Google Scholar
• C. Romero, J. L. Olmo, and S. Ventura, “A meta-learning approach for recommending a subset of white-box classification algorithms for Moodle datasets,” in International Conference on Educational Data Mining (EDM), pp. 268–271, 2013.
   Google Scholar
• J. Lara, D. Lizcaino, M. Martinez, J. Pazos, and T. Riera, “A system for knowledge discovery in e-learning environments within the european higher education area - application to student data from Open University of Madrid, UDIMA,” Computers & Education, vol. 72, no. 0, pp. 23–36, 2014.
   Google Scholar
• C. Romero, M. Lopez, J. Luna, and S. Ventura, “Predicting students’ final performance from participation in on-line discussion forums,” Computers & Education, vol. 68, no. 0, pp. 458–472, 2013.
   Google Scholar
• Z. Kovacic, “Predicting student success by mining enroment data,” Research in Higher Education Journal, vol. 15, pp. 1–20, 2012.
   Google Scholar
• S. Huang and N. Fang, “Predicting student academic performance in an engineering dynamics course: A comparison of four types of predictive mathematical models,” Computers & Education, vol. 61, pp. 133–145, 2013.
   Web of Science ®Google Scholar
• G. Zhang, T. J. Anderson, M. W. Ohland, and B. R. Thorndyke, “Identifying factors influencing engineering student graduation: A longitudinal and cross-institutional study,” Journal of Engineering Education, vol. 93, no. 4, pp. 313–320, 2004.
   Web of Science ®Google Scholar
• S. Agarwal, G. N. Pandey, and M. D. Tiwari, “Data mining in education: Data classification and decision tree approach,” International Journal of e-Education, e-Business, e-Management and e-Learning, vol. 2, no. 2, pp. 140–144, 2012.
   Google Scholar
• B. Minaei-Bidgoli, D. Kashy, G. Kortemeyer, and W. Punch, “Predicting student performance: an application of data mining methods with an educational web-based system,” in ASEE/IEEE Frontiers in Education Conference, pp. 13–18, 2003.
   Google Scholar
• C. Ma´rquez-Vera, A. Cano, C. Romero, and S. Ventura, “Predicting student failure at school using genetic programming and different data mining approaches with high dimensional and imbalanced data,” Applied Intelligence, vol. 38, no. 3, pp. 315–330, 2013.
   Web of Science ®Google Scholar
• J. L. Olmo, J. R. Romero, and S. Ventura, “Swarm-based metaheuristics in automatic programming: a survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 4, no. 6, pp. 445–469, 2014.
   Web of Science ®Google Scholar
• S. Dzˇeroski and B. Zˇenko, “Is combining classifiers with stacking better than selecting the best one?,” Machine Learning, vol. 54, no. 3, pp. 255–273, 2004.
   Web of Science ®Google Scholar
• R. Vilalta and Y. Drissi, “A perspective view and survey of meta-learning,” Artificial Intelligence Review, vol. 18, no. 2, pp. 77–95, 2002.
   Web of Science ®Google Scholar
• D. H. Wolpert, “Stacked Generalization,” Neural Net-works, vol. 5, pp. 241–259, 1992.
   Web of Science ®Google Scholar
• M. Antenreiter, R. Ortner, and P. Auer, “Combining Classifiers for Improved Multilabel Image Classification,” in Proceedings of the 1st workshop on learning from multilabel data (MLD) held in conjunction with ECML/PKDD, pp. 16–27, 2009.
   Google Scholar
• S. Godbole and S. Sarawagi, “Discriminative Methods for Multi-Labeled Classification,” in Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pp. 22–30, 2004.
   Google Scholar
• K. Kurach, K. Pawlowski, L. Romaszko, M. Tatjewski, A. Janusz, and H. S. Nguyen, “An ensemble approach to multi-label classification of textual data,” in Advanced Data Mining and Applications, LNCS, pp. 306–317, Springer, 2012.
   Google Scholar
• T. Li, C. Zhang, and S. Zhu, “Empirical Studies on Multi-label Classification,” in ICTAI ’06: Proceedings of the 18th IEEE International Conference on Tools with Artificial Intelligence, p. 86–92, 2006.
   Google Scholar
• F. Markatopoulou, V. Mezaris, and I. Kompatsiaris, “A comparative study on the use of multi-label classification techniques for concept-based video indexing and annotation,” in Multimedia Modeling, vol. 8325 of LNCS, pp. 1–12, Springer, 2014.
   Google Scholar
• E. Montan˜es, R. Senge, J. Barranquero, J. R. Quevedo, J. J. del Coz, and E. Hu¨llermeier, “Dependent binary relevance models for multi-label classification,” Pattern Recognition, vol. 47, no. 3, pp. 1494–1508, 2014.
   Web of Science ®Google Scholar
• F. Pachet and P. Roy, “Improving Multilabel Analysis of Music Titles: A Large-Scale Validation of the Correction Approach,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 17, no. 2, pp. 335–343, 2009.
   Web of Science ®Google Scholar
• G. Tsoumakas, A. Dimou, E. Spyromitros, V. Mezaris, I. Kompatsiaris, and I. Vlahavas, “Correlation-Based Pruning of Stacked Binary Relevance Models for Multi-Label Learning,” in Proceedings of the 1st International Workshop on Learning from Multi-Label Data (MLD’09) (G. Tsoumakas, M. Zhang, and Z. e. Zhou, eds.), pp. 101–116, 2009.
   Google Scholar
• J. Kanda, C. Soares, E. Hruschka, and A. de Carvalho, “A meta-learning approach to select meta-heuristics for the traveling salesman problem using mlp-based label ranking,” in Proceedings of the 19th International Conference on Neural Information Processing-Volume Part III, ICONIP’12, pp. 488–495, Springer,y2012.
   Google Scholar
• J. Read, Scalable Multi-label Classification. PhD thesis, University of Waikato, Sept. 2010.
   Google Scholar
• K. Bache and M. Lichman, “UCI machine learning repository,” 2013.
   Google Scholar
• M. D. M. Molina, C. Romero, S. Ventura, and J. M. Luna, “Meta-learning approach for automatic parameter tuning: A case of study with educational datasets,” in International Conference on Educational Data Mining (EDM), pp. 180–183, 2012.
   Google Scholar
• M. Zorrilla and D. Garcia-Saiz, “Meta-learning: Can it be suitable to automatise the kdd process for the educational domain?,” in Rough Sets and Intelligent Systems Paradigms (M. Kryszkiewicz, C. Cornelis, D. Ciucci, J. Medina-Moreno, H. Motoda, and Z. Ras, eds.), vol. 8537 of Lecture Notes in Computer Science, pp. 285–292, Springer International Publishing, 2014.
   Google Scholar
• A. Zapata, V. Menendez, M. Prieto, and C. Romero, “Evaluation and selection of group recommendation strategies for collaborative searching of learning objects,” International Journal of Human-Computer Studies, vol. 76, pp. 22–39, 2015.
   Web of Science ®Google Scholar
• J. Demsˇar, “Statistical comparisons of classifiers over multiple data sets,” Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006.
   Web of Science ®Google Scholar
• S. Natek and M Zwilling. Expert Systems with Applications, vol. 41, no. 14, pp. 6400–6407, 2014.
   Google Scholar
• W. Rice, Moodle 2.0 E-Learning Course Development. Community experience distilled, Packt Publishing Ltd, 2011.
   Google Scholar
• K. Cios, W. Pedrycz, R. Swiniarski, and L. Kurgan, Data Mining: A Knowledge Discovery Approach. Springer, 2010.
   Google Scholar
• S. B. Kotsiantis, I. D. Zaharakis, and P. E. Pintelas, “Machine learning: a review of classification and combining techniques,” Artificial Intelligence Reviews, vol. 26, pp. 159–190, 2006.
   Web of Science ®Google Scholar
• M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The weka data mining software: An update,” SIGKDD Explorations, vol. 11, no. 1, pp. 10–18, 2009.
   Google Scholar
• R. Kohavi, “The power of decision tables,” in European Conference on Machine Learning (ECML), pp. 174–189, 1995.
   Google Scholar
• M. Hall and E. Frank, “Combining naive bayes and decision tables,” in International Florida Artificial Intelligence Research Society Conference (FLAIRS), pp. 318–319, 2008.
   Google Scholar
• W. Cohen, “Fast Effective Rule Induction,” in International Conference on Machine Learning (ICML), pp. 115–123, 1995.
   Google Scholar
• B. Martin, “Instance-based learning: Nearest neighbor with generalization,” Master's thesis, University of Waikato, New Zealand, 1995.
   Google Scholar
• R. C. Holte, “Very simple classification rules perform well on most commonly used datasets,” Machine Learning, vol. 11, no. 1, pp. 63–90, 1993.
   Web of Science ®Google Scholar
• E. Frank and I. H. Witten, “Generating accurate rule sets without global optimization,” in International Conference on Machine Learning (ICML), pp. 144–151, 1998.
   Google Scholar
• D. Richards, “Two decades of ripple down rules research,” The Knowledge Engineering Review, vol. 24, pp. 159–184, 6 2009.
   Web of Science ®Google Scholar
• R. T. Jerome Friedman, Trevor Hastie, “Additive logistic regression : A statistical view of boosting,” Annals of statistics, vol. 28, no. 2, pp. 337–407, 2000.
   Web of Science ®Google Scholar
• J. J. Oliver and D. Hand, “Averaging over decision stumps,” in Machine Learning: ECML-94 (F. Bergadano and L. Raedt, eds.), vol. 784 of LNCS, pp. 231–241, Springer, 1994.
   Google Scholar
• J. R. Quinlan, C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.
   Google Scholar
• G. I. Webb, “Decision tree grafting from the all-tests-but-one partition,” in International Joint Conference on Artificial Intelligence (IJCAI), pp. 702–707, 1999.
   Google Scholar
• G. Holmes, B. Pfahringer, R. Kirkby, E. Frank, and M. Hall, “Multiclass alternating decision trees,” in European Conference on Machine Learning (ECML), pp. 161–172, 2002.
   Google Scholar
• N. Landwehr, M. Hall, and E. Frank, “Logistic model trees,” Machine Learning, vol. 59, no. 1-2, pp. 161–205, 2005.
   Web of Science ®Google Scholar
• R. Kohavi, “Bayes rule based and decision tree hybrid classifier,” 2001. US Patent 6,182,058.
   Google Scholar
• L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.
   Web of Science ®Google Scholar
• T. Elomaa and K. Kaariainen, “An analysis of reduced error pruning,” Journal of Artificial Intelligence Research, vol. 15, pp. 163–187, 2001.
   Web of Science ®Google Scholar
• L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen, Classification and regression trees. CRC press, 1984.
   Google Scholar
• T. Pang-Ning, M. Steinbach, and V. Kumar, Introduction to Data Mining. Addison-Wesley, 2nd ed., 2014.
   Google Scholar
• A. Cano, A. Zafra, and S. Ventura, “Weighted data gravitation classification for standard and imbalanced data,” Cybernetics, IEEE Transactions on, vol. 43, no. 6, pp. 1672–1687, 2013.
   Web of Science ®Google Scholar
• T. Fawcett, “An introduction to roc analysis,” Pattern Recogniticon Letters, vol. 27, no. 8, pp. 861–874, 2006.
   Web of Science ®Google Scholar
• “Shared domains of competence of approximate learning models using measures of separability of classes,” Information Sciences, vol. 185, no. 1, pp. 43–65, 2012.
   Google Scholar
• T. K. Ho and M. Basu, “Complexity measures of supervised classification problems,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 24, no. 3, pp. 289–300, 2002.
   Web of Science ®Google Scholar
• J. M. Sotoca, R. A. Mollineda, and J. S. Sa´nchez, “A meta-learning framework for pattern classification by means of data complexity measures,” Revista Iberoamericana de Inteligencia Artificial, vol. 10, no. 29, pp. 31–38, 2006.
   Google Scholar
• E. Leyva, A. Gonzalez, and R. Perez, “Knowledge-based instance selection: A compromise between efficiency and versatility,” Knowledge-Based Systems, vol. 47, no. 0, pp. 65–76, 2013.
   Google Scholar
• A. Hoekstra and R. Duin, “On the nonlinearity of pattern classifiers,” in Proceedings of the 13th International Conference on Pattern Recognition (ICPR), vol. 4, pp. 271–275, 1996.
   Google Scholar
• L. Frank and E. Hubert, “Pretopological approach for supervised learning,” in Proceedings of the 13th International Conference on, vol. 4, pp. 256–260 vol.4, 1996.
   Google Scholar
• N. Macia`, Data complexity in supervised learning: A far-reaching implication. PhD thesis, La Salle Universitat Ramon Llull, October 2011.
   Google Scholar
• G. Tsoumakas, I. Katakis, and I. Vlahavas, Data Mining and Knowledge Discovery Handbook, Part 6, ch. Mining Multi-label Data, pp. 667–685. Springer, 2010.
   Google Scholar
• G. Tsoumakas, I. Katakis, and I. Vlahavas, “Random k-Labelsets for Multi-Label Classification,” Knowledge and Data Engineering, IEEE Transactions on, vol. 23(7), pp. 1079–1089, 2010.
   Web of Science ®Google Scholar
• Y. Yang, “An Evaluation of Statistical Approaches to Text Categorization,” Information Retrieval, vol. 1, pp. 69–90, 1999.
   Google Scholar
• Y. Yang and X. Liu, “A re-examination of text categorization methods,” in International SIGIR Conference on Research & Development on Information Retrieval, pp. 42–49, 1999.
   Google Scholar
• J. Pestian, C. Brew, P. Matykiewicz, D. Hovermale, N. Johnson, K. Cohen, and W. Duch, “A shared task involving multi-label classification of clinical free text,” in Proceedings of ACL BioNLP, 2007.
   Google Scholar
• R. E. Schapire and Y. Singer, “Improved Boosting Algorithms Using Confidence-rated Predictions,” Machine Learning, vol. 37(3), pp. 297–336, 1999.
   Web of Science ®Google Scholar
• K. Crammer and Y. Singer, “A family of additive online algorithms for category ranking,” Journal of Machine Learning Research, vol. 3, pp. 1025–1058, 2003.
   Web of Science ®Google Scholar
• J. Read, B. Pfahringer, and G. Holmes, “Multilabel Classification Using Ensembles of Pruned Sets,” in IEEE International Conference on Data Mining (ICDM), vol. 0, pp. 995–1000, 2008.
   Google Scholar
• K. Brinker, J. Fu¨rnkranz, and E. Hu¨llermeier, “A Unified Model for Multilabel Classification and Ranking,” in European Conference on Artificial Intelligence (ECAI), pp. 489–493, 2006.
   Google Scholar
• J. Read, B. Pfahringer, G. Holmes, and E. Frank, “Classifier chains for multi-label classification,” Machine Learning, vol. 85, no. 3, pp. 1–27, 2011.
   Google Scholar
• W. Cheng and E. Hu¨llermeier, “Combining instance-based learning and logistic regression for multilabel classification,” Machine Learning, vol. 76, no. 2-3, pp. 211–225, 2009.
   Web of Science ®Google Scholar
• M.-L. Zhang and Z.-H. Zhou, “A k-Nearest Neighbor Based Algorithm for Multi-label Classification,” in IEEE International Conference on Granular Computing (IEEE GrC), vol. 2, pp. 718–721, 2005.
   Google Scholar
• R. McGill, J. W. Tukey, and W. A. Larsen, “Variations of box plots,” The American Statistician, vol. 32, no. 1, pp. 12–16, 1978.
   Web of Science ®Google Scholar