The Problem of Limited Inter-rater Agreement in Modelling Music Similarity

References

• Allan, H., Müllensiefen, D. & Wiggins, G.A. (2007). Methodological considerations in studies of musical similarity. In Proceedings of the 8th International Conference on Music Information Retrieval (pp. 473–478).
   Google Scholar
• Aucouturier, J.J. (2009). Sounds like teen spirit: Computational insights into the grounding of everyday musical terms. In J.W. Minett & W.S.-Y. Wang (Eds.), Language, evolution and the brain (pp. 35–64). Hong Kong: City University of Hong Kong Press.
   Google Scholar
• Aucouturier, J.J., & Bigand, E. (2013). Seven problems that keep MIR from attracting the interest of cognition and neuroscience. Journal of Intelligent Information Systems, 41(3), 483–497.
   Web of Science ®Google Scholar
• Bimbot, F., Deruty, E., Sargent, G., & Vincent, E. (2012). Methodology and conventions for the latent semiotic annotation of music structure (Technical report: INRIA-IRISA-METISS). France: INRIA.
   Google Scholar
• Bruderer, M.J., McKinney, M., & Kohlrausch, A. (2006). Structural boundary perception in popular music. In Proceedings of the 7th International Conference on Music Information Retrieval (pp. 198–201). Victoria, BC: University of Victoria.
   Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: L. Erlbaum Associates.
   Google Scholar
• Downie, J.S. (2006). The Music Information Retrieval Evaluation eXchange (MIREX). D-Lib Magazine, 12(12), doi:10.1045/december2006-downie.
   Google Scholar
• Downie, J.S., Ehmann, A.F., Bay, M., & Jones, M.C. (2010). The music information retrieval evaluation exchange: Some observations and insights. In Advances in music information retrieval (Studies in Computational Intelligence, Vol. 274, pp. 93–115). Berlin, Heidelberg: Springer.
   Google Scholar
• Downie, J.S., Hu, X., Lee, J.H., Choi, K., Cunningham, S.J., Hao, Y., & Bainbridge, D. (2014). Ten years of MIREX (Music Information Retrieval Evaluation eXchange): reflections, challenges and opportunities. In Proceedings of the International Society for Music Information Retrieval Conference (pp. 657–662). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Ehmann, A.F., Bay, M., Downie, J.S., Fujinaga, I., & De Roure, D. (2011). Music structure segmentation algorithm evaluation: Expanding on MIREX 2010 analyses and datasets. In Proceedings of the 12th International Society for Music Information Retrieval Conference (pp. 561–566). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
   Google Scholar
• Flexer, A. (2006). Statistical evaluation of music information retrieval experiments. Journal of New Music Research, 35(2), 113–120.
   Web of Science ®Google Scholar
• Flexer, A. (2014). On inter-rater agreement in audio music similarity. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 245–250). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Flexer, A., & Schnitzer, D. (2010). Effects of album and artist filters in audio similarity computed for very large music databases. Computer Music Journal, 34(3), 20–28.
   Web of Science ®Google Scholar
• Flexer, A., Schnitzer, D., & Schlüter, J. (2012). A MIREX meta-analysis of hubness in audio music similarity. Proceedings of the 13th International Society for Music Information Retrieval Conference (pp. 175–180). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Gjerdingen, R.O., & Perrott, D. (2008). Scanning the dial: The rapid recognition of music genres. Journal of New Music Research, 37(2), 93–100.
   Web of Science ®Google Scholar
• Goto, M. (2004). Development of the RWC music database. In Proceedings of the 18th International Congress on Acoustics (pp. I-553–556). Tokyo: Acoustical Society of Japan.
   Google Scholar
• Goto, M. (2006). AIST annotation for the RWC music database. In Proceedings of the 7th International Conference on Music Information Retrieval (pp. 359–360). Victoria, BC: University of Victoria.
   Google Scholar
• Grill, T., Flexer, A., & Cunningham, S. (2011). Identification of perceptual qualities in textural sounds using the repertory grid method. In Proceedings of the 6th Audio Mostly Conference (pp. 67–74). New York: ACM.
   Google Scholar
• Grill, T., & Schlüter, J. (2015). Music boundary detection using neural networks on combined features and two-level annotations. In Proceedings of the 16th International Society for Music Information Retrieval Conference (pp. 531–537). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Jones, M.C., Downie, J.S., & Ehmann, A.F. (2007). Human similarity judgments: Implications for the design of formal evaluations. In Proceedings of the 8th International Conference on Music Information Retrieval (pp. 539–542). Vienna: Austrian Computer Society (OCG).
   Google Scholar
• Klien, V., Grill, T., & Flexer, A. (2012). On automated annotation of acousmatic music. Journal of New Music Research, 41(2), 153–173.
   Web of Science ®Google Scholar
• Hu, X., & Liu, J. (2010). Evaluation of music information retrieval: Towards a user-centered approach. In Proceedings of the 4th Workshop on Human-Computer Interaction and Information Retrieval (pp. 111–114). New Brunswick, NJ: Rutgers University.
   Google Scholar
• Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.
   PubMed Web of Science ®Google Scholar
• Lee, J.H., & Cunningham, S.J. (2013). Toward an understanding of the history and impact of user studies in music information retrieval. Journal of Intelligent Information Systems, 41, 499–521.
   Web of Science ®Google Scholar
• Lee, J.H., Hu, X., Choi, K., & Downie, J.S. (2015). MIREX Grand Challenge 2014 on User Experience: Qualitative analysis of user feedback. Proceedings of the 16th International Society for Music Information Retrieval Conference (pp. 779–785). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Lippens, S., Martens, J., Mulder, T.D., & Tzanetakis, G. (2004). A comparison of human and automatic musical genre classification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 4, iv-233–236.
   Google Scholar
• McFee, B., Nieto, O., & Bello, J.P. (2015). Hierarchical evaluation of segment boundary detection. In Proceedings of the 16th International Society for Music Information Retrieval Conference (pp. 406–412). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Nieto, O., Farbood, M.M., Jehan, T., & Bello, J.P. (2014). Perceptual analysis of the f-measure for evaluating section boundaries in music. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 265–270). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Novello, A., McKinney, M.F., & Kohlrausch, A. (2006). Perceptual evaluation of music similarity. In Proceedings of the 7th International Conference on Music Information Retrieval (pp. 246–249). Victoria, BC: University of Victoria.
   Google Scholar
• Pampalk, E. (2006). Computational models of music similarity and their application to music information retrieval (Doctoral thesis). Vienna University of Technology, Austria.
   Google Scholar
• Pampalk, E., Dixon, S., & Widmer, G. (2004). Exploring music collections by browsing different views. Computer Music Journal, 28(2), 49–62.
   Web of Science ®Google Scholar
• Paulus, J., Müller, M., & Klapuri, A. (2010). Audio-based music structure analysis. In Proceedings of the 11th International Conference on Music Information Retrieval (pp. 625–636). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Pohle, T., Schnitzer, D., Schedl, M., Knees, P., & Widmer, G. (2009). On rhythm and general music similarity. In Proceedings of the 10th International Society for Music Information Retrieval Conference (pp. 525–530). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Raffel, C., McFee, B., Humphrey, E.J., Salamon, J., Nieto, O., Liang, D., & Ellis, D.P.W. (2014). mir\_eval: A transparent implementation of common MIR metrics. In Proceedings of the 15th International Society for Music Information Retrieval Conference (pp. 367–372).
   Google Scholar
• Schedl, M., Flexer, A., & Urbano, J. (2013). The neglected user in music information retrieval research. Journal of Intelligent Information Systems, 41(3), 523–539.
   Web of Science ®Google Scholar
• Schnitzer, D., Flexer, A., Schedl, M., & Widmer, G. (2012). Local and global scaling reduce hubs in space. Journal of Machine Learning Research, 13, 2871–2902.
   Web of Science ®Google Scholar
• Serra, X., Magas, M., Benetos, E., Chudy, M., Dixon, S., Flexer, A., Gomez, E., Gouyon, F., Herrera, P., Jorda, S., Paytuvi, O., Peeters, G., Schl{\"u}ter, J., Vinet, H., & Widmer, G. (2013). Roadmap for Music Information ReSearch, Peeters, G. (editor).
   Google Scholar
• Serrà, J., Müller, M., Grosche, P., & Arcos, J.L. (2014). Unsupervised music structure annotation by time series structure features and segment similarity. IEEE Transactions on Multimedia, Special Issue on Music Data Mining, 16(5), 1229–1240.
   Web of Science ®Google Scholar
• Seyerlehner, K., Widmer, G., & Knees, P. (2010). A comparison of human, automatic and collaborative music genre classification and user centric evaluation of genre classification systems. In Proceedings of the 8th International Workshop on Adaptive Multimedia Retrieval (pp. 118–131). New York: ACM.
   Google Scholar
• Smith, J.B.L., Burgoyne, J.A., Fujinaga, I., De Roure, D., & Downie, J.S. (2011). Design and creation of a large-scale database of structural annotations. In Proceedings of the International Society for Music Information Retrieval Conference (pp. 555–560). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Smith, J.B.L., & Chew, E. (2013). A meta-analysis of the MIREX structure segmentation task. In Proceedings of the 14th International Society for Music Information Retrieval Conference (pp. 251–256). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Sturm, B.L. (2013). Classification accuracy is not enough. Journal of Intelligent Information Systems, 41(3), 371–406.
   Web of Science ®Google Scholar
• Urbano, J., Downie, J.S., McFee, B., & Schedl, M. (2012). How significant is statistically significant? The case of audio music similarity and retrieval. In Proceedings of the 13th International Society for Music Information Retrieval Conference (pp. 181–186). Canada: International Society for Music Information Retrieval.
   Google Scholar
• Vignoli, F. (2004). Digital music interaction concepts: A user study. In Proceedings of the 5th International Conference on Music Information Retrieval (pp. 415–420). Barcelona: Universitat Pompeu Fabra.
   Google Scholar
• West, K. (2008). Novel techniques for audio music classification and search (PhD thesis). University of East Anglia, Norwich, UK.
   Google Scholar
• Wiggins, G. (2009). Semantic gap?? Schemantic schmap!! Methodological considerations in the scientific study of music. Proceedings of the 11th IEEE International Symposium on Multimedia (pp. 477–482). Piscataway, NJ: IEEE Press.
   Google Scholar