A multi-genre model for music emotion recognition using linear regressors

Figures & data

Table 1. Recent work in music emotion recognition (MER).

Source	Model	Emotion	Method	Materials
Lu et al. (2005)	Thayer AV	Perceived	GMM	Western classical music
Y. H. Yang, Lin, Su, et al. (2008)	Thayer AV	Perceived	SVR and AdaBoost.RT	Western, Chinese, and Japanese popular music
Y. H. Yang, Lin, Cheng, et al. (2008)	Russell	Perceived	Regression	Western popular music
Jun et al. (2008)	Thayer AV	Induced Fuzzy logic	Western popular music from Allmusic.com
Sun and Tang (2009)	Thayer ES	Perceived	ICMM (GMM Variant)	Western classical music
Eerola et al. (2009)	Thayer and Russell ETV	Perceived	Multiple Linear Regression, PCA, Partial Least Squares Regression, and Box–Cox	360 soundtrack excerpts
Schmidt et al. (2010)	Thayer AV	Induced	Regression	USPOP2002 (Berenzweig et al., 2004): Western popular music
Chang et al. (2010)	Thayer AV	Induced SVM	Not specified
Myint and Pwint (2010)	Thayer ES	Perceived	SVM and Fuzzy SVM	Western popular music
Deng and Leung (2012)	Russell C	Perceived	Graph embedding, SVR and PCA	Western classical music
Hu and Yang (2014)	Russell C	Perceived	SVR with Radial Basis Function	CH496 dataset (Y. H. Yang & Hu, 2012): Chinese popular music; MER60 dataset (Y. Yang & Chen, 2011): Western popular music; DEAP120 (Koelstra et al., 2011) dataset: video clips from Europe and North America
Roda et al. (2014)	Russell C	Perceived	Regression	Western classical music
Baume et al. (2014)	Semantic Mood Model (5-D MDS) (Barthet et al., 2013)	Perceived (Saari et al., 2013)	SVR	Production music
Wang et al. (2015)	Russell C	Perceived	AEG Extension of GMM	DEAP and MER 60 datasets
Saari et al. (2015)	Valence, arousal, tension	Perceived	Semantic tagging and semantic tagging/audio feature	TR100k TR10k TE600
Mo and Niu (2017)	Thayer ES and Eerola and Vuoskoski categorical model (Eerola & Vuoskoski, 2011)	Not stated	OMPGW (Orthogonal Matching Pursuit, Gabor functions, Wigner distribution function) SVM	Soundtracks; MIREX dataset; MTV database; MediaEval 2015
Hu and Yang (2017)	Russell C	Perceived	Regression	MER60; CH818; AMG1608
Malheiro et al. (2016)	Russell C	Not stated	Regression	AllMusic

Figure 1. Overview of the research process.

Figure 2. A graphical representation of the affective online self-report question format.

Figure 3. Russell's eight categories placed in a circular order on the circumplex model.

Table 2. Songs selected for model creation.

Genre	Song	Artist
Avant Garde	Passage De Recherche	Scanner
Blues	Reconsider Baby	Jools Holland and his Rhythm and Blues Orchestra
Classical	Adagio for Strings	London Philharmonic Orchestra
Country	9 to 5	Dolly Parton
Dance	Pjanoo	Eric Pridz
Disco	Relight my Fire	Vertigo – Dan Hartmann
Easy Listening	(They Long to Be) Close to You	The Carpenters
Electronic	Oxygene Part 4	Jean Michel Jarre
Folk	Gosport Nancy	Bellowhead
Funk/Soul	Across 110th Street	Bobby Womack
Gospel/Choral	Oh! It is Jesus	Soweto Gospel Choir
Heavy Metal	For Whom the Bell Tolls	Metallica
Hip Hop	Girls, Girls, Girls	Jay-Z
Indie	I Wanna be Adored	The Stone Roses
Jazz	Take Five	Dave Brubeck
Pop	Feels Like Heaven	Fiction Factory
Reggae	Stir it Up	Bob Marley
RnB	Try Sleeping with a Broken Heart	Alicia Keys
Rock	Back in Black	AC/DC
World Music	Chan Chan	Buena Vista Social Club

Table 3. Age distribution of participants in development study.

15–24	25–54	55–64	65 and over
8	31	4	1

Table 4. Summary of emotional ratings from participants.

	Perceived emotion				Induced emotion
Genre	Emotion	θ	ρ	${\chi }^2(7)$	Emotion	θ	ρ	${\chi }^2(7)$
Avant Garde	Sad	225	4	62.55	Sad	225	2	36.73
Blues	Sad	225	2	66.55	Relaxed	315	2	76.36
Classical	Sad	225	3	162.91	Sad	225	4	69.82
Country	Happy	0	3	49.45	Happy	0	3	94.18
Dance	Excited	45	3	188.00	Excited	45	4	77.09
Disco	Excited	45	3	78.18	Happy	0	3	87.27
Easy Listening	Happy	0	3	66.55	Relaxed	315	3	43.27
Electronic	Relaxed	315	3	40.73	Relaxed	315	2	53.82
Folk	Happy	0	3	122.91	Happy	0	3	139.27
Funk/Soul	Sad	225	2	17.82	Relaxed	315	2	30.55
Gospel/Choral	Happy	0	3	141.82	Happy	0	3	56.73
Heavy Metal	Angry	90	3	184.00	Excited	45	4	52.00
Hip Hop	Happy	0	3	21.45	Relaxed	315	2	31.27
Indie	Miserable	180	2	25.09	Happy	0	4	31.27
Jazz	Relaxed	315	3	125.45	Relaxed	315	4	72.36
Pop	Happy	0	3	50.55	Relaxed	315	3	46.55
Reggae	Relaxed	315	4	75.64	Relaxed	315	4	118.18
RnB	Sad	225	4	36.73	Happy	0	1	12.73
Rock	Excited	45	3	165.82	Excited	45	3	94.18
World Music	Relaxed	315	3	98.18	Relaxed	315	3	119.27

Table 5. Audio features correlated with arousal and valence.

Arousal		Valence
Feature	Spearman's ρ	Feature	Spearman's ρ
Energy	0.95	Spectral spread	0.86
Standard deviation energy	0.95	Median spectral spread	0.88
Median energy	0.96	Spectral flatness	0.88

Table 6. Linear regression models for perceived arousal and valence.

Audio feature	Regression equation	SSE	$R^2$	RMSE
Arousal
Energy	$f(x)=36.83x-2.809$	2.956	0.921	0.444
Standard deviation energy	$f(x)=166.3x-2.875$	2.905	0.923	0.440
Median energy	$f(x)=37.23x-2.791$	2.941	0.922	0.443
Valence
Spectral spread	$f(x)=\phantom{00.}60x-8.326$	3.942	0.959	0.468
Medial spectral spread	$f(x)=59.94x-8.302$	3.942	0.959	0.468
Spectral flatness	$f(x)=17.56x-3.963$	3.59	0.962	0.447

Table 7. Songs selected for model evaluation.

ID	Genre	Song	Artist
1	RnB	7 Days	Craig David
2	Jazz	Alone Together	Paul Desmond
3	RnB	Blow	Beyoncé
4	Reggae	Bush Doctor	Peter Tosh
5	World Music	Coumba	Orchestra Baobab
6	Hip Hop	Ghetto Gospel	2Pac
7	Rock	Good Times Bad Times	Led Zeppelin
8	Classical	Grieg: Peer Gynt, Op. 23−2	Budapest Philharmonic Orchestra
9	Funk/Soul	Hold on I'm coming	Sam and Dave
10	Rock	I Want It All	Queen
11	World Music	Ibanuje Mon Iwon	King Sunny Ade
12	Reggae	King Tubby Meets Rockers Uptown	Augustus Pablo
13	Folk	Lady Percy	King Charles
14	Electronic	Love on a Real Train	Tangerine Dream
15	Easy Listening	Make It with You	Bread
16	Easy Listening	Mandy	Barry Manilow
17	Heavy Metal	Mouth for War	Pantera
18	Blues	Never Goina Change	Buddy Guy
19	Funk/Soul	Never Never Goina Give Ya Up	Barry White
20	Gospel/Choral	Ntate Mandela	Vusi Mahlasela
21	Hip Hop	Phenomenon	LL Cool J
22	Gospel/Choral	Rain, Rain Beautiful Rain	Ladysmith Black Mambazo
23	Avant Garde	Re-Platform	Yoshihiro Hanno
24	Disco	Ride Like the Wind	Christopher Cross
25	Dance	Right In	Skrillex
26	Country	Rose Garden	Lynn Anderson
27	Dance	Some Chords	Deadmau5
28	Pop	Somewhere in my Heart	Aztec Camera
29	Blues	Stack-A-Lee	Dr. John
30	Indie	Street Spirit	Radiohead
31	Jazz	Summertime	Louis Armstrong
32	Avant Garde	O Superman (for Massenet)	Laurie Anderson
33	Country	Tell Me Why	Taylor Swift
34	Classical	The Death of Falstaff	Neville Marriner
35	Electronic	The Model	Kraftwerk
36	Disco	This Time Baby	Jackie Moore
37	Folk	Truth Is	Levellers
38	Heavy Metal	Wicker Man	Iron Maiden
39	Pop	Wouldn't It Be Good	Nik Kershaw
40	Indie	You Do Something to Me	Paul Weller

Table 8. Age distribution of participants in evaluation study.

15–24	25–54	55–64	65 and over
31	116	3	8

Figure 4. Ground-truth versus predicted-truth mean arousal and valence values.

Figure 5. GT versus PT for arousal dimension showing outliers.

Figure 6. GT versus PT for Valence dimension showing outliers.

Table 9. Performance metrics for affective model.

Emotion dimension	Outliers removed	$R^2$	SSE	RMSE	R^2 lower- bound 95%	R^2 upper- bound 95%
Arousal	N	0.708	29.700	0.884	0.691	1.059
Valence	N	0.674	30.581	0.897	0.562	0.894
Arousal	Y	0.850	12.341	0.585	0.841	1.118
Valence	Y	0.776	19.700	0.740	0.630	0.910

Table 10. Evaluation of proposed system against existing work: (${}^{\ast }$, 1st; ${}^{\ast \ast }$, 2nd; ${}^{\ast \ast \ast }$, 3rd place ranking).

Source	Arousal $R^2$	Valence $R^2$
Y. H. Yang, Lin, Su, et al. (2008)	0.58	0.28
Eerola et al. (2009)	0.85${}^{\ast }$	0.72${}^{\ast \ast }$
Hu and Yang (2014)	0.80${}^{\ast \ast \ast }$	0.26
Roda et al. (2014)	0.75	0.34
Baume et al. (2014)	0.11	0.22
Saari et al. (2015)	0.74	0.46
Mo and Niu (2017)	0.84${}^{\ast \ast }$	0.29
Hu and Yang (2017)	0.84${}^{\ast \ast }$	0.53
Malheiro et al. (2016)	0.59	0.61
Our system	0.71	0.67${}^{\ast \ast \ast }$
Our system (outliers removed)	0.85${}^{\ast }$	0.78${}^{\ast }$

Lu, L., Liu, D., & Zhang, H. J. (2005). Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio, Speech, and Language Processing, 14(1), 5–18. https://doi.org/https://doi.org/10.1109/TSA.2005.860344

 Google Scholar

Yang, Y. H., Lin, Y. C., Su, Y. F., & Chen, H. H. (2008). A regression approach to music emotion recognition. IEEE Transactions on Audio, Speech, and Language Processing, 16(2), 448–457. https://doi.org/https://doi.org/10.1109/TASL.2007.911513

 Web of Science ®Google Scholar

Yang, Y. H., Lin, Y. C., Cheng, H. T., & Chen, H. H. (2008, October 26–31). Mr. Emo: Music retrieval in the emotion plane. 16th ACM International Conference on Multimedia, Vancouver, British Columbia, Canada.

 Google Scholar

Jun, S., Rho, S., Han, B J., & Hwang, E. (2008, July 29–August 1). A fuzzy inference-based music emotion recognition system. 5th International Conference on Visual Information Engineering (VIE 2008), Xi'an, China.

 Google Scholar

Sun, X., & Tang, Y. (2009, December 12–14). Automatic music emotion classification using a new classification algorithm. Second International Symposium on Computational Intelligence and Design, Changsha, China.

 Google Scholar

Eerola, T., Lartillot, O., & Toivianen, P. (2009, October 26–30). Prediction of multi-dimensional emotion ratings in music from audio using multivariate regression models. 10th International Society for Music Information Retrieval Conference (ISMIR 2009), Kobe, Japan.

 Google Scholar

Schmidt, E. M., Turnbull, D., & Kim, Y. E. (2010, March 29–31). Feature selection for content-based, time-varying musical emotion regression. International Conference on Multimedia Information Retrieval (MIR '10), Philadelphia, PA.

 Google Scholar

Berenzweig, A., Logan, B., Ellis, D. P., & Whitman, B. (2004). A large-scale evaluation of acoustic and subjective music-similarity measures. Computer Music Journal, 28(2), 63–76. https://doi.org/https://doi.org/10.1162/014892604323112257

 Web of Science ®Google Scholar

Chang, C. Y., Lo, C. Y., Wang, C. J., & Chung, P. C. (2010, December 16–18). A music recommendation system with consideration of personal emotion. International Computer Symposium (ICS2010), Tainan, Taiwan.

 Google Scholar

Myint, E. E. P., & Pwint, M. (2010, July 5–7). An approach for multi-label music mood classification. 2nd International Conference on Signal Processing Systems, Dalian, China.

 Google Scholar

Deng, J. J., & Leung, C. (2012, October 23–25). Emotion-based music recommendation using audio features and user playlist. 6th International Conference on New Trends in Information Science, Service Science and Data Mining (ISSDM2012), Taipei, Taiwan.

 Google Scholar

Hu, X., & Yang, Y. H. (2014, September 14–20). A study on cross-cultural and cross-dataset generalizability of music mood regression models. 40th International Computer Music Conference (ICMC 2014), Athens, Greece.

 Google Scholar

Yang, Y. H., & Hu, X. (2012, October 8–12). Cross-cultural music mood classification: A comparison on English and Chinese songs. 13th International Society for Music Information Retrieval Conference (ISMIR 2012), Porto, Portugal.

 Google Scholar

Yang, Y., & Chen, H. (2011). Prediction of the Distribution of Perceived Music Emotions Using Discrete Samples. IEEE Transactions on Audio, Speech, and Language Processing, 19(7), 2184–2196. https://doi.org/https://doi.org/10.1109/TASL.2011.2118752

 Google Scholar

Koelstra, S., Muhl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2011). Deap: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 3(1), 18–31. https://doi.org/https://doi.org/10.1109/T-AFFC.2011.15

 Web of Science ®Google Scholar

Roda, A., Canazza, S., & De Poli, G. (2014). Clustering affective qualities of classical music: Beyond the valence-arousal plane. IEEE Transactions on Affective Computing, 5(4), 364–376. https://doi.org/https://doi.org/10.1109/TAFFC.2014.2343222

 Web of Science ®Google Scholar

Baume, C., Fazekas, G., Barthet, M., Marston, D., & Sandler, M. (2014, January 26–29). Selection of audio features for music emotion recognition using production music. Audio Engineering Society Conference: 53rd International Conference: Semantic Audio, London, UK.

 Google Scholar

Barthet, M., Marston, D., Baume, C., Fazekas, G., & Sandler, M. (2013, November 4–8). Design and evaluation of semantic mood models for music recommendation. Proceedings of 14th International Society for Music Information Retrieval Conference, Curitiba, PR, Brazil.

 Google Scholar

Saari, P., Barthet, M., Fazekas, G., Eerola, T., & Sandler, M. (2013). Semantic models of musical mood: Comparison between crowd-sourced and curated editorial tags. 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 1–6.

 Google Scholar

Wang, J. C., Yang, Y. H., Wang, H. M., & Jeng, S. K. (2015). Modeling the affective content of music with a Gaussian mixture model. IEEE Transactions on Affective Computing, 6(1), 56–68. https://doi.org/https://doi.org/10.1109/TAFFC.2015.2397457

 Web of Science ®Google Scholar

Saari, P., Fazekas, G., Eerola, T., Barthet, M., Lartillot, O., & Sandler, M. (2015). Genre-adaptive semantic computing and audio-based modelling for music mood annotation. IEEE Transactions on Affective Computing, 7(2), 122–135. https://doi.org/https://doi.org/10.1109/TAFFC.2015.2462841

 Web of Science ®Google Scholar

Mo, S., & Niu, J. (2017). A novel method based on OMPGW method for feature extraction in automatic music mood classification. IEEE Transactions on Affective Computing, 10(3), 313–324. https://doi.org/https://doi.org/10.1109/T-AFFC.5165369

 Web of Science ®Google Scholar

Eerola, T., & Vuoskoski, J. K. (2011). A comparison of the discrete and dimensional models of emotion in music. Psychology of Music, 39(1), 18–49. https://doi.org/https://doi.org/10.1177/0305735610362821

 Web of Science ®Google Scholar

Hu, X., & Yang, Y. H. (2017). Cross-dataset and cross-cultural music mood prediction: A case on western and Chinese pop songs. IEEE Transactions on Affective Computing, 8(2), 228–240. https://doi.org/https://doi.org/10.1109/TAFFC.2016.2523503

 Web of Science ®Google Scholar

Malheiro, R., Panda, R., Gomes, P., & Paiva, R. P. (2016). Emotionally-relevant features for classification and regression of music lyrics. IEEE Transactions on Affective Computing, 9(2), 240–254. https://doi.org/https://doi.org/10.1109/TAFFC.2016.2598569

 Web of Science ®Google Scholar