Image Is All for Music Retrieval: Interactive Music Retrieval System Using Images with Mood and Theme Attributes

Abstract

We propose an intuitive image-to-music retrieval (IMR) framework to improve the user experience on these platforms. The proposed method extracts mood and theme tags by searching for images from a pre-built database that are similar to a query image and then retrieves music with matching tag information. We investigated the system’s effectiveness by comparing participants’ satisfaction, intention to use, and valence between those who interacted with the system and those who did not. We also examined whether using mood or theme attributes affected the user-perceived suitability of the retrieved music. Results showed that all three variables of the interaction group were significantly higher than that of the non-interaction group and that there was no difference in the perceived suitability of music between the mood and theme attributes. Our study concludes that image attributes are effective in successful music retrieval and that interaction is a crucial factor in designing IMR systems.

Keywords:

• Image-to-music retrieval system
• user interface
• human–computer interaction
• multimodal information

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The music data that support the findings of this study are openly available in the MTG-Jamendo Dataset at http://hdl.handle.net/10230/42015, and the image data are available from the first author J. Park upon reasonable request.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C200337911 and No. RS-2023-00220762).

Notes on contributors

Jeongeun Park

Jeongeun Park is currently a PhD candidate at the Graduate School of Information at Yonsei University. She received a BS in electrical and electronic engineering and a master’s degree in information systems from Yonsei University. Her primary research areas are human–computer interaction, generative models, and deep-learning applications.

Minchae Kim

Minchae Kim is a master’s student at the Graduate School of Information at Yonsei University. She obtained her BS degree in law at Sookmyung Women’s University. She is currently conducting research on the application of deep-learning algorithms. Her main research areas are data analysis, image processing, and multimodal learning.

Ha Young Kim

Ha Young Kim is an Associate Professor at the Graduate School of Information at Yonsei University. She received her PhD from the Department of Mathematics at Purdue University. She was a researcher at the Samsung Electronics. Her primary research areas are deep learning, human–computer interaction, and intelligent applications.