Prediction of Musical Affect Using a Combination of Acoustic Structural Cues

Abstract

This study explores whether musical affect attribution can be predicted by a linear combination of acoustical structural cues. To that aim, a database of sixty musical audio excerpts was compiled and analyzed at three levels: judgments of affective content by subjects; judgments of structural content by musicological experts (i.e., “manual structural cues”), and extraction of structural content by an auditory-based computer algorithm (called: acoustical structural cues). In Study I, an affect space was constructed with Valence (gay-sad), Activity (tender-bold) and Interest (exciting-boring) as the main dimensions, using the responses of a hundred subjects. In Study II manual and acoustical structural cues were analyzed and compared. Manual structural cues such as loudness and articulation could be accounted for in terms of a combination of acoustical structural cues. In Study III, the subjective responses of eight individual subjects were analyzed using the affect space obtained in Study I, and modeled in terms of the structural cues obtained in Study II, using linear regression modeling. This worked better for the Activity dimension than for the Valence dimension, while the Interest dimension could not be accounted for. Overall, manual structural cues worked better than acoustical structural cues. In a final assessment study, a selected set of acoustical structural cues was used for building prediction models. The results indicate that musical affect attribution can partly be predicted using a combination of acoustical structural cues. Future research may focus on non-linear approaches, elaboration of dataset and subjects, and refinement of acoustical structural cue extraction.

Acknowledgements

This work is supported by the EU project MEGA IST-20410 and a grant from BOF 011V6701, Ghent University. The authors wish to thank J. Taelman for assistance, and an anonymous reviewer of JNMR for useful comments.

Notes

¹In estimating the frequency of terms used for music description and search by non-music experts, Kim and Belkin (2002) make a distinction between different categories: terms related to movements, neutral concepts, emotions, nature, objects, occasions or filmed events and musical features. In a music description task, emotion-related terms are used most often, namely in 31% of the cases, while terms related to occasions or filmed events are used in 23% of the cases. In search tasks, emotion-related terms are used in 24% of the cases, while terms related to occasions or filmed events are used in 29%.

²A number of computational models are nowadays available that extract perception-related properties from musical audio such as onset (e.g., Klapuri, 1999; Smith, 1994), beat (e.g., Toiviainen, 2001; Large & Kolen, 1994; Scheirer, 1998; Laroche, 2003), consonance (e.g., Aures, 1985; Daniel & Weber, 1997; Leman, 2000a), pitch (e.g., Clarisse et al., 2002; De Mulder et al., 2004), harmony, tonality (e.g., Terhardt, 1974; Parncutt, 1989, Leman, 1995, 2000b), timbre (e.g., Cosi et al., 1994; Toiviainen, 1996; De Poli & Prandoni, 1997).

³Apart from a preliminary study by Scheirer et al. (2000) , we know of no other attempts that relate these and similar audio-extracted structural features to affect-based description of music. Some researchers, however, have used artificial stimuli to study affect perception using an analysis-by-synthesis methodology. This methodology allows a good control of conditions so that very specific relationships can be revealed between structural properties of the musical audio and affect descriptions (e.g., Hevner, 1936; Rigg, 1939; Thompson & Robintaille, 1992; Juslin, 1997; Gagnon & Peretz, 2003). The method, however, does not guarantee that the resulting model will work for a large set of natural musical stimuli in application domains which are difficult to control.

⁴The French rationalist philosopher Réné Descartes (1596 – 1650) held that the clear and distinct ideas of our intuition form the atomic entities from which all knowledge can be derived by deduction. In this article, we adopt a Cartesian methodology in the sense that all knowledge about affect descriptions of music will be based on predefined atomic entities – namely, auditory modules which extract particular structural features from musical audio. These auditory modules are inspired by the human auditory system, but they are largely predefined in the sense that they are based on the intuition of the programmer. Moreover, they are not learned or trained by any empirical method, although their accuracy has been tested on local problems; for example, the roughness module has been tested on data available in the psycho-acoustical literature. In a similar way, other modules have been tested using data available in the literature (see Appendix B).

⁵The latter term, however, should be used with caution because the term “arousal”, in this context, does not reflect psychophysical conditions but rather straightforward cognitive appraisals of perceived qualities in music. In that context, we believe it is better to use the term “activity”.

⁶Juslin (1997), using an analysis-by-synthesis methodology, uses multiple regression analysis starting from the pooled ratings of 54 subjects. The stimuli and experimental design led to high inter-rater-reliabilities. The approach circumvents the problem of having to extract the acoustical features and seems to work well with high-level structural descriptors. However its success may be due to carefully chosen examples of affective expressiveness. The strategy is less evident when natural musical stimuli from commercial recordings are used, together with a large number of subjects, and automatically extracted low-level structural features for application domains such as music information retrieval and interactive music systems.

⁷For a more detailed account of the conceptual model, see Lesaffre et al., 2003, Camurri et al., 2001.

⁸The semantic differential is a contrasting pair of adjectives that can be used as a scaling instrument to quantitatively measure perceived qualities of an object. The choice of an initial set of m-semantic differentials in the experimental setup induces an m-multidimensional quantitative affect space.

⁹The negative and positive poles of the axis have to be taken into account in the interpretation of the models that are developed in subsequent parts of this article.

¹⁰Ambitus concerns the frequency range from low to high frequency.

¹¹Notice, however, that it would be possible to construct a low-dimensional affect space for each individual subject. We do not do this, because the obtained spaces would tend to be different for each subject, and that would prevent us from comparing the models we set up for each individual subject. Instead, a projection of the individual affect descriptions into the inter-subjective space is made. The individual affect descriptions, in that sense, are constrained by the inter-subjective affect semantics. The approach has the advantage that it provides a suitable inter-subjective basis for comparing individual subjects on which a search for trends can be based. The projection is justified by the fact that the individuals can be assumed to be part of the same population as the subjects that yield the inter-subjective (Valence-Activity-Interest) affect space. We work with individual data, rather than global data, because we want somehow to take into account the subtle differences among individual subjects.

¹²The present model, despite its limited accuracy, has been successfully used in an artistic production called The Unfolding of an Impossible Building - Short Version, a composition by René Mogensen. The performance took place during the IPEM40! festival at Vooruit, 17 – 18 October, 2003 in Ghent in collaboration with dancers Domenico Giustino and Ophra Wolf, and guitarist Laura Maes. Valence and Activity parameters were extracted from musical audio in real time and used for projecting images in different colors on the scene. The system used a combination of EyesWeb (for video processing) and Pure Data (as host for the real-time auditory model).