Can computational music analysis be both musical and computational?

Abstract

This special issue of the Journal of Mathematics and Music addresses the topic of computational music analysis. It arose from a series of two international workshops on the topic, one in Berlin and one in Paris, both of which created interesting discussions and debates. In the call for papers, this special issue welcomed previously unpublished contributions that presented computational approaches of any type of music analysis. As a special focus, all papers were asked to analyse the same piece: the first movement of Brahms’ String Quartet No. 1. The aim was to bring together diverse computational analytical approaches and methodologies, such as structural, motivic, semiotic, comparative, reductional, harmonic, transformational, and others, using a variety of computational implementation techniques. By focusing on to the same piece, similarities, differences, and complementarities among the approaches on both the methodological and the analytical results levels could be more easily observed. Authors were particularly encouraged to consider Forte's [1] and Huron's [2] analyses of the string quartet, and relate them to their own work if possible. Three papers were chosen for publication, which reflect the various aspects and levels of computation involved.

1. A few words on computational music analysis as a discipline

When discussing music analysis, Ian Bent's words have been quoted several times: Analysis is the means of answering directly the question ‘How does it work?’. Its central activity is comparison. By comparison it determines the structural elements and discovers the functions of these elements [3, p. 342]. This definition, or rather common starting point for music analysis discussions, although challenged often, can be particularly appropriate for this sub-area of music analysis which deals with computational approaches of various types, referred to in recent years as computational music analysis (CMA). The reason why Bent's words are significant in this context is that they make clear two issues: the focus and the method. The focus is the musical work; the main method is comparison, related to the concept of similarity.

The focus of music analysis has been extensively discussed by Nattiez, in his account of the neutral level 4. The neutral level in music analysis is the study of a piece (or pieces) of music, without taking into account the composer's intentions (poetic level) or the listener's cognitive mechanisms, intuitions, aesthetic judgements, emotions, or reactions (aesthesic level). An analysis, Nattiez claims, can be neutral, systematic, rigorous, and scientific.

Nattiez was criticized for his ideas at many levels; among them, that an analysis cannot itself be neutral, since it reflects the analyst's perceptions and cultural background 5. This issue, however, to what extent can an analysis be neutral, has a new pertinence nowadays when we consider computational means to analyse music. Can this type of analysis be closer to what Nattiez originally thought about the neutrality, objectivity and scientific nature of music analysis? Researchers working in CMA are called to address the issue.

In an analytical method, the idea of comparison is central in order to establish musical concepts or units of some type which can be used in the analysis, whether these are notes, motives, chords, segments, pitch class sets, patterns, musical parameters, structures, sections, and so on. By comparison, it can be seen how these musical concepts are related to each other, and often (though not always) how they are placed in time. The concept of similarity is therefore paramount and present in all formal analytical approaches, even reductional ones.

There can be several aims of CMA. First, just like in any type of music analysis, the main objective is to produce musicologically interesting results. This might sound obvious, however, it might not always be trivial and straightforward. Music analysis should not be a mere ‘output’ of an algorithm. Secondly, the formalization of an analytical process can be just as important, especially when thinking of music analysis as a human task which could be formally modelled, making sure that there are no hidden steps and assumptions in the procedure. The analyst always needs to take certain decisions, especially regarding analytical criteria, which in music are always context dependent. There are many degrees of automation and assistance, and in this context, a computer-assisted analysis is just as significant, and perhaps even more meaningful, than a fully automated one.

A third related aim, obvious though sometimes frowned upon, would therefore be simply to assist an analyst, not in formalizing a process, but simply to perform calculations that would have otherwise been difficult, tedious or somehow impossible.

Often, in practice there is a fourth aim: to test computational methodologies in a very challenging and abstract domain, such as music. There is nothing inherently wrong with this goal, though thinking music as simply a domain, where computational approaches can be tested unfortunately can disregard the musical validity of the results, putting more emphasis on the computational methodology and less on its appropriateness.

Perhaps the most interesting challenge in this enterprise is to combine the objectivity and scientific rigour of informatics, with the interpretative nature of music analysis. Is it possible? The answer of this special issue is yes, this is possible with caution, and can be achieved in several ways. Most methods attempt to accommodate both, in different ways, as explained below.

Related to this discussion, a question which needs to be addressed in any computational analytical task is the restrictions and allowances given to the analyst. Can the analyst be eliminated? Can we create intelligent systems that analyse music, without taking into account the human factor? In this issue, our answer is certainly not. None of the three approaches presented are fully automated, in that the analyst (programmer) is needed to take decisions, whether these are to specify a threshold, a parameter, or a more substantial role than that; Also, to interpret and evaluate the system's results.

In practical terms, the field of CMA has developed not without its own problematic issues, that in future research in the area perhaps need to be addressed. Some of them are:

1. The division there exists between the analysis starting from various symbolic representations and the audio signal. Often these two worlds do not meet, and this becomes apparent in the various conferences that exist on the topic.

2. The lack of emphasis on the representational issues, which nevertheless are crucial both for the formalization and the result aspects

3. The plethora of approaches, with a distinct lack of comparisons and discussions between them – although MIREX 6 is an important step towards this direction

4. The lack of connections to the field of more traditional music analysis, very clearly pointed out by Marsden [7 , a fact which has resulted in our opinion from both sides.

5. The musical evaluation of a system's results

Finally, CMA should be distinguished from Music Information Retrieval (MIR), though there is an obvious large overlap. In the first case, the primary aim is to find answers about the piece of music, and its structural characteristics. In the second case, the aim is to pick out specific features or patterns across large bodies of music, in order to classify them, compare them, or produce any type of similarity judgments (such as pattern discovery). Quite often, MIR interestingly focuses on pop and folk music, as there exist relatively homogeneous musical corpora which can be compared and contrasted. There are usually real life applications associated with the field, often with commercial use. However, methodologies from music information retrieval, or data mining, can be used for music analysis. The difference is that the focus in this case is the piece of music, which may or may not be a stylistic analysis, and that the evaluation of the results in musical analytical terms is essential in order to produce a meaningful music analysis.

Lartillot 8 and Van Kranenburg et al. 9 provide two interesting related overviews: The first concentrates on motivic analysis as pattern extraction. The second provides a sensitive account of MIR and folk music analysis, pointing out the essential role of musicologically informed approaches. Both papers, as well as Marsden's 7 include good accounts of related works in the area, where the reader is pointed at.

2. A brief history of the special issue

This special issue takes its origins from the two workshops on the topic: The first was the Workshop on comparative CMA 10 at the Society for Mathematics and Computation in Music first biennial conference, that is Mathematics and Computations in Music 2007 Conference (MCM 2007), that took place in Berlin, Germany, on 18–20 May 2007. At this workshop, participants prepared their proposed computational analysis method around few selected musical works proposed by participants. The aim was to focus on methodological discussions and challenging issues, rather than results. However, the number of music pieces being almost equal to the number of participants (see the comparative computational analysis section in 11), the planned comparative discussions at the workshop had to be adapted to more collaborative sessions of analytical work on the spot. The potential and interest in the comparison of different approaches turned out to be such that we decided to host a second 2-day workshop, the International workshop on CMA 12, as part of the Séminaire MaMuX: Mathématiques, musique et relations avec d'autres disciplines at IRCAM (Paris, France) on 5–6 April 2008. At this event, we beforehand consulted with potential participants in regard to selecting a unique music piece for the analysis. We ended up proposing Brahms’ String Quartet op. 51 First Movement, and the reasons for this choice are explained below. It ensued with a proposal to the Journal of Mathematics and Music Editors-in-Chief, Thomas Noll and Robert Peck, to follow up the workshop with an issue on the topic. Although a call for papers was sent through various sources, all three papers of this special issue turn out to be among the contributions presented at that MaMuX workshop.

This special issue distinguishes from the Special Issue on Computations 13 because of its more focused aim on music analysis. However, it is worth mentioning that despite its broader topic and aim of ‘featur[ing] articles that connect mathematical models in music theory, analysis, composition/improvization, performance and cognition to computational techniques, in novel and integral ways’ [13, p. 57], the Special Issue on Computations finally resulted in a choice of three articles discussing various issues and techniques related to computational analysis.

3. The choice of Brahms’ String Quartet Op. 51 first movement

The piece chosen for analysis is an example of Brahms’ most advanced writing, where the composer destroyed some 20 versions before the version we know; the quartet was first composed in 1865–1866, then finalized in 1873. The quartet's first movement, in C minor, has a general sonata form: an Exposition (bars 1–83) with two contrasting subject groups often heard together; Development (bars 84–132) using materials from exposition; Recapitulation (133–260) with two subject groups; and Coda (bars 224–260). For an interesting brief historical account of the quartet, we refer the reader to 14; see 15 for a detailed discussion on its structure.

As has been stressed above, the decision to focus on a common piece of music for all contributions was a first step towards comparing, complementing, and contrasting methodologies and results of different approaches (though with the downside of potentially limiting the contribution submissions). The quartet was chosen mainly due to its richness and depth, and due to the fact that there already exist interesting proposed analyses in the literature that could be discussed and compared with; in particular, Forte's 1 and Huron's 2 which are briefly discussed below. It is a piece in four voices, however, it is possible to single out one, if needed, for those computational approaches focusing on melodic lines.

In his analysis of Brahms’ quartet, Forte 1 addresses the importance of the motive. Using the set theory and Schenkerian principles, he summarizes the motives of the quartet in a table, and stresses that they are all related to each other in various ways. According to Forte, the main motive of the quartet first movement is the opening C-D-Eb motive, which he calls alpha motive. In his analysis, motives are pitch, pitch-class specific or pitch-class interval specific, and are essentially interval motives, with their transformations, thus leaving out the rhythmic component. He also considers the boundary interval feature of motive (e.g., C-Eb for the alpha motive). Transformations are considered, such as retrograde, inversion, combination thereof, minor-to-major, and major-to-minor change. He finds some motives in a middle-ground Schenkerian structure.

Huron's analysis 2 of the quartet is a response to Forte's paper 1. In his paper, Huron discusses musical features, and their ability to distinguish between pieces. He proposes the notions of presence, salience, distinctiveness, and significance. His analysis of Brahms’ quartet first movement serves as an illustration of his proposed theory. He finds that Forte's alpha motive does not distinguish the quartet from other ones by Brahms. However, when using the prime form of the alpha motive linked with its rhythmic pattern (long-short-long), Huron shows that the motive then becomes distinctive.

4. The papers in this special issue

Three papers have been selected for publication in the current special issue, which present three different approaches to CMA, using different computational methods and aiming at describing different aspects of music. The three in this issue use symbolic representations, look for patterns of different types in the musical surface, and above all, they stress the importance of the validity of the musical results. All three authors were part of either or both previous workshops, and their work had already been extensively discussed in these contexts.

The issue opens with Conklin's Distinctive patterns in the first movement of Brahms's String Quartet in C minor, which describes a motivic analysis using data mining methodologies. More specifically, rather than a classical motivic analysis (Forte), or a deductive computational query of a single motive (Huron), Conklin considers whether the data itself could reveal what Forte and Huron propose, from an inductive computational analysis point of view. In this context, a maximally general distinctive pattern is a pattern that is frequent and over-represented in a piece under analysis, as compared with an anti-corpus that is other comparable pieces, and not subsumed by any other more general distinctive pattern. For comparison purposes, he uses the three Brahms’ string quartets, and comes to the conclusions that: with two expected exceptions, all of Forte's motive groups can be found using the proposed pattern discovery method. Regarding specifically the alpha motive group, in agreement with Huron, only the prime form is found to be distinctive. In contrast to Huron, he finds that a horizontal or sequential specialization of Forte's alpha motive is distinctive. An additional interesting result is that the alpha motive is not ranked the highest, and several other motives are found to be more distinctive. The author in his paper concentrates on the claims by Forte and Huron, and the few novel motives proposed could be investigated further in future research. A more complete list of related approaches to patterns and data mining in music analysis can be found in 8. There also exist computational inductive approaches to motivic analysis outside the context of data mining, such as Buteau and Mazzola's topological approach 16.

In the second paper, Tenkanen's Tonal trends and α -motif in the first movement of Brahms’ String Quartet op. 51 nr. 1, the author principally proposes a computational approach that aims at describing the tonality evolution of the piece. Building on an overlapping chord segmentation of a score, Tenkanen measures the similarity between each chord segment with a fixed selected comparison structure, that is, a certain set of pitch-classes related to the key signature. By averaging similarity values at each onset, the resulting description is represented by a tonality trend curve over time onsets (what he calls ‘tonality CSA-curves’). As such, Tenkanen's approach could be reviewed as a specific case of a Riemann Logic 17 with one single harmonic function. In the context of the special issue, Tenkanen furthermore looked into the tonal environment of the alpha motive in Brahms’ String Quartet Nr. 1. This interesting attempt to combine both tonal and motivic analyses led him to perform statistical tests on three different representations of the motive (mainly, Forte's, Huron's, and Tenkanen's). The author concludes that the alpha motive pattern, independently of its three representations, occurs more prominently in remote tonal regions.

Cathé’s Harmonic vectors and stylistic analysis: A computer-aided analysis of the first movement of Brahms’ String Quartet Op. 51-1, the final paper of this issue, presents a harmonic analysis of the first movement, based on Nicolas Meeùs’ theory of harmonic vectors. The author looks exhaustively at local patterns of root progressions, classifying them according to the theory. First he considers intervals between pairs of successive root notes, and then longer interval patterns between three consecutive root progressions. These patterns are classified into types of harmonic vectors. As a special case, he discusses an interval pattern that he calls harmonic pendulum, where the first and third roots are the same. The involved computation in this work, as stressed by the author, is not complex, making this an example where the computer is used as a tool to assist the analyst in calculations. At the same time, there is a rare depth in the evaluation and discussion of the analytical results presented, which focus both on the composer's musical style and the piece under analysis. The work is heavily comparative, discussing many other composers in the course of the paper, and the author finds that harmonic vectors can distinguish between Brahms and other composers, as well as this quartet from another Brahms piece. He places Brahms in a historical and geographical context, claiming that his harmonic language was closer to the French composers of the time.

5. Comparison of the approaches

Following MCM 2007 10, Buteau et al. 18 proposed a preliminary typology for the comparison of computational approaches to melodic analysis. In the 2008 MaMuX workshop, the typology was refined and extended by all participants to better categorize various CMA approaches, making explicit their musical, technical and conceptual differences. When approaches are applied to the same music piece, as it is the case in this Issue, it provides a more transparent background to discuss differences, similarity, contradictions, or complementarities of analytical results.

Most computational analytical methods become faced with akin underlined design choices that need to be addressed. Below we look at some of the most common ones, discussing how the three papers attempt to answer them, starting by specifying the general analytical framework involved.

Analytical method

All three authors are close to the motivic and comparative analytical framework. In the case of Conklin, by taking maximally general patterns, he ensures that these patterns are as short as they can be, while still retaining distinctiveness. In his subsumption tree, he finds the motives already discussed by Forte and Huron, as well as proposing his other new interesting ones. His method is therefore motivic, as well as comparative, since in order to find the distinctive patterns, the quartet is compared with the other two by the same composer. Tenkanen's work focuses on the harmonic and related tonality dimensions, comparing chordal segments to a given tonal structure. In this respect, it is hard to compare his approach to more established analytical techniques. The comparison is not to other pieces of music, thus his analysis is harmonic and structural. Furthermore, Tenkanen also attempts to combine his harmonic analysis to motivic structure by making use of statistical tests. Cathé looks exhaustively at all root progressions, providing calculations at different levels of abstraction, and in comparison with different composers of the same origin and/or area. He looks at successive root progressions, in segments of length three. This is not a motivic analysis in terms of motives discovered. However, these patterns are compared with existing structures of intervals in order to be classified. His work is also heavily comparative, comparing his findings, on other Brahms and other composers’ pieces.

Musical objects and input format

Symbolic approaches to music analysis often choose to use MIDI as their input format, due to its availability, and so does Tenkanen in this case. The specific piece, however, also exists in Humdrum, and forms Conklin's starting point. Cathé uses a manually entered score providing a list of chords. Tenkanen and Cathé look at chord sequences in respect to tonality, while Conklin looks for patterns within melodic sequences. Tenkanen also investigates certain melodic motives. Cathé more specifically looks for patterns of root progressions.

Representations and musical parameters

The musical properties chosen by Forte and Huron in their description of the piece's motives provide a context in choosing representations and, in consequence, analytical criteria. Conklin uses the multiple viewpoint representation formalism, where musical parameters are looked at independently and linked to each other, according to the analyst's preferences. In the current paper, he chooses to concentrate on intervals alone to allow comparisons with Forte's results. In his approach, Tenkanen follows the set theory and represents chords, as well as what he calls ‘comparison structure’ (key signature), as sets of pitch-classes. When considering motives, he chooses multiple representations involving pitch class intervals and onset intervals or patterns (long-short). Cathé looks specifically at various intervals between root chord progressions and their classifications. Like Conklin, the only musical parameter that he concentrates on is intervals, which are adequate to point out differences between various composers and pieces.

Segmentation

In Conklin, there is no initial segmentation, and pattern boundaries are specified by repetition, while he is reporting the maximally general ones (shorter patterns) which cannot be subsumed. Tenkanen uses an automatic segmentation procedure, with contiguous note segments with a fixed cardinality. In Cathé, all interval tuplets of root progressions (therefore patterns of three roots) are investigated, providing an exhaustive account of all patterns found, and discussed with respect to Meeus’ theory.

The issue of similarity

Conklin's paper, using pattern discovery, looks for identical patterns within the chosen musical parameter, intervals, which are translated as resulting similarities in the musical surface. The different levels of abstraction in the representation allow different levels of resulting similarities. Cathé’s looks for identities, but to given specific patterns of different abstraction levels, as defined by the Meeus theory. Each two-interval pattern of root chord progressions is compared with the patterns suggested and labelled by the theory, in order to be classified. Tenkanen takes on a different approach, looking at the similarity between chords using a distance function (involving Pearson correlation coefficient), while, similarly to Conklin, he uses strict identity of motive representations for similarity of motives.

Computational tools

Conklin uses a sophisticated computational and statistical framework in a machine learning approach, implemented in Perl and C. Tenkanen uses a mathematical and statistical framework with hypothesis testing, and similarity distance functions, implemented in R. Cathé uses basic statistics on an exhaustive list of patterns in the piece, using Microsoft's Excel to perform calculations.

Other considerations

One interesting point is the controversy on the treatment of Forte's alpha motive (Cathé did not discuss this in his paper): On the one hand, Tenkanen (Section 3, Paragraph 2) allows 2,2 and −2,−2 (epsilon inverse and epsilon) as instances of alpha, justifying this by quoting Huron. Conklin, on the other hand, in accordance with Forte, keeps the two motives, alpha and epsilon, as separate instances. Huron's claim was that Forte made alpha and epsilon different categories because he was forced to do so by the use of the set theory. It is true that alpha and epsilon cannot be combined together in the set theory, but even if they could, would Forte have chosen to combine them? Conklin, however, discovered that both alpha and epsilon are independently distinctive, a fact which supports Forte's original decision.

Another distinct point that should be made here is the concept of tonality in Tenkanen and Cathé: In Tenkanen, this is a more rigid concept, where the author calculates the presence of tonic and tonal strength at each stage. In Cathé, the concept of tonality is not apparent as the author focuses on local root progressions and their respective motions in terms of intervals and direction. These motions are classified into dominant and subdominant vectors, forming various local root progression patterns, which turn out to be able to differentiate between various composers and styles.

6. Concluding remarks

In this introduction, discussing certain aspects related to CMA and taking as special cases the three approaches described in the special issue, we tried to address the delicate issue of the balance between computation, musicologically sound methodology, and evaluation of results. We highlighted the choices of analytical criteria and representations related to any approach, explaining how, in our opinion, no analysis can ever be fully automated. The question whether CMA can be both musical and computational has purposely not been answered directly, and is left as an exercise for the reader. It is interesting that sometimes approaches with the least computational complexity can reveal the most musically relevant results.

It would have also been an interesting aim for this special issue to discuss the results with respect to one another. Despite the common piece and the original intention, this was not possible, as the three papers focused on different aspects of the music.

Present work in CMA would certainly benefit from more comparative studies between approaches as a way forward, rather than research taking on individual paths. This special issue is an attempt towards bringing the work of different researchers closer, hoping to point out a gap in the literature on which future work can concentrate.

Finally, we argue that CMA comes much closer to what Nattiez had originally envisaged: An analysis becomes more objective and therefore closer to its neutral level when the analyst's intuitions and choices have been made explicit and the analytical procedure is formalized computationally. However, the human factor is still crucial and necessary in any analytical approach, as is the inherent diversity in music analysis.

This special issue is dedicated to the memory of Professor Raymond Monelle, who passed away on 12 March 2010. With his work on systematizing musical semantics and semiotics in general, Monelle set a basis for a lot of the work in the area of formal and CMA that followed. In later years he became interested in the idea of computational music modelling and, although not always in agreement with its main principles, was looking forward to seeing this special issue in print.

Acknowledgements

Many thanks go to all the reviewers and to the Journal Editors, Thomas Noll and Bob Peck for all their help. Many thanks also to everyone who submitted papers, and to the three selected authors who worked closely with the reviewers and the guest editors. Special thanks to Alan Smaill, David Huron, and Darrell Conklin for interesting exchanges of opinions and feedback. Also, to http://a.nnotate.com, for using their helpful website for collaborative annotations during the reviewing and correction stages.