|
MUSEC: Music Sentiment based Expression and Composition Framework
| Ralph Abboud and Joe Tekli |
| SOE, Dept. of Electrical & Computer Eng. Lebanese American University 36 Byblos, Lebanon |
| ralph.abboud@lau.edu, joe.tekli@lau.edu.lb |
I. Introduction
Over the past years, several approaches have been developed to create autonomous algorithmic music composers using the latest developments in evolutionary computing, machine learning, and music-theoretical frameworks. Most existing approaches in this area focus on composing musical pieces that appear theoretically correct or interesting to the listener. However, very few methods have targeted sentiment-based music composition: generating music that expresses human emotions. The few existing methods are restricted in the spectrum of emotions they can express (usually to two dimensions: valence and arousal) as well as the level of sophistication of the music they compose (usually monophonic, following translation-based or heuristic textures). The main goal of our study is to develop a sentiment-based music composer that can produce musical pieces that are: i) expressive in the human emotions they can convey, ii) more advanced in the level of sophistication of the music textures that they produce, while being iii) appealing and enjoyable by listeners. To this end, we introduce MUSEC , a framework for autonomous Music Sentiment-based Expression and Composition, designed to perceive an extensible set of six human emotions (e.g., anger, fear, joy, love, sadness, and surprise) expressed by a symbolic MIDI musical file, and then compose (create) new original musical pieces (in MIDI format) with sophisticated polyphonic textures that can express these sentiments. To achieve this, MUSEC first "learns" how to perceive emotions in music, using a supervised machine learning process to perform music sentiment analysis. We view this as a required step toward sentiment-based composition, similarly to human composers who first need to appreciate a human emotion to be able to truly reflect it in their compositions. Then, MUSEC starts composing using a dedicated evolutionary-developmental approach, by evolving simple (atomic) music patterns into more sophisticated and full-fledged pieces that express the user's target sentiments.
II. System Architecture
The general architecture of our MUSEC is shown in Fig. 1. It consists of four main modules: i) music (high-level symbolic and low-level frequency-domain) feature parsing (FP), ii) music theory knowledge base (KB , including music-theoretic operations, rules, and parameters to produce "correct" music), iii) music s entiment learner (SL, consisting of a non-parametric fuzzy classifier coined with a dedicated music similarity evaluation engine) that learns to infer sentiments in music from exposure to previously analyzed and newly composed musical pieces, and iv) music sentiment-based composer (MC : the core component of MUSEC, consisting of an evolutionary-developmental framework, with specially tailored music evolution, mutation, and sentiment-based trimming operators), allowing to generate new, original, and diversified music compositions that express target emotions. Unlike existing evolutionary approaches to music composition, MUSEC's assessment of composition quality does not hinge on music theoretical correctness only (which is covered by its KB module), but rather on the target sentiments that a composition needs to express at the behest of the user. More specifically, MUSEC's evolutionary composition module (MC) integrates the sentiment learner (SL) as its fitness evaluation function, in order to select (among the large number of) produced musical pieces those that best express the user's target sentiments. In turn, the composer (MC) feeds the learner (SL) in order to better infer sentiments from exposure to newly composed pieces. Also, MUSEC is expressive and extensible in the panel of emotions it can convey, producing pieces that reflect a target crisp sentiment (e.g., love) or a collection of fuzzy sentiments (e.g., 65% happy, 20% sad, and 15% angry), where the number of sentiment categories can be extended following the user's preferences, compared with crisp-only or two-dimensional (valence / arousal) sentiment models used in existing solutions. In addition, it utilizes an extensive set of 18 different music-theoretic mutation operators (trille , staccato , repeat , compress , single suspension , etc.), which usage is stochastically orchestrated within the evolutionary process, to add atomic (individual chord-level) and thematic (chord pattern-level) variability to the composed polyphonic pieces, compared with simpler monophonic music produced by existing solutions.
 |
| Fig. 1. Overall MUSEC architecture |
III. Implementation and Tests
We conducted a battery of tests to evaluate the effectiveness and efficiency of MUSEC's modules, including feature parsing time, sentiment expression accuracy, as well as composition time, quality, and accuracy in expressing target emotions, involving assessments by non-expert listeners as well as expert composers and music instructors. Our system was trained on a specially constructed set of 120 MIDI pieces, including 70 sentiment-annotated pieces: the first significant dataset of sentiment-labeled MIDI music made available online as a benchmark for future research in this area . Results are very encouraging and highlight the potential of our approach in different application domains, ranging over music information retrieval, music composition, assistive music therapy, and social intelligence.
The prototype system, dataset, and experimental results can be downloaded from the following links:
Acknowledgments
This study is partly funded by the National Council for Scientific Research - Lebanon (CNRS-L), by LAU, as well as the Fulbright Visiting Scholar program (sponsored by the US Department of State). Special thanks go to music experts: Anthony Bou Fayad, Robert Lamah, and Joseph Khalifé, who helped evaluate the synthetic compositions, as well as Jean Marie Riachi for his participation in a live demonstration of the system. We would also like to thank the non-expert testers (including LAU students, faculty, staff, and friends) who volunteered to participate in the experimental evaluation.