Music recommender system based on emotions from facial expression

Abstract

Music classification algorithms have become an important component of musical systems. Although current research has had some success in using audio features to classify music, there is a lack of analysis on other crucial musical components, such as the lyrics of a song. Song lyrics can reveal the artist’s intention, which may not be fully conveyed through audio features alone. Hence, this paper explores the extent to which song lyrics can further improve the accuracy of music classification based on emotion. The dataset was created by scraping song lyrics from Genius and extracting audio features using the Spotify API. The songs are split into four basic emotion categories: angry, calm, happy and sad. Both deep learning and transfer learning approaches were employed to build models capable of predicting the emotion based on song lyrics and audio features. Results showed an improvement in accuracy when combining both model predictions. Furthermore, given the deterioration in mental health worldwide, music recommender systems can benefit from an enhanced classification model to recommend music that can improve people’s mood. As such, simple desktop application was also developed to recommend music to users based on their facial emotions detected in real-time. The application integrated the combined model predictions for music recommendation and utilised Spotify API to generate playlists.