Title:Creating A Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications

Abstract:We introduce a dataset for facilitating audio-visual analysis of musical performances. The dataset comprises a number of simple multi-instrument musical pieces assembled from coordinated but separately recorded performances of individual tracks. For each piece, we provide the musical score in MIDI format, the audio recordings of the individual tracks, the audio and video recording of the assembled mixture, and ground-truth annotation files including frame-level and note-level transcriptions. We anticipate that the dataset will be useful for developing and evaluating multi-modal techniques for music source separation, transcription, score following, and performance analysis. We describe our methodology for the creation of this dataset, particularly highlighting our approaches for addressing the challenges involved in maintaining synchronization and naturalness. We briefly discuss the research questions that can be investigated with this dataset.