Multi-Modal Instrument Performances (MMIP): A Musical Database

Simple item page
dc.contributor.authorKyriakou, Theodorosen_US
dc.contributor.authorAristidou, Andreasen_US
dc.contributor.authorCharalambous, Panayiotisen_US
dc.contributor.editorBousseau, Adrienen_US
dc.contributor.editorDay, Angelaen_US
dc.date.accessioned2025-05-09T09:11:21Z
dc.date.available2025-05-09T09:11:21Z
dc.date.issued2025
dc.description.abstractMusical instrument performances are multimodal creative art forms that integrate audiovisual elements, resulting from musicians' interactions with instruments through body movements, finger actions, and facial expressions. Digitizing such performances for archiving, streaming, analysis, or synthesis requires capturing every element that shapes the overall experience, which is crucial for preserving the performance's essence. In this work, following current trends in large-scale dataset development for deep learning analysis and generative models, we introduce the Multi-Modal Instrument Performances (MMIP) database (https://mmip.cs.ucy.ac.cy). This is the first dataset to incorporate synchronized high-quality 3D motion capture data for the body, fingers, facial expressions, and instruments, along with audio, multi-angle videos, and MIDI data. The database currently includes 3.5 hours of performances featuring three instruments: guitar, piano, and drums. Additionally, we discuss the challenges of acquiring these multi-modal data, detailing our approach to data collection, signal synchronization, annotation, and metadata management. Our data formats align with industry standards for ease of use, and we have developed an open-access online repository that offers a user-friendly environment for data exploration, supporting data organization, search capabilities, and custom visualization tools. Notable features include a MIDI-to-instrument animation project for visualizing the instruments and a script for playing back FBX files with synchronized audio in a web environment.en_US
dc.description.number2
dc.description.sectionheadersBringing Motion to Life: Motion Reconstruction and Control
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume44
dc.identifier.doi10.1111/cgf.70025
dc.identifier.issn1467-8659
dc.identifier.pages16 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.70025
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf70025
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.rightsAttribution-NonCommercial 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by-nc/4.0/
dc.subjectCCS Concepts: Computing methodologies → Motion capture; Motion processing; Mixed / augmented reality; Virtual reality; Machine learning; Applied computing → Performing arts; Digital libraries and archives; Information systems → Information retrieval
dc.subjectComputing methodologies → Motion capture
dc.subjectMotion processing
dc.subjectMixed / augmented reality
dc.subjectVirtual reality
dc.subjectMachine learning
dc.subjectApplied computing → Performing arts
dc.subjectDigital libraries and archives
dc.subjectInformation systems → Information retrieval
dc.titleMulti-Modal Instrument Performances (MMIP): A Musical Databaseen_US
Files
Original bundle
Now showing 1 - 2 of 2
Thumbnail Image
Name:
cgf70025.pdf
Size:
21.84 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
paper1058_1.mp4
Size:
72.49 MB
Format:
Video MP4
Download
Collections
EG 2025 - Full Papers - CGF 44-Issue 2
44-Issue 2