Skip to main content
    Publications

    PianoVAM: A multimodal piano performance dataset

    Y. Kim, J. Park, Joonhyung Bae, K. Kim, T. Kwon, A. Lerch, J. Nam

    Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), Daejeon, Korea, Sep. 21–25, 2025

    2025 · Conference Paper · Published

    arXiv:2509.08800PDF

    In one sentence

    A multimodal dataset of amateur pianists' daily practice recorded on a Disklavier: top-view video, audio, MIDI, hand landmarks and semi-automatic fingering labels.

    Abstract

    The multimodal nature of music performance has driven increasing interest in data beyond the audio domain within the music information retrieval (MIR) community. This paper introduces PianoVAM, a comprehensive piano performance dataset that includes videos, audio, MIDI, hand landmarks, fingering labels, and rich metadata. The dataset was recorded using a Disklavier piano, capturing audio and MIDI from amateur pianists during their daily practice sessions, alongside synchronized top-view videos in realistic and varied performance conditions. Hand landmarks and fingering labels were extracted using a pretrained hand pose estimation model and a semi-automated fingering annotation algorithm. We discuss the challenges encountered during data collection and the alignment process across different modalities. Additionally, we describe our fingering annotation method based on hand landmarks extracted from videos. Finally, we present benchmarking results for both audio-only and audio-visual piano transcription using the PianoVAM dataset and discuss additional potential applications.

    Source: arXiv:2509.08800

    Key facts

    • Real practice sessions of amateur pianists (not studio recordings)
    • Video + audio + MIDI + hand landmarks + fingering labels + metadata
    • Benchmarks for audio-only and audio-visual piano transcription

    When this paper is relevant

    • You train or benchmark audio-visual piano transcription
    • You extract piano fingering automatically from video hand-pose estimation
    • You need piano data from realistic practice conditions rather than studio recordings
    • You deal with synchronising and aligning video, audio and MIDI

    Related

    Cite

    Y. Kim, J. Park, Joonhyung Bae, K. Kim, T. Kwon, A. Lerch, J. Nam (2025). PianoVAM: A multimodal piano performance dataset. Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), Daejeon, Korea, Sep. 21–25, 2025. https://arxiv.org/abs/2509.08800

    @inproceedings{Kim2025pianovam,
  title = {PianoVAM: A multimodal piano performance dataset},
  author = {Y. Kim and J. Park and Joonhyung Bae and K. Kim and T. Kwon and A. Lerch and J. Nam},
  booktitle = {Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), Daejeon, Korea, Sep. 21–25, 2025},
  year = {2025},
  eprint = {2509.08800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.08800}
}

    Machine-readable copy: /papers/pianovam-2025.md