# PianoVAM: A multimodal piano performance dataset

- Authors: Y. Kim, J. Park, Joonhyung Bae, K. Kim, T. Kwon, A. Lerch, J. Nam
- Venue: Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), Daejeon, Korea, Sep. 21–25, 2025
- Year: 2025
- Type: Conference Paper
- arXiv: https://arxiv.org/abs/2509.08800
- PDF: https://drive.google.com/file/d/1uDZkNIy-hFeJhQXola3xsc6Ufm9mciSK/view?usp=drive_link
- Keywords: piano performance, multimodal dataset, music information retrieval, performance analysis, audio-visual data
- Page: https://joonhyungbae.com/publication/pianovam-2025

## In one sentence

A multimodal dataset of amateur pianists' daily practice recorded on a Disklavier: top-view video, audio, MIDI, hand landmarks and semi-automatic fingering labels.

아마추어 피아니스트의 일상 연습을 디스클라비어로 기록한 멀티모달 데이터셋 — 상단 영상, 오디오, MIDI, 손 랜드마크, 준자동 운지 라벨.

## Abstract

The multimodal nature of music performance has driven increasing interest in data beyond the audio domain within the music information retrieval (MIR) community. This paper introduces PianoVAM, a comprehensive piano performance dataset that includes videos, audio, MIDI, hand landmarks, fingering labels, and rich metadata. The dataset was recorded using a Disklavier piano, capturing audio and MIDI from amateur pianists during their daily practice sessions, alongside synchronized top-view videos in realistic and varied performance conditions. Hand landmarks and fingering labels were extracted using a pretrained hand pose estimation model and a semi-automated fingering annotation algorithm. We discuss the challenges encountered during data collection and the alignment process across different modalities. Additionally, we describe our fingering annotation method based on hand landmarks extracted from videos. Finally, we present benchmarking results for both audio-only and audio-visual piano transcription using the PianoVAM dataset and discuss additional potential applications.

(Source: arXiv:2509.08800)

## Key facts

- Real practice sessions of amateur pianists (not studio recordings)
- Video + audio + MIDI + hand landmarks + fingering labels + metadata
- Benchmarks for audio-only and audio-visual piano transcription

## When this paper is relevant

- You train or benchmark audio-visual piano transcription / 오디오-비주얼 피아노 전사(transcription) 모델을 학습·평가할 때
- You extract piano fingering automatically from video hand-pose estimation / 영상 기반 손 포즈 추정으로 피아노 운지를 자동 추출하려 할 때
- You need piano data from realistic practice conditions rather than studio recordings / 스튜디오 녹음이 아닌 실제 연습 환경의 피아노 데이터가 필요할 때
- You deal with synchronising and aligning video, audio and MIDI / 여러 모달리티(영상·오디오·MIDI)의 동기화·정렬 문제를 다룰 때

## Related

- SKY-Piano: A Multimodal Piano Performance Dataset (2026): https://joonhyungbae.com/papers/sky-piano-2026.md
- Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation (2025): https://joonhyungbae.com/papers/two-web-toolkits-2025.md
- PiAnnotate: A Web Annotation Tool for Piano Fingering, with a Diagnostic Probe (2026): https://joonhyungbae.com/papers/piannotate-2026.md

## How to cite

Y. Kim, J. Park, Joonhyung Bae, K. Kim, T. Kwon, A. Lerch, J. Nam (2025). PianoVAM: A multimodal piano performance dataset. Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), Daejeon, Korea, Sep. 21–25, 2025. https://arxiv.org/abs/2509.08800

```bibtex
@inproceedings{Kim2025pianovam,
  title = {PianoVAM: A multimodal piano performance dataset},
  author = {Y. Kim and J. Park and Joonhyung Bae and K. Kim and T. Kwon and A. Lerch and J. Nam},
  booktitle = {Proc. 26th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), Daejeon, Korea, Sep. 21–25, 2025},
  year = {2025},
  eprint = {2509.08800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.08800}
}
```