SKY-Piano: A Multimodal Piano Performance Dataset

In one sentence

An 11-hour multimodal piano dataset from 7 professional and 12 amateur pianists with hand and body motion capture, multi-view video, audio, MIDI and MusicXML scores.

Abstract

Music information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.

Source: arXiv:2607.27296

Key facts

• 11 hours; 7 professional + 12 amateur pianists; shared core repertoire
• Hand + body motion, released in both flagged (occlusion-marked) and imputed form, plus Visual3D body-segment kinematics
• Includes pseudo fingering annotations, an interactive web browser, and a MIDI-to-motion fine-tuning baseline

When this paper is relevant

• You need a piano performance dataset with finger-level motion capture
• You compare performers of different expertise (professional vs. amateur) on shared repertoire
• You train or evaluate MIDI-to-motion or audio-to-motion generation
• You need both raw (occlusion-flagged) and gap-filled motion data
• You study performance analysis or pedagogy with score–performance–motion alignment

Related

• PianoVAM: A multimodal piano performance dataset (2025)
• PiAnnotate: A Web Annotation Tool for Piano Fingering, with a Diagnostic Probe (2026)
• Pianoid: 3D Pianist Body Motion Generation from MIDI (2026)
• Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation (2025)

Cite

Joonhyung Bae, D. Park, T. Kwon, Y.S. Choi, H. Hur, S. Obata, S. Kai, Y. Wada, Y. Takahashi, A. Maezawa, J. Park, J. Park, J. Nam (2026). SKY-Piano: A Multimodal Piano Performance Dataset. Proc. 27th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2026. https://arxiv.org/abs/2607.27296

@inproceedings{Bae2026sky,
  title = {SKY-Piano: A Multimodal Piano Performance Dataset},
  author = {Joonhyung Bae and D. Park and T. Kwon and Y.S. Choi and H. Hur and S. Obata and S. Kai and Y. Wada and Y. Takahashi and A. Maezawa and J. Park and J. Park and J. Nam},
  booktitle = {Proc. 27th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2026},
  year = {2026},
  eprint = {2607.27296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.27296}
}

Machine-readable copy: /papers/sky-piano-2026.md