# SKY-Piano: A Multimodal Piano Performance Dataset

- Authors: Joonhyung Bae, D. Park, T. Kwon, Y.S. Choi, H. Hur, S. Obata, S. Kai, Y. Wada, Y. Takahashi, A. Maezawa, J. Park, J. Park, J. Nam
- Venue: Proc. 27th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2026
- Year: 2026 (Accepted)
- Type: Conference Paper
- arXiv: https://arxiv.org/abs/2607.27296
- PDF: https://arxiv.org/pdf/2607.27296
- Keywords: dataset, multimodal, piano performance, motion capture, MIDI
- Page: https://joonhyungbae.com/publication/sky-piano-2026

## In one sentence

An 11-hour multimodal piano dataset from 7 professional and 12 amateur pianists with hand and body motion capture, multi-view video, audio, MIDI and MusicXML scores.

프로 7명·아마추어 12명의 11시간 피아노 연주를 손·전신 모션캡처, 다시점 영상, 오디오, MIDI, MusicXML 악보로 함께 담은 멀티모달 데이터셋.

## Abstract

Music information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.

(Source: arXiv:2607.27296)

## Key facts

- 11 hours; 7 professional + 12 amateur pianists; shared core repertoire
- Hand + body motion, released in both flagged (occlusion-marked) and imputed form, plus Visual3D body-segment kinematics
- Includes pseudo fingering annotations, an interactive web browser, and a MIDI-to-motion fine-tuning baseline

## When this paper is relevant

- You need a piano performance dataset with finger-level motion capture / 손가락 단위 모션캡처가 포함된 피아노 연주 데이터셋이 필요할 때
- You compare performers of different expertise (professional vs. amateur) on shared repertoire / 같은 곡을 숙련도가 다른 연주자들이 친 비교 데이터(프로 vs 아마추어)가 필요할 때
- You train or evaluate MIDI-to-motion or audio-to-motion generation / MIDI→모션, 오디오→모션 생성 모델을 학습·평가할 때
- You need both raw (occlusion-flagged) and gap-filled motion data / 마커 가림(occlusion) 결측을 표시한 원본과 보간본이 모두 필요할 때
- You study performance analysis or pedagogy with score–performance–motion alignment / 악보(MusicXML)–연주(MIDI)–동작을 정렬한 데이터로 연주 분석·교육 연구를 할 때

## Related

- PianoVAM: A multimodal piano performance dataset (2025): https://joonhyungbae.com/papers/pianovam-2025.md
- PiAnnotate: A Web Annotation Tool for Piano Fingering, with a Diagnostic Probe (2026): https://joonhyungbae.com/papers/piannotate-2026.md
- Pianoid: 3D Pianist Body Motion Generation from MIDI (2026): https://joonhyungbae.com/papers/pianoid-2026.md
- Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation (2025): https://joonhyungbae.com/papers/two-web-toolkits-2025.md

## How to cite

Joonhyung Bae, D. Park, T. Kwon, Y.S. Choi, H. Hur, S. Obata, S. Kai, Y. Wada, Y. Takahashi, A. Maezawa, J. Park, J. Park, J. Nam (2026). SKY-Piano: A Multimodal Piano Performance Dataset. Proc. 27th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2026. https://arxiv.org/abs/2607.27296

```bibtex
@inproceedings{Bae2026sky,
  title = {SKY-Piano: A Multimodal Piano Performance Dataset},
  author = {Joonhyung Bae and D. Park and T. Kwon and Y.S. Choi and H. Hur and S. Obata and S. Kai and Y. Wada and Y. Takahashi and A. Maezawa and J. Park and J. Park and J. Nam},
  booktitle = {Proc. 27th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), 2026},
  year = {2026},
  eprint = {2607.27296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.27296}
}
```