# Real-time flutist gesture cue detection system for auto-accompaniment

- Authors: J. Choi, T. Kwon, Joonhyung Bae, J. Park, Y. Kim, J. Nam
- Venue: Proc. 25th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), San Francisco, CA, USA, Nov. 10–14, 2024
- Year: 2024
- Type: Abstract/Demo
- PDF: https://drive.google.com/file/d/1wf9jb7gRIKpsvgVEuOpnT9jkEBQ7cv1C/view?usp=drive_link
- Keywords: real-time detection, flutist gesture, auto-accompaniment, music technology, gesture recognition
- Page: https://joonhyungbae.com/publication/flutist-gesture-cue-2024

## In one sentence

Detects flutists' gesture cues in real time to start/resume auto-accompaniment and gives visual feedback on detection status — a bidirectional human–computer ensemble cue system.

플루트 연주자의 몸짓 큐를 실시간 검출해 자동 반주를 시작·재개하고, 검출 상태를 시각 피드백으로 돌려주는 양방향 시스템.

## Abstract

Visual cues are essential for synchronization in ensemble performances, especially at the start of a piece or during fermatas with sudden tempo changes, where gestures facilitate coordination. However, in human-computer ensembles, traditional gesture cues are inapplicable due to the absence of a pianist. To address this, we propose a real-time gesture cue detection and visual feedback system which is structured into two main components: (1) the development of an accompaniment system that detects flutists' gesture cues and predicts onset timings for initiating playback, and (2) the provision of real-time visual feedback regarding cue detection status, enabling the human performer to recognize the status of the system. The bidirectional gesture cue detection and visual feedback system was empirically validated in live performances with professional musicians, demonstrating its capability to maintain a natural musical flow. This approach overcomes the limitations of previous audio-based accompaniment systems by incorporating gesture cues at critical moments.

(Source: ISMIR 2024 Late-Breaking Demo (author PDF))

## When this paper is relevant

- You work on auto-accompaniment or human–computer ensemble synchronization / 자동 반주(auto-accompaniment)·인간-컴퓨터 앙상블의 동기화를 연구할 때
- You need to recognise performers' visual gesture cues (piece starts, fermatas) / 연주자의 시각적 몸짓 큐(페르마타, 곡 시작)를 인식해야 할 때

## How to cite

J. Choi, T. Kwon, Joonhyung Bae, J. Park, Y. Kim, J. Nam (2024). Real-time flutist gesture cue detection system for auto-accompaniment. Proc. 25th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), San Francisco, CA, USA, Nov. 10–14, 2024. https://drive.google.com/file/d/1wf9jb7gRIKpsvgVEuOpnT9jkEBQ7cv1C/view?usp=drive_link

```bibtex
@inproceedings{Choi2024flutist,
  title = {Real-time flutist gesture cue detection system for auto-accompaniment},
  author = {J. Choi and T. Kwon and Joonhyung Bae and J. Park and Y. Kim and J. Nam},
  booktitle = {Proc. 25th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), San Francisco, CA, USA, Nov. 10–14, 2024},
  year = {2024},
  url = {https://drive.google.com/file/d/1wf9jb7gRIKpsvgVEuOpnT9jkEBQ7cv1C/view?usp=drive_link}
}
```