Real-time flutist gesture cue detection system for auto-accompaniment

In one sentence

Detects flutists' gesture cues in real time to start/resume auto-accompaniment and gives visual feedback on detection status — a bidirectional human–computer ensemble cue system.

Abstract

Visual cues are essential for synchronization in ensemble performances, especially at the start of a piece or during fermatas with sudden tempo changes, where gestures facilitate coordination. However, in human-computer ensembles, traditional gesture cues are inapplicable due to the absence of a pianist. To address this, we propose a real-time gesture cue detection and visual feedback system which is structured into two main components: (1) the development of an accompaniment system that detects flutists' gesture cues and predicts onset timings for initiating playback, and (2) the provision of real-time visual feedback regarding cue detection status, enabling the human performer to recognize the status of the system. The bidirectional gesture cue detection and visual feedback system was empirically validated in live performances with professional musicians, demonstrating its capability to maintain a natural musical flow. This approach overcomes the limitations of previous audio-based accompaniment systems by incorporating gesture cues at critical moments.

Source: ISMIR 2024 Late-Breaking Demo (author PDF)

When this paper is relevant

• You work on auto-accompaniment or human–computer ensemble synchronization
• You need to recognise performers' visual gesture cues (piece starts, fermatas)

Cite

J. Choi, T. Kwon, Joonhyung Bae, J. Park, Y. Kim, J. Nam (2024). Real-time flutist gesture cue detection system for auto-accompaniment. Proc. 25th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), San Francisco, CA, USA, Nov. 10–14, 2024. https://drive.google.com/file/d/1wf9jb7gRIKpsvgVEuOpnT9jkEBQ7cv1C/view?usp=drive_link

@inproceedings{Choi2024flutist,
  title = {Real-time flutist gesture cue detection system for auto-accompaniment},
  author = {J. Choi and T. Kwon and Joonhyung Bae and J. Park and Y. Kim and J. Nam},
  booktitle = {Proc. 25th Int. Soc. for Music Inf. Retrieval Conf. (ISMIR), San Francisco, CA, USA, Nov. 10–14, 2024},
  year = {2024},
  url = {https://drive.google.com/file/d/1wf9jb7gRIKpsvgVEuOpnT9jkEBQ7cv1C/view?usp=drive_link}
}

Machine-readable copy: /papers/flutist-gesture-cue-2024.md