Visual cues are essential for synchronization in ensemble performances, especially at the start of a piece or during fermatas with sudden tempo changes, where gestures facilitate coordination. However, in human-automated piano ensembles, traditional gesture cues are inapplicable due to the absence of a pianist. To address this, we propose a real-time gesture cue detection and visual feedback system which is structured into two main components: (1) the development of an accompaniment system that detects flutists’ gesture cues and predicts onset timings for initiating playback, and (2) the provision of real-time visual feedback regarding cue detection status, enabling performers to synchronize with the system effectively. The bidirectional gesture cue detection and visual feedback system was empirically validated in live performances with professional musicians, demonstrating its capability to maintain a natural musical flow. This approach surpasses the limitations of previous audio-based accompaniment systems by incorporating gesture cues at critical moments, such as the start of pieces and fermata passages, enabling optimized reactive accompaniment.