This paper received generally good ratings from the four reviewers, as the proposed model achieves high performance while working in real time, which has been less studied. The main weakness is the lack of clarity in the model descriptions. In particular, reviewer #1 pointed out a lot of missing details that make it difficult to understand the exact model operation, and thus recommended a weak rejection. Through discussion, the reviewers concluded that many of the issues can be addressed in the camera-ready version and that the main contributions of the paper deserve to be presented at the ISMIR conference. Therefore, the authors are strongly encouraged to incorporate all comments in the revision and improve the clarity.

This paper proposes a streaming-capable automatic piano transcription method. Automatic piano transcription is a well-known task with a somewhat standardized evaluation method used since the Onsets and Frames paper (Hawthorne 2018). To make the model streaming-capable, the authors improve upon the sequence-to-sequence piano transcription model (Hawthorne 2021) so that rather than generating a full sequence of MIDI-like tokens, the decoder concerns onset or offset events in a single time frame only, at each time. The onset decoder autoregressively predicts the sequence of note onsets, and the offset decoder takes the set of active onsets and non-autoregressively predicts if any of them reached the offset.

The results in Table 1-2 show that the proposed method performs competitively with other SOTA models, while achieving 380 milliseconds of streaming latency, which I consider a strong result. While real-time / streaming capabilities are often overlooked in academic settings, a streaming-capable model can have a huge impact on the usability and interactive applications.

The proposed method section could be improved a lot to have clearer descriptions of the model:

• some details of the model architecture are missing, such as how the encoder features are fed to the decoders; I could infer from Figure 2 that it's using cross attention (aka encoder-decoder attention) on the features with a causal mask to enable streaming, but this should have been explained in the main body of the text.
• In Figure 2, each decoder layer seems to have layernorms in both the beginning and the end of the block, which is unconventional. (For context, the original transformer paper had layernorms after attention and mlp, but most implementations including T5 put the layernorms before them).
• There should be a more detailed description on the sustain pedal detection, which in the title. I'm guessing from the output vocabulary that the pedal states are predicted as part of the onset decoder's prediction, but I can't tell if the tokens represent the states (presence or absence, as written in Section 4.1.3) or the state changes (press / release).
• In that sense, it'd be helpful to include some example output sequences, to clarify if there is a preferred order of the tokens, e.g. BOS, any pedal tokens followed by notes lower to higher, and then EOS.
• The offset decoder is described to be operating non-autoregressively, which makes sense because it just needs to determine if each active onset note ended or not at each frame, but it's easy to (mis?)understand from the paper that offset decoder also does sequence prediction, from the notations in Section 3.1 and 3.3.
• I find the notation in Algorithm 1 is a little inconsistent. I have to guess that the superscript notations like 1:k_1 and 0:n_1 denote the variable-length onset and offset sequences in each time frame, but it's unclear why Y_t^{0:k_t} was computed but only Y_t^{1:k_t} is outputted. Maybe it is to exclude EOS, or the pedal token? In any case, a caption to the algorithms block would be nice, to provide a high-level overview of the algorithm explaining the overall flow. Lines 19-26 is pretty standard greedy LM decoding, so it could become a single line.
• Given all the intricacies like the above, an open-source code and model release would be able to foster faster and wider adoption of the streaming piano transcription model.

Given that we have more space, further analyses like the following would have made it a stronger paper.

• A piano roll representation of the predicted MIDI or the "posteriorgrams" could be useful, space permitting, to show a qualitative visualization of the model behavior.
• Latency analysis: while 380ms is the systematic latency of the streaming architecture, the real-world latency would depend on the time between the actual and predicted onsets/offset events. A histogram of those latencies would be useful for understanding the real-world latency and its variability.
• A bonus point if a video of a piano performance getting transcribed in real time, showing the player and animated piano roll representation on the screen.

To conclude, I really like the strong results and would love to see the paper accepted and the model becoming available for streaming piano transcription, but as above, I find the overall quality of the text to be below ISMIR's acceptance bar.