This paper prompted a lively discussion amongst the reviewers, ultimately reaching a consensus that it should be accepted.

We ask that the authors make minor revisions to better-motivate the use of a query-based system (as opposed to instrument labels). The advantages of the query-based approach are a bit unclear, given that the paper does not study generalization to instruments unseen during training (which would obviously preclude a label-based approach).

Please also incorporate the missing references identified in the reviews.

Finally, the comments about Moises around Line 85 could be interpreted as advertisement, which has no place in an academic paper. Please consider toning down this discussion of Moises, or adding commentary on other >4 stem commercial systems (e.g., Lalal.ai, audioshake).

This paper presents a system called Banquet, which aims to separate musical audio into different parts (stems) using a single decoder. The system is designed to handle more than the usual four stems (vocals, drums, bass, and other) and uses a query-based approach to identify which instrument to separate. The strategy of using one decoder for multiple stems is useful because it simplifies the model and reduces computational complexity. The system integrates a bandsplit source separation model with a query-based setup and a music instrument recognition model (PaSST). The results are competitive wrt. the SOTA HT-Demucs, even though they only exceed its SNR slightly for guitar and piano.

The authors show that Banquet can perform well on common instrument types like vocals, drums, and bass, and even some less common ones like guitar and piano using the MoisesDB dataset. This suggests that their approach has potential in broader applications. I appreciate that the authors aim to push beyond the standard four-stem setup and address the need for more flexible source separation systems.

The paper is well-written and well structured, has extensive discussion and a valid scientific approach.

My main concern is that the authors don't motivate clearly their use of a query-based approach. At train time, they use instrument labels to retrieve queries from other songs. Therefore, a naive approach to simply use the class labels as conditioning (with some embedding layer) could work equally well. Apparently, the authors also didn't understand that conditioning on queries from different tracks defies the idea of a query-based approach, as they note in the discussion "Interestingly, it appears that querying with excerpts from the same or different track did not affect the model performance for most stems except for electric piano." However, it is expected that the model cannot do better with same-song queries, as it was trained on all-song queries using instrument labels.

More detailed remarks:

• The performance on long-tail instruments (less common instruments) is relatively weak suggesting that the model couldn't generalize by interpolating in the PaSSt space. This puts into question the query-based approach.
• The model’s training approach involves using queries from the same stem type across different songs. I feel this undermines the self-supervised potential of the method because it relies on knowing the instrument class. It seems like a missed opportunity to fully explore the benefits of self-supervised learning. When using queries in a supervised setting, one would at least need to compare to a non-query based approach (i.e., conditioning on the instrument label) to show that there are advantages.
• The authors missed some important references for query-based approaches, which makes it harder to place their work in the context of existing research. It is generally difficult to locate the work within other works.
• Single decoder approaches already existed. I am not sure what the new contribution is in this paper, besides using the not-yet-used MoisesDB dataset. It would be nice if the authors would have provide a better motivation and explanation to show what makes this work different or better compared to others.

Minor remarks:

• The use of the term "stem" is confusing. I think, "stem type" or "instrument type/class" would work better.
• The training data for long-tail instruments is very limited. The authors could have considered more aggressive data augmentation and transfer learning techniques to help improve the model’s performance on less common instruments. Another option to achieve more training examples would be to randomly mix stems of the dataset, as it is common in many SOTA source-separation works.
• The paper could benefit from more detailed comparisons with existing query-based and single-decoder systems to clearly show improvements or differences. This would help in understanding the real contributions of this work.

Conclusion:

The paper proposes an interesting system for music source separation using a single decoder and a query-based approach. I appreciate the authors' effort to create a more flexible system that can handle more than just the standard four stems. However, it has some weaknesses, especially it doesn't motivate the query-based approach and doesn't show that it improves over simply using class labels. The contribution of this work is unclear given existing single decoder and query-based approaches.

Consequently, I don't think the paper is ready for publication in ISMIR 2024.