Music mastering style transfer involves applying the mastering-related audio features of a reference track to another, simulating the professional mastering process that enhances overall sound quality. In this paper, we propose the ITO-Master framework, which introduces Inference Time Optimization (ITO) on reference embeddings to improve the mastering style transfer process. Our approach achieves effective automatic mastering and gives users flexibility, enabling them to adapt the system to their preferences by adjusting the reference song or specific audio effects traits. We explore both black-box and differentiable methods, demonstrating that ITO improves performance on key metrics. The framework provides flexible, user-driven mastering style transfer with an interactive demo available on our demo page: https://tinyurl.com/ITO-Master.