I enabled Advanced Player Tracking with DUMP and Notification Listener permission, and it seems like this solved the issue for me. I use Lossless for downloads and Hi-res Lossless for WiFi streams, and enabled Spatial Audio, for now I don't have the problem.