mx13 Yes, same thing with pipewire, you can't uninstall it practically.
Although I don't know how to solve your problem, I find your post *interesting.
I gave this a thought and it's unfortunate, I don't have the AV dolby multichannel, whatnot, amplifier to test this problem myself. Everything else I think I got: Solus/PipeWire, audio interface with s/pdif output and enough speakers (testing 5.1 surround, two of them would be guitar speakers, but still).
I've always preferred (good, hi-fi) stereo sound over dolby etc. multichannel. But then again it's all music for me, no gaming. I can imagine better experience with multichannel audio system, when gaming, even being essential!
Best of luck to you figuring this out.
*I've had my share of pipewire (I think) problems, see here:
https://discuss.getsol.us/d/10423-pipewire-pro-audio