OK, so after a little bit of trouble shooting I think I found the problem. It's something to do with my headset or the way Budgie handles SPDIF, because when I plugged my headset's base box in by USB and selected that, the windup was gone and I could hear the full audio. Is there something with how Budgie uses SPDIF, by chance? I think it might be that, or how my headset base is handling it and I'm trying to eliminate possibilities now.
Weird thing, though is that when it isn't plugged in via USB, then the option for Digital Output - SPDIF is gone and I only have SPDIF Output - USB Audio. With the second one selected, the warm up is there but with the first one it isn't.