I use either BT speaker or headphones when I play audio and I'm always having to open this up to turn off these two devices at every reboot and every time the screen turns off, it's annoying to say the least, sometimes even if I'm not playing any sound it defaults back to one of them, extremely annoying when that happens.

I used to have this as well back when I was still running Plasma. Apparently Plasma Settings doesn't work nicely with Pulseaudio settings.
My notes say I fixed* it by doing the following:

Turn off both 'switch on port' and 'switch on connect' plugins of PulseAudio to disable this feature. Probably KDE forces a default even though the device is not default or turned off in PulseAudio. We need to create a new user config file, which will override the default system file. This is needed as the system file will be overridden upon an update:

  • Create the following directory if it doesn't already exist: /etc/pulse
  • Copy the default.pa file from /usr/share/pulseaudio to /etc/pulse
  • In /etc/pulse/default.pa comment line load-module module-switch-on-port-available
  • In /etc/pulse/default.pa comment line load-module module-switch-on-connect
  • After any changes, restart with pulseaudio -k followed by pulseaudio --start or reboot
  • Not really a fix, as it essentially turns off the automatic device switching, so it will not change to another device unless you tell it to. So if you really love such a feature, this fix will still annoy you.