Thank you for looking into it!
Could you check your CPU utilization when you get significantly low fps? Because on Solus my CPU went to almost 100% when the fps dropped vs. 50% average on Windows (with spikes to 70-80% on huge fights).
Your RX580 should be capable to run the game on 60+ fps with most settings maxed out.
One thing that hits hard are shadows. Turning them off gives you a huge fps increase in large battles.
I tried the lowest available graphics settings with shadows turned off, but still horrible drops on medium+ fights with extremely high CPU usage.