Odd, it says right on the HP website it is made for windows or linux.
Have you tried a different USB port? Also in some BIOS' you have to manually select the EFI file to boot from.
I also found this: https://h30434.www3.hp.com/t5/Desktop-Operating-Systems-and-Recovery/How-to-install-Ubuntu-Linux-on-HP-Z2-Mini-G4-Workstation/td-p/6953180/page/2
All below is from the post, not mine, for some reason I cannot put it in italics..
_I have now installed Ubuntu 18.4 Server on my Z2 Mini G4 and it works just great, these are the steps I took.
1) Download Ubuntu 18.4 ISO. Use the alternative download from here:
http://cdimage.ubuntu.com/releases/18.04.3/release/ubuntu-18.04.3-server-amd64.iso
Do NOT use the 'live' ISO
2) Use rufus version 3.8 to create a bootable USB
a) Select the downloaded ISO as the disk image
b) Change the partition type to NTFS
c) When prompted choose DD image as the image type (NOT ISO).
3) Change the Z2 Mini Bios Settings
a) Turn fast boot OFF
b) USB bootable storage ON
c) Set Legacy boot disable Secure boot disable
c) Enable USB in UEFI boot list
4) Power off Z2 Mini
5) Power on Z2 Mini into startup menu (esc at splash screen)
6) Select Boot options. Your bootable USB should now be in the list
7) Select your USB drive
π Follow the Ubuntu installation instructions.
NOTE: This is not dual boot so your original windows will be destroyed if you install to the primary SSD!!!
If anyone is interested I have also installed the original Windows 10 Pro and macOS Mojave as virtual machines running under Ubuntu using QEMU/KVM and both work nicely._