The reality is that unless you install a service like openssh-server etc then there is nothing on your system listening on a port to allow authorised access and so a firewall is rather pointless.
If you do run services like openssh-server then you can allow access to only IP's you intend to connect from on a specific port.
Since deny incoming, allow outgoing is the default on ufw and you said you have no other rules that implies you are not using any other services such as openssh-server because without additional rules opening a hole it would not be able to accept any connections, which means you might as well uninstall the firewall.
