It's not that difficult. First of all MP4, MKV, AVI, WebM, VOB[...] are container formats that stores the video und audio streams, subtitles etc.
Different containers support different video codecs such as AVC/h.264, HEVC/h.265, VP8, VP9, AV1, MPEG-2, XviD.
For example x264 is an software encoder for h.264 and x265 is an software endcoder for h.265.
To check whats inside a container simply user the tool called Mediainfo. It will give you all important information about bitrate, framerate, resolution, codec etc.
sudo eopkg install mediainfo
Just open a video file with the GUI and switch to text view or use it with the terminal
mediainfo [path videofile]
The reason your output video got a bigger size then your source video is because it was already encoded with a pretty high quantizer. A high quantizer means high compression. So to shrink it even more you need an unusal high quantization. x264 and x265 control this with the constant(quality) rate factor(CRF) in Handbrake. A higher CRF will shrink the video more and of course decreases the quality. The Bitrate is the only thing that affects the size of an video.
If your video is 50Mb and has a duration of 5 minutes it's bitrate will be ~ 1300kbps(minus bitrate for the audio stream) which is already pretty low. For example DVDs usually have an average video bitrate of ~ 6000kbps. (but use a very very outdated codec)
With a modern codec like h.265 or VP9 or AV1 it could be possible to reduce the size about 50% (depents on the source and it's amount of grain, noise and motion) but shrinking it to 10Mb will definitely destroy the picture quality way too hard.
To shrink an video to a preferred file size I would recommend to use one or better multipass encoding with an fixed average bitrate. (can be done in Handbrake too).
Another very good video editing and transcoding tool is Avidemux.
