It’s not impossible, been running my own email server for about 10 years and I inbox pretty much everywhere. I even emailed my work address and straight to inbox. I do have the full SPF, DKIM and DMARC stuff set up, for which I get notices from several email provides of failed spoof attempts.
Takes a while and effort to gain that reputation, but it’s doable. And OVH’s IPs don’t exactly have a great reputation either. Once you’re delisted from most spam databases / old spam reputation is expired, it’s not that bad.
Although I do agree it’s possibly one of the hardest services to self host. The software to run email servers is ancient and weird, and takes a lot to set up right. If you get it wrong you relay spam and start over, it’s rough.
Is it directly exposed over the Internet? If you only port forward the VPN on your router, I wouldn’t worry about it unless you’re worried about someone else already on your LAN.
And even then, it’s really more like an extra layer of security against accidentally running something exposed publicly that you didn’t intend to, or maybe you want some services to only be accessible via a particular private interface. You don’t need a firewall if you have nothing to filter in the first place.
A machine without a firewall that doesn’t have any open port behave practically the same from a security standpoint: nothing’s gonna happen. The only difference is the port showing as closed vs filtered in nmap, and the server refusing to send any response not even a rejection, but that’s it.