Mastodon server: email
Always a hassle to get mail delivery to work.
Had a similar problem with a VoIP (Nexmo SMS/call forwarding) tool that just refused to work using local mail servers without a valid cert. Gave up and started using Mailgun.
Long story short: use something like Mailgun or another provider.
Using localhost SMTP server support seems to be limited if you don't have working certs. The documentation is also lacking as to what does what. Didn't figure out how to have it ignore SSL.