OK, I cracked it 
It was in fact simpler than I thought if I knew where to dig...
This was a DNS issue - iRedMail is in one VM and Mautic is in a separate VM. Both VMs are on the same virtula LAN - connected to the same vSwitch. So, when Mautic was trying to connect to the SMTP server it was bumping it on the private IP address bypassing the firewall and because of that there was no TLS encryption enforced.
Editing host file on the application server and adding a reference to the public IP address of the mail server (as mapped on the firewall) allowed me to connect to iRedMail and set up the profile.
So, everything works.... except I have a new issue now. I guess I will need to open a separate post for it.