I still cant telnet into my ports of 110 and 25 but I was able to get greylisting off by doing this apt-get autoremove postfix-cluebringer from the system then restart the server.
I think you are definetely mixing things...
Greylist is a feature of policyd (on Ubuntu I think it's cluebringer) and it has absolutely nothing to do with the SSL/noSSL ports.
It's purpose is to temporarily refuse messages (for a few minutes) who haven't been authenticated by the triplet IP/sender/recipient.
It's a very good antispam measure, as trusted and real mail servers will resend the message after a few minutes, as spammers won't (spam is in majority sent by botnets -different IPs- and/or softwares who won't bother resend a message after a refusal).
For the ports problems you should follow my points 2 and 3. I repeat, this has nothing to do with greylist.
PS: I don't know if it's a good idea to remove policyd/cluebringer... I think that postfix expects to find the filter, and I'm not sure that things will run smoothly if it won't find it at all...