The Mechanics of Email

Sunday, September 30, 2007

I was leafing though some forum archives the other day and found that not being able to send email properly through SMTP was quite popular. Some email providers do not allow usage of their SMTP servers for mass mailing, and others have limits. Still others require complicated authentication processes. Understanding SMTP will help you get past these restrictions painlessly.

Traditionally, mail servers have been used to deliver mail to members of an organization. When a message arrives at the mail server, it parses the headers and routes the mail to the right person. Mail servers have also been used to handle outgoing mail from within the organization. The server forwards each bit of outgoing mail to its intended organization, letting the destination server route the mail internally.

Let’s say you’re a member of Yahoo.com and your friend is a member of Gmail.com. When you send an email to your friend, this is [a simplification of] how the message is passed:

message => Yahoo.com => Yahoo.com MX => Gmail.com MX => your friend’s account

An MX host takes care of the mail exchange of an organization. Generally, if you want to send email, you’d pass your message to the MX host using a language called the Simple Mail Transfer Protocol (SMTP). (If you send email through your browser, the server-side program handling your data input takes care of the SMTP.)

Next, Yahoo.com’s MX server sees that it’s meant for someone on Gmail.com, so it passes the data off to Gmail.com’s MX server, again using SMTP.

Gmail.com’s MX server, upon receiving the message, passes it off to a mechanism that distributes mail to its users.

In some cases, though, neither the sender nor the recipient of the message is physically in the organization. For example, Yahoo.com’s CEO Jerry Yang is on a business trip and wants to send an email to Larry Page (he works at Google, rofl). He would then dig out his Outlook Express (or a similar email program), connect to Yahoo.com’s MX host, and pass the message using SMTP. Or he would open his Web browser to mail.yahoo.com. Same thing.

As you’d imagine, Yahoo.com would not let just anyone send email to anywhere. It has to first make sure that Jerry Yang is actually a member. Hence, he has to authenticate himself and whatnot. And it would not let anyone send unlimited amounts of mail, because that’s how Internet spam happens. So Jerry Yang authenticates himself and his message goes on its merry way, using Yahoo.com’s MX host as a relay station.

Now, the way we set up our script to send email is this. First, it connects to the SMTP server we want it to use, sending along with all sorts of credentials. Then it tells the server the return address, followed by a bunch of recipient addresses. Finally, it pushes through the message body with headers and whatnot. Using SMTP, of course. So in effect, we’re using our “SMTP server” to relay mail to all our recipients who may or may not be internal to the SMTP server’s organization. Hence, the restrictions.

However, we do know that an MX host will accept mail for members of its organization. So all we have to do is group the recipients by organization (email provider) and deliver their mail directly to their organizations’ MX hosts. Here’s the procedure:

  1. split the email address (username@hostname) into username and host.
  2. perform a DNS query on the host, and extract all the MX hosts. if there is none, just use the original host.
  3. dispatch the email, using the MX host as the “SMTP server”. if it fails, use the next MX host on file.

2 Comments

  1. Carlo Mario says:

    Interesting article. I got 2 questions: Does it work if you have port 25 blocked by your ISP? and: Is there any program out there that can help me achieve sending this way?

    Thank you!

  2. Jiang Yio says:

    For outgoing mail, your ISP must allow remote connections to port 25, and this is most likely allowed. For incoming mail (i.e. you’ve got your own SMTP server processing your mail), your ISP must allow incoming connections on your local port 25.

    Most people interested in this method are involved in writing email-sending scripts, so I’ll assume that you are too. It’s actually almost the same as relaying email through your own SMTP server; this time, you find out what the remote SMTP server is and use that instead.

    Incidentally, this allows you to send email even if you don’t have an email account or inbox.