TheEmailAuthority

TTL in DNS part 3

2012-12-18T16:42:00.002-05:00

When we last talked in part 2, we were discussing what TTL is not (it is not how long the name server takes to update) and what it is (it is how long a computer takes to update it's own, local, cache).

We ended with an object lesson where www.TheEmailAuthority.com has a 1 hour TTL on it's A record, and I change the A record and shut down the server at my old IP promptly at 3 PM.

User1 was just reading my site at 2:30, so when they go to load another page anytime after 3 and before their own, local, cached record expires at 3:30 they won't be able to load the page.

However, User2 was lucky enough to request my A record at 3:10, so they've got twenty full minutes of Email Authority goodness while User1 is still just reading an error screen.

We've now seen how two users each with their own cache and their own TTL respond to DNS changes. This simple example only used one record type, and only one "hop" between the client and the server. Let's take this to the next level and try a more complex, email related, change.

Domain1.com is changing their email server, and their online ordering system sends them emails every time someone places an order. Because of this, they cannot afford to have any down time. We'll presume that they're open 24/7/365 so late-night and weekend migrations are out of the question. Take these three examples into consideration, presuming the following zone file:

domain1.com. 86400 IN NS ns1.domain1s-ISP.com.
domain1.com. 86400 IN NS ns2.domain1s-ISP.com.

domain1.com. 14400 IN MX mail.domain1.com.

mail.domain1.com. 3600 IN A 10.11.12.13.

Example 1: Change the MX record

Domain1.com's email administrator sets up mail2.domain1.com and places it into the MX records. He sends a test email from his gmail account, and it routes correctly. However, the online ordering emails are not coming through to mail2.domain1.com.

This is because the online ordering server still has his old MX records cached, and is sending the order emails to the old server until it updates it's cache sometime in the next 4 hours. It could be less than that depending on when the online ordering server last cached the records, but there is still an outage and the admin loses his job.

Moral of this story: MX record changes aren't good enough.

Example 2: Set the TTL to a low value and change the IP of the A record

The next admin understands TTL, and logs in to his DNS interface to change the TTL in the morning before the migration that evening. He sets the A record's TTL to 1 minute in the morning, and then changes the A record's IP address to point to the new server

This way the cached MX records include the same A record, and the short TTL means that the connecting mail servers *should* update the IP every minute.

Unfortunately, the owner's wife uses an AOL address, and they cache records for their own pre-set value (24+ hours), so since the owner's wife's email wasn't coming through the new admin got fired too. He at least got a severance package...

Example 3: Set the original server to relay mail to the new server

The final admin understands that things don't always go according to plan, and leaves the old server online for several days after the migration. He sets the old server to act as a mail relay though, so it's not delivering locally to any of the users (and thereby interrupting mail flow). He waits until no legitimate email is seen in his logs for a day or so and then takes the server offline.

This admin also could have used a mail relay service such as a hosted anti-spam service to act as the relay for extra points :-)

Well, that was a much longer look into TTL than anyone would ever want to know. Hopefully you enjoyed it, and can now impress your friends (or put them to sleep) with your new-found TTL skills.

-TEA

TTL in DNS part 2

2012-12-07T14:18:00.000-05:00

In part 1 of this series, we discussed what TTL is, and provided a few exercises to help strengthen your understanding of TTL. We'll pick up where we left off with our same DNS zone for domain1.com:

domain1.com. 86400 IN NS ns1.domain1s-ISP.com.
domain1.com. 86400 IN NS ns2.domain1s-ISP.com.

domain1.com. 14400 IN MX mail.domain1.com.

mail.domain1.com. 3600 IN A 10.11.12.13.

As we discussed, the NS records will expire 1 day after they were last cached, the MX records will expire 4 hours after they were last cached, and the A record will expire 60 minutes after it was last cached. What does this mean for you and your planned migration?

A common misconception about TTL is that it's the amount of time the Name Server requires to update the record. This is absolutely not the case. Do you remember what RFC 1034 said a TTL is? A TTL is:

...primarily used by resolvers when they cache RRs. The TTL describes how long a RR can be cached before it should be* discarded.

Where in there does it say that a TTL is how long a name server takes to update? What does it say? It says that a DNS resolver (which is anything that resolves DNS, your computer has a DNS resolver in it) caches the records, and the TTL describes how long the DNS resolver should cache the record before discarding it (and then re-requesting it).

What this means is that (for the sake of simplicity) every computer connected to the internet has it's own local cache of websites it's visited, mail servers it's talked to, etc. and will not re-request those records until it's own, local, cache expires. TTL is how long it should* keep the records until it discards them and re-requests them.

Object lesson time...

UserPC1.ValuedReader.com visits www.TheEmailAuthority.com at 2:30 PM. www.TheEmailAuthority.com has a 1 hour TTL for it's A record. If I move the website at 3 PM, and the old IP stops serving the site, how long would it be until UserPC1.ValuedReader.com can read my site again?

UserPC-2.ValuedReader.com visits www.TheEmailAuthority.com at 3:10 PM How long will it be until they can read the site?

The answers and explanations will be in part three.

TTL in DNS part 1

2012-12-07T13:44:00.001-05:00

TTL, or Time To Live, is a confusing subject for many people. Due to DNS's distributed nature, there are many different scenarios to consider. In this post we will explain what TTL is, and what it is not, using a real world example.

First, I suppose we'll answer why TTL is important. TTL can impact your email infrastructure any time you make changes to a Name Server that hosts your DNS, the A record that your mail server resolves to (for instance, when you change to a new hosting provider), or when you change your MX records (for a new hosting provider, or a new mail relay). By no means is this a comprehensive list, but it will suffice for this example. Since DNS records can be cached for days sometimes, it's important to understand how they work when planning a migration so you can account for several day's worth of traffic.

How it works:
According to RFC 1034, "DOMAIN NAMES - CONCEPTS AND FACILITIES", each DNS record (or "Resource Record", "RR" for short) has a TTL or Time To Live. It describes TTL as:

TTL which is the time to live of the RR. This field is a 32 bit integer in units of seconds, an is primarily used by resolvers when they cache RRs. The TTL describes how long a RR can be cached before it should be* discarded.

A "Resolver" is anything that resolves a DNS record. This could be your ISP's name servers, or your local computer. Anything that queries and resolves DNS records is a "DNS Resolver".

Let's take the below zone into consideration:

domain1.com. 86400 IN NS ns1.domain1s-ISP.com.
domain1.com. 86400 IN NS ns2.domain1s-ISP.com.

domain1.com. 14400 IN MX mail.domain1.com.

mail.domain1.com. 3600 IN A 10.11.12.13.

So, let's take our three examples above and do some "real world" exercises. Remember, TTL is expressed in seconds. You might want to open your calculator app now ;-)

If domain1.com wanted to switch their Name Server records to a different ISP, how many days would it be until the NS TTL expired?

If domain1.com wanted to add a hosted anti-spam service (which would require them to change the MX records), how many hours would it take for the MX TTL to expire?

If domain1.com wanted to upgrade their mail server and point the A record to a new IP, how many minutes would it take for the A record's TTL to expire?

Ok, so we've gotten through the math portion of this post. Pencils on the table please. Now we know that records can take DAYS to update depending on their TTL. How does this impact our migration? I'll answer in part two...

Stopping spam

2010-07-10T10:41:00.005-04:00

Albert Einstein said “We can't solve problems by using the same kind of thinking we used when we created them". This is certainly true of spam.

In the early days of email being widely adopted people were excited to use this new tool - and unaware of the threats that this new tool came with - so they handed out their email address willy-nilly. Every site that asked them for it, they gave it. If they wanted a personal response on a forum, they posted their email address for someone to respond to. In order to communicate with customers, they'd put their email address proudly on their "Contact us" page to show how "tech savvy" they were.

Then the unthinkable happened. People started emailing them.

Not about the thing they downloaded, not in response to their forum post, or even any customers from their "contact us" page. These emails were offers to buy prescription drugs or pornography. The senders had the unfortunate duty of informing them of the death of a rich loved one in Africa somewhere, or had good news of their winning a lottery Microsoft was running right now.

By the time they realized what was happening, their email address had already been abused, traded to other spammers, or sold in batches of "opt in" email addresses to the highest bidder. Their new email "toy" rapidly became much less fun to play with.

This is the "kind of thinking" that "we used when we created (the problem)". Email is a phenomenal tool, but it is a tool not a toy. You wouldn't let someone who had never used a chainsaw play with yours, and you wouldn't use a bandsaw to hammer a nail. Tools have purposes, and misuse can be dangerous.

So, what do we do? What's the "right" way of thinking we should start using?

1. Protect the email addresses you have.

Don't give it out to just anybody, and block unauthorized access to it. Use a spam blocker with not just a high block rate, but that also won't block your real email. This will save untold amounts of time (and therefore money) troubleshooting issues or even just searching through your junk looking for legit messages.

2. Use a unique email address.

This doesn't always mean creating new addresses every time you want to sign up for something. If your email address is YourUsername@YourDomain.com, try sending email to YourUsername+Whatever@YourDomain.com. If you receive it, you can do this when you sign up for things online and use something unique for each.

For instance, YourUsername+TheEmailAuthority@YourDomain.com if you sign up for a newsletter from me or YourUsername+TigerDirect@YourDomain.com if you sign up with Tiger Direct.

This will allow you to see WHO has compromised your email address, and also set up filtering rules if one of your "new addresses" gets compromised.

3. Use a "disposable" email address.

Ugh. We live in such a disposable culture. Disposable razors, disposable diapers, disposable this, that and something else. You mean to tell me there are disposable email addresses now too?!?! Yes. And they have a GREAT purpose.

When you sign up for a service, or a newsletter, coupons, etc you WANT to continue to receive emails from them. If you didn't, you wouldn't have signed up, right? Well, what about all of the myriad of websites that you need to give them your email address just to sign up and then you never want to receive another email from them, EVER? Use a disposable address, that's what.

Keep in mind, password resets, changes in TOSes and other vital email will go to this address as well, so don't use it for everything, but it's a great addition to your privacy protecting arsenal. Someone could guess to strip off everything after the "+" sign if you use the method above, but they'll never guess the real address associated to a completely fake address.

There are several services out there. I've used several, and I like Mailinator, but you can find a ton with this Google search.

So, here's the breakdown:

Give your real email address to friends, family and business partners.
Give your "+" address to sites you need to have ongoing communication with, but aren't in your immediate "circle of friends". Examples are stores, newsletters, blogs/websites, etc.
Use a disposable address for anyone you only need to communicate with once.

Using these simple steps will help keep your inbox clean and your privacy protected. To help protect against everything else, you need a good antispam service.

-TEA

Bounce message spam

2010-07-08T20:31:00.000-04:00

There's a new spam in the wild claiming to be bounce messages to messages that you've sent, but that actually send you to a meds spammer's site. They've been mutating for several days, so the body and attachment change fairly rapidly, but here's what one looks like as of the date of this post:

--- begin spam ---

Delivery Status Notification (Failure) Note: Forwarded message is attached.

This is an automatically generated Delivery Status Notification

THIS IS A WARNING MESSAGE ONLY.

Delivery to the following recipient has been delayed:

user@domain.com

Message will be retried for 2 more day(s)

---end spam---

Then there is an HTML attachment which redirects to the spammer's web site.

It's really quite ingenious really. Who wouldn't open a bounce, even if they didn't remember sending the original email, or maybe especially if they didn't remember sending it?

Another one, this time stating that it's from "mailfilter@yourdomain.com":

---another spam---

Note: Forwarded message is attached.

An email you sent could not be delivered.
Subject: Hello

Delivery to the following address has failed...
user@RecipientDomain.com

Technical information about this permanent failure:

-- Transcript of session follows --

While talking to [some.server.com] :

>>> rcpt to: user@RecipientDomain.com
<<<>: Recipient address rejected: User unknown

---end spam---

Users are so accustomed to worrying thet they're "infected" every time the get a bounce (when in fact, often it's just "spoofing") that these fake spam bounces must have a huge click through rate.

Although it's difficult to block "backscatter", I have to imagine that these are relatively easy to block. All the HTML attachment includes is a refresh to their site, any content based scanner should pick them up pretty easily.

As for the user's perspective, the old rules (slightly modified) still pertain here.

If you do not know the sender, do not open the attachment.

With these new "bounce message" spams, you can modify the above rule to:

If you did not send the email, do not open any attachments.

If you DO know the sender, and it's not a file you requested, call them to see if they actually sent it. Especially if it's in the following formats: .exe, .pif, .zip, .rar, .bat, .pdf, .htm, .html.

Of course, these spam messages have .htm file attachments, so the above pertains to them as well. Although these are a simple spam, websites can be infected and spread malware through your browser. Always exercise extreme caution when opening .htm files.

-TEA

Spoofing

2010-07-08T00:56:00.000-04:00

"Spoofing" someone's email address is really hard, right? Only the most elite hackers can do it, right? It's not something that I can try at home, right?

no, no and no.

In the post "Testing SMTP" we discussed all of the commands necessary to fully deliver an email. Nothing special, no encryption, just a plain text email (without MIME-types, etc), but an email nonetheless. Below is the list of commands again in case you missed that post:

---------------------------------------------------------------------------
HELO your hostname

MAIL FROM:

RCPT TO:

DATA

FROM:
TO:
SUBJECT: test

this is a test.
.

----------------------------------------------------------------

You'll notice something is missing in there. Something important for security. Something you do to many websites, to view your email, and often the first thing when you turn on your computer.

Don't scroll down, take a guess :-)

There's no login/password!

You could send yourself an email from GeorgeWashington@whitehouse.gov, or Bill@Microsoft.com saying whatever you like. Of course, you're still incredibly traceable, so don't get any bright ideas about sending an email "from" your boss giving yourself a raise or anything.

This is one way though that spammers and viruses hide their tracks or create a false sense of trust from their victim. SPF was implemented as a direct result of this flaw in SMTP's security.

Many times, spammers will "spoof" other users in their database of email addresses. Spam/virus filtering after the "Border-MTA" results then in backscatter, or bounces from email that was not sent by you. This is of course different than the article earlier about fake bounce message spam. Backscatter is legitimate bounces, but sent in response to an illegitimate use of the victim's email address.

Sometimes backscatter can inundate an inbox so thoroughly that it's hard to get any work done. In cases like this, ALL bounce messages should be blocked or moved to a folder keeping them out of the inbox. A common way to do this is to make a rule for these senders:

<>
postmaster@
mailer-daemon@

And that includes the following phrases:

"hi, this is the qmail send program"
"returned mail: see transcript for details"
"undeliverable mail returned to sender"
"delivery status notification (failure)"
"delivery status notification (delay)"
"out of office autoreply"

and any others that come in.

One of the reasons to not use a challenge-response anti-spam service is that they accept all email for your domain, then deliver "challenges" to which the sender must "respond". Since they accept the email, many deliver challenges to innocent users who are simply being abused by a spammer. Therefore you'll also want to include challenge response vendors in your filter.

Unfortunately, the lack of security in SMTP has been abused to no end. That's why it's imperative for your reputation, and a good "good neighbor" policy to implement SPF records. Although they're a little hard to grasp at first, They're the best way to be sure that nobody is using your email address without your permission. Unless you've been hacked, but that's a whole other post....

-TEA

Email and the OSI model.

2010-07-04T11:34:00.003-04:00

All People Seem To Need Data Processing, and when it comes to networking, understanding the OSI model is imperative. However you remember this seven layer schema, it's vital when troubleshooting difficult issues to remember them. Use these handy dandy mnemonics to help:

Please Do Not Throw Sausage Pizza Away (layer 1-7)
All People Seem To Need Data Processing (backwards, from layer 7-1)
Please Do Not Trust Sales People Always (1-7 again)

More information can be found in A+ and Network+ cert books, and other study guides I'm sure. Wikipedia has more information here too.

It's important to understand that although rare, your server logs can lie. Your mail server runs on the Application layer, so it only "sees" what that layer sees. Although of course it relies on all layers on down to the Physical layer, one single server may have several daemons running on different layers intercepting the data at different points along the way.

For instance, on a *nix box, you may have Sendmail logging that it's connecting to mail.domain.com [1.2.3.4].

Sendmail made a call to the DNS resolver and pulled the MX records. Again it called the resolver to resolve the A records listed in the MX. Then it tries to connect, but IPtables has a static route for 1.2.3.4 to send the data to 5.6.7.8, which is now refusing the connection.

Since Sendmail is running on the application layer, it doesn't log (or even see) when IP it's ACTUALLY connecting to, only where it *should* be connecting to. In order to see the packets you're going to have to do some traffic dumps using tcpdump, tcpick or Wireshark.

Of course, this is a very unique situation, either intentionally done by a sophisticated admin as a workaround or as a temporary "fix" for DNS issues, etc. Either way it's a kludge, and this is why each server should keep a changelog for later reference including the exact lines entered for later grep-ability (yes, I just made that word up :-) )

Also, there obviously could be many other issues with routing or other types of packet interference from router/firewalls locally to ISP related routing issues, but this was one instance that boggled me for several hours, so it stands out as an excellent reference.

-TEA

Testing SMTP

2010-06-21T23:08:00.002-04:00

Sometimes it's important to be able to manually reproduce what your computer/server is doing manually so you can see exactly where it's failing. Troubleshooting "utilities" are great in that they can often diagnose hundreds of issues in less time than it would take you to troubleshoot one, but they lack the nuance sometimes necessary to accurately diagnose issues.

A simple pass/fail is nice, but if you don't know WHY it's failing, what good is it?

When troubleshooting email, there are websites, utilities, diagnostic messages and more designed to help figure out what's going wrong. I have never found a single one that has been more informative than a simple SMTP session.

If you have telnet on your computer, that'll work fine. Netcat or any other command line tool that you can specify the ports to connect on will work too.

On your command/shell prompt type: telnet <the server's IP> 25

Then, when prompted, type the following commands:

---------------------------------------------------------------------------
HELO your hostname
<wait for response>
MAIL FROM: <your email address>
<wait for response>
RCPT TO: <recipient'semail address>
<wait for response>
DATA
<wait for response>
FROM: <your email address>
TO: <recipient'semail address>
SUBJECT: test

this is a test.
.

----------------------------------------------------------------

Note the additional CR/LF after the subject. That separates the headers from the body (visible part) of the message. It's very important.

Also important is the line with a "." by itself. This tells the recipient's server that the message is complete and to process it.

On to troubleshooting....

In each part above where it says "wait for a response" you're expecing a response from the server, either something like "ok, go ahead", an error, or sometimes nothing. Here are some examples:

HELO

If there's no response, this can indicate that the server is extremely busy or that sometimes there's packet filtration between you and the server that's intercepting and silently dropping the packets. Either way, if there's no response, there's no transmission, and therefore no email.

Sometimes you'll get an error here referencing that it does not like your IP (if you're blacklisted), or that it's imposing some type of lookup on your HELO argument. It's best to be sure thet you give your FQDN here to prevent issues.

MAIL FROM:

You might get SPF errors here if your SPF record is wrong, or if your email address is blacklisted you might get a response here.

RCPT TO:

"Relaying denied", "mailbox is full", or any number of other recipient level issues may result as a response to this line. Any server that has different rules per user or domain (like a shared spam filter with different settings per domain/user) might also process here.

DATA

You're now giving the actual data that will be shown to the user. Generally there won't be an issue here until you close the transmission channel (with the "." on a line by itself), but anywhere you wait for a response is a potential place for errors.

"." (or closing the transmission channel)

This is where the email in it's entirety can be scanned and processed. Sometimes nothing will be scanned until this point, so even though SPF *could* be looked up after the "mail from:" command, it *might* not be until the entire message has been transmitted. Any of the above errors can happen here as well as content based scanning of the body, etc.

Some mail servers will also accept the email here, and then scan it later before delivery. That's a horrible idea, but that doesn't stop anyone usually, so be sure to copy down any information the acceptance line gives you like the message ID, so you can reference it later if the test does not complete.

Generally, you'll only be able to run this test on a border MTA unless it's your own mail server or the recipient's firewall allows you through. This can definitely complicate troubleshooting, but if you can prove that the issue is with the border MTA, or that the border MTA accepts the email but it is not relayed it at least absolves you from any finger pointing.

Once you have these commands memorized (it doesn't take long), you'll often be able to troubleshoot and diagnose issues faster than submitting tickets to your vendors. There's nothing more frustrating than either waiting for a response only to find out you could have fixed it hours ago, or conversely that you know exactly where the error occurred, but the other admin requires "proof" that they should bother fixing the issue when the DSN or other logs obviously state the issue.

Send a couple SMTP sessions their way and copy the session to them. It's hard to argue with facts :-)

-TEA

subdomain MX records

2010-06-20T13:13:00.004-04:00

We've all used subdomains. The popular naming convention "www.domain.com" is in itself a subdomain (the subdomain "www" in the "domain.com" domain), but did you know each subdomain can also have it's own MX records?

The obvious questions are "why would anyone do that" and "how would anyone do that?". Here's some food for thought.

Way back in the day, things were simpler. Email specifically didn't require tons of MX and A record lookups, there weren't even DNS servers to query let alone DNS record types. Users had accounts on their local servers, and the servers had names that although understandable would be different from organization to organization. Email addresses weren't "bob@company.com", they were "bob@accounting.company.com" for Bob in accounting and "bob@engineering.company.com" for Bob the engineer. Each organization would have their own server names, and although often they'd have meaning within the organization they might be goofy to outsiders. Also, desktop PCs were uncommon then, so you would connect to the server for almost all of your daily tasks. Therefore, rocky@bullwinkle.company.com was literally addressing the email to the user account "rocky" on the server named bullwinkle.company.com. Some places still conform to these odd naming conventions. I knew someone who went to the University of Michigan's Engineering school who had an email address @engin.umich.edu. It took me months to figure out that "engin" was for " school of ENGINeering".

Nowadays it's slightly different. "company.com" generally will have all of it's users email addresses directly @company.com, but not always. For instance, let's say there's a domain BigCompany.com. BigCompany.com is all over the world, and they want their customers to know that they're dealing with their "local" office. So, they may have John@USA.BigCompany.com and John@Canada.BigCompany.com, but they're totally different Johns, in totally different regions.

Let's say they not only want to have email addresses at the country level, but also at the state level. They could also have John@California.USA.BigCompany.com and John@NewYork.USA.BigCompany.com.

Now, most companies wouldn't do this because they don't want to confuse their customers by having them talk to the wrong John all the time, but you commonly see this with schools for instance. They might be something like "Principal@SchoolName.NY.k12.us".

Ok, now we know why someone would have subdomain MX records. Either to conform to a legacy naming convention or to help routing between different regions/offices. Now the question is "how do we do that"? The answer shouldn't be too surprising, we make MX records for each subdomain.

Just like each domain name can have MX records (and should have them if email is addressed to it), subdomains can get in on the fun too with their own set of records. Let's say you have an email address webmaster@www.domain.com, but you don't want to run a mail server on the same box as your web server (which is a good idea btw!). You can simply have MX records for www.domain.com go to the same server as your regular domain like this:

Host Priority Record
domain.com 10 mail.domain.com
www.domain.com 10 mail.domain.com

Now, you may have to create a separate zone for this subdomain if you're running your own DNS server, or if you're using some provider's interface they may require additional steps to configure this but most of the time it's possible. Some services don't allow subdomain records to be made if you're using their DNS, but DNS hosting is cheap. If they don't allow this, and they have ridiculously bad support who won't discuss the finer side of RFCs with you (cough, yahoo.com, cough), just change your DNS host. Outside of this issue, you're likely to find other reasons to switch providers later if you're not already disgusted with their lack of support.

It's important to keep in mind, that this is not something you can do after the fact. Once your users have you in their address book as user@domain.com, they will ALWAYS query domain.com's MX records regardless of whether you add an alias for user@sub.domain.com unless of course you choose to change your email address and deal with all of the headaches associated with that.

It also should not be used after the border MTA using address rewriting without serious contemplation. Although there are some excellent reasons to have your border MTA rewrite the delivery addresses to "user@sub.domain.com" while the rest of the world still emails the user@domain.com" address, there are serious considerations to take into account at the very least the reply-to address, or sender rewriting back to user@domain.com on the way out.

This is not likely something that many admins have dealt with before, and could be conceptually confusing at first. Don't worry, it's nothing different than what you do on the main domain, it's just a second place to deal with. User@domain.com queries domain.com's MX and sends them there. User@Sub.Domain.com queries sub.domain.com's MX and sends them there. It's pretty simple actually once you get your head around it. :-)

-TEA

DNS doesn't end at MX you know...

2010-06-20T11:53:00.004-04:00

Ok, so you understand MX records, you're ready to take on the world, right?

Not quite yet. Patience young grasshopper.

So, you're sending email to user@domain.com, you queried dns1.domain.com for the MX records and now you've got these MX records:

Host Priority Record
domain.com 10 mail1.domain.com
domain.com 20 mail2.domain.com
domain.com 30 mail3.domain.com

Ummm, so, computers don't speak English, they don't know what a "mail1.domain.com" is. They want digits that can turn into nice 1s and 0s that they can compute . What's a poor mail server to do with this stuff?
While we're at it, how did we even know to talk to dns1.domain.com to get these records?
Also, once we knew to talk to dns1.domain.com, we're back to the first question of how does the mail server know WTH a "dns1.domain.com" is, and where are the 1s and 0s it needs?

I is so confus :-(

Don't worry, there's more to an email's life than just MX records. There's lots of types of records including type A, type SOA, and type CNAME.

I. Type A records

Type A records ("A" for "Address") turn "mail1.domain.com" into a nice pretty number (the "IP Address" or "Internet Protocol address") for our mail server (and all other computers) to compute. The A record might look like this:

Mail1.domain.com IN A 10.0.100.28

Now we can see that the first MX record for domain.com has a name of "mail1.domain.com", but that name actually resides at 10.0.100.28. This number means nothing to most of us silly humans, but to a computer it's a nice string of digits that it can compute and then connect to.

TCP/IP and all of it's routing mechanisms are beyond the scope of this article, so you'll just have to make do with "once you have the IP you can connect to the server". :-P pbbt!

II. SOA records

OK, so this is a little backwards since you need the SOA records before you can get the MX records or A records, but you should already understand MX records, and you need to understand A records before you can get SOA records.

SOA records are the "Start Of Authority" records. There are literally millions of DNS servers in the world. How do you know which ones are responsible for the records you're querying? Well, that's where SOA records come into play. SOA records are given by the "root" name servers to offload queries to other DNS servers, and to provide a distributed architecture that is easier to maintain and provides more fault tolerance.

Root name servers are the biggest and baddest name servers in the world. There are two sets of them for your .com, .net, and the like, one at root-servers.net and one at gtld-servers.net. If queried, they will either reference the other one (like a father who says "go ask your mother", or vice-versa) or they will give the DNS server registered with them as authoritative.

You can play with this yourself using nslookup on Windows or dig on *nix.

Once the root servers give the name of the DNS server, that server's A record must be resolved then it can be queried for the MX, A, CNAME, or any other type of records you wish.

III. CNAME records

CNAME records are basically aliases to A records. For instance, "www.domain.com" might be the same server as "ftp.domain.com", so you could make ftp.domain.com a CNAME for www.domain.com, then an additional query would happen for the A record "www.domain.com" which when resolved would then allow transmission.

Typically, CNAMEs should not be in MX records. There's a ton of debate over whether they're allowed in the MX at all according to RFCs. I'm not going to enter that debate, but I will say that there are enough mail servers on the planet that don't respect CNAMEs in the MX that you should do everything in your power not to put them there. At the very least it adds an additional query to your DNS server (to get the IP of the A record in the CNAME), so for simplicity's sake I'd recommend against it as well.

One time a CNAME might be used is if you have a server that accepts email that does not have MX records. Originally, user@domain.com was actually literally user@some.server.com. You could even address email to an IP address like "user@10.0.100.28" for example and it would work. Most mail servers will fall back to this convention if MX records cannot be resolved, so you could for instance have a CNAME errors.domain.com pointing to "mail1.domain.com", then you could email "www-server@errors.domain.com", "email-server@errors.domain.com", etc which would then actually send the email to mail1.domain.com.

NOTE: this does not alias other record types, only A records. For instance, setting domain2.com as a CNAME for domain1.com DOES NOT make domain2.com's MX records the same as domain1.com's. CNAME records are not copies of the zone, they are address aliases and will only route traffic to the A record they point to. Additionally, it's technically against RFCs to have both MX and CNAMEs for the same zone. I.e. if you have www.domain.com as a CNAME, you can't also have MX for www.domain.com.

So, there you have it. MX record resolution requires SOA lookups, which require their own A record resolution. MX records include A records typically, which require their own resolution by the SOA. CNAMEs shouldn't be used in the MX, and can't have their own MX, but they can still be used to route email in a pinch.

Try using nslookup or dig to do your own DNS recursion. Once you realize how many DNS queries each email requires, be sure to give your DNS server a big pat on the back for so many jobs well done.

-TEA

Email Feedback Report for IP

2010-06-10T22:34:00.002-04:00

Really AOL, really?

Over the years I have seen remarkable feedback reports from AOL. Everything from obvious user error like newsletters, etc. to bills that the user did not want to receive (but were completely legit), to those "chain letters" telling you that you and your entire family will die painfully unless you forward the message to 200 of your closest friends.

Recently, I saw a feedback report for a read receipt. A READ RECEIPT. Someone sent an email, got the read receipt back and then reported it as spam. That part doesn't surprise me (nothing much does any more), but a provider that has been in business as long as AOL (remember when they "were" the WWW for many Americans?) not having processes in place to deal with a user error this remarkably dumb amazes me.

First, their entire concept of blacklisting IPs is so 1999. It's 2010, use a content scanner!

Second, relying on their users to create an automated IP blacklist for them misplaces the burden onto their customer base, rather than trained techs (who can tell the difference between spam and a read receipt) and simultaneously creates a worse experience for all of their other users when "spam button" happy users wreck the IPBL for everyone else.

Third, if you're going to have an IP blacklist, and it's going to automatically block based on user reports, AT LEAST try to filter out obvious user error. A READ RECEIPT! Their logs show the message leaving, and the message clearly states it's a read receipt, the headers even had the appropriate "notification" message type. AOL, do you really have to make it this easy to poke fun at you?

AOL, I would like to offer my services to you. I'll even do it for half my hourly rate. Just please, please, please revisit this horribly planned out mechanism and get the correct processes in place to handle your spam problems.

While you're at it, could you please fix your outbound spam problems too? How many of your customers are really in Nigeria that know dead rich people with my last name?

-TEA

What's an "MX Record"?

2010-06-09T21:33:00.004-04:00

Unfortunately, MX records are one of the most misunderstood points of email transmission. Padawan mail admins have many misconceptions. Here are a few:

1. That the MX records have a specific naming convention like mail.domain.com or smtp.domain.com

2. That every server that handles email must be in the MX record

3. That the destination MTA must be in the MX record

4. That an "MX server" is another name for a mail server

5. That a "reverse DNS" lookup queries the sending domain's MX

Now that we know a few of the things an MX record is NOT, let's take a look at what it IS.

An MX record is simply an entry in a domains DNS zone. Typically they consist of one or more "A" type records (which then require resolution) with priorities assigned to them. Also, as with all DNS record a TTL value, or Time To Live, is assigned to them as a whole (and to each type A record as well).

They are simply references to the border MTA(s) responsible for relaying incoming email for a domain. When an email server is asked to email user@domain.com, it queries the DNS servers responsible for the domain (often by recursing to the root name servers looking for SOA records) and asks for the MX records. Typically, the MX records consist of type A records so then another query is given to find the IP addresses associated to them.

Let's take this story as an example. Let's say you're at a friends house discussing a long lost friend.

Friend: Hey I bumped into John last week at the mall
You: Really, I haven't seen him in years! Did you get his phone number?
Friend: Yes, but I don't remember it. You'll have to look in my phone.
You: Where's your phone? (recursive DNS query to the root NS)
Friend: Over on the table
You: What's his entry? (MX record query)
Friend: It's "John Doe"
You open the phone and look at "John Doe's" entry (type A record lookup) and call him on your phone. (transmission)

This simple story, that has probably never actually happened to any of us but should be able to be understood by most of us, is as simple as MX gets. Of course there's more to it (TTL values, "border MTAs", lots more DNS lookups, and any number of issues routing), but those are really more relegated to the realm of troubleshooting, and not something to worry our pretty little heads with right now :-)

Once it's been determined where to send the email, the transmission ensues and sooner or later after being relayed, archived, scanned for spam/viruses, and many more operations the email will eventually find it's home at the destination server and be picked up by the client to be read by the user.

Just think of all of the servers across the globe that played a part in getting your email to it's destination sometime when you're sending email. It really is amazing how it all comes together so seamlessly.

The next time you get the call from your user wondering why it took five seconds to get an email try explaining this to them. Boggle their mind a little bit ;-)

-TEA

Temporary and permanent errors

2010-06-03T23:50:00.001-04:00

Typically, you'll only see two types of errors "Temporary" and "Permanent" errors. Standard codes can be found in RFC 2821 section 4.2.3, but there are many different servers that reply with alternate codes as well.

Temporary errors are expressed through 4.x.x error codes. These might be due to the recipient server being busy, the recipient server greylisting the sender, or closing the connection prematurely. The sending server can also generate 4.x.x errors if they can't resolve the recipient's DNS, the recipient's server is offline, the recipient server is connected to and responds but does not fully accept the email or any other issue that does not result in the recipient server accepting the email (with a 2.x.x status) or outright rejecting the message (with a 5.x.x message). Sometimes firewalls intercept the packets and mangle them or drop them, this can also cause a 4.x.x error since either the recipient's server does not receive the correct packets to accept the message and times out the connection eventually, or the sending server does not know that the recipient's server accepted the message and also times out the connection eventually.

While temporary errors occur due to transmission issues or issues that are expected to resolve themselves typically, Permanent errors mean that the server was connected to, delivered a response, and that that response was that it refused to accept the message... EVER.

Typical reasons for this are that the mailbox is full or nonexistent, the message was blocked as spam/virus, the mail server has vital errors or a policy violation such as non-local delivery for the domain, too many hops per message, or others. This error means that the server is online, listening for connections, accepted your connection but then actively refused to accept the message. It's the difference between sending someone you don't want to talk to to voicemail versus picking up the phone and telling them to stop calling. There is no confusion.

A sending server can also generate a permanent error as a result of timing out the message after repeated temporary errors, but this will typically be hours or days later, whereas a recipient server's rejection is almost instant usually. Obviously there will also be repeated 4.x.x errors in the logs to reference.

Understanding the difference between these errors is also important to determine where to troubleshoot first. A permanent error is almost always the recipient server, whereas a temporary error could be one of many things.

Senders side:

DNS issues (queries)
ISP/routing issues
Firewall issues (packet filtering)
Addressing issues

Recipient's side:

DNS issues (serving)
ISP/routing issues
Firewall issues
Greylisting
Rate Limiting
Hardware issues
NAT issues
and many more....

As always, the bounce itself will have a lot of good information, the "How to read a bounceback" post should be the first place to check if a NDR/DSN has been generated, but otherwise use of your troubleshooting tools in conjunction with the log data should provide plenty of information to determine the source of the issue.

TEA

Reading message headers

2010-06-03T22:15:00.001-04:00

The routing of the email section is reasonably straightforward if you keep in mind that they are backwards.

1. Look for lines in the headers that say "Received: from".
2. Look for the last "Received: from" line and read upwards.

For instance, let’s assume we’re tracing an email with the following headers:

Received: from unknown (HELO BorderMTA.MyDomain.com) (9.10.11.12) by mail.MyDomain.com with SMTP; 13 Feb 2009 23:23:29 +0200
Received: from outbound.sendingdomain.com (outbound.sendingdomain.com [5.6.7.8]) by BorderMTA.MyDomain.com (8.14.1/8.14.1) with ESMTP id n1DLNSS7000915 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Fri, 13 Feb 2009 16:23:28 -0500
Received: from mail.sendingdomain.com (mail.sendingdomain.com [1.2.3.4]) by outbound.sendingdomain.com (8.14.1/8.14.1) with ESMTP id n1DLNQgX009122 for ; Fri, 13 Feb 2009 16:23:26 -0500
Received: from 10.0.0.100 ([10.0.0.100]) by mail.sendingdomain.com ([10.0.0.1]) with Microsoft Exchange Server HTTP-DAV ; Fri, 13 Feb 2009 21:23:21 +0000

When read from bottom to top says it went from 10.0.0.100 to mail.sendingdomain.com in the first line: “Received: from 10.0.0.100 ([10.0.0.100]) by mail.sendingdomain.com ([10.0.0.1])”

1. From mail.sendingdomain.com to outbound.sendingdomain.com in the second line: “Received: from mail.sendingdomain.com (mail.sendingdomain.com [1.2.3.4]) by outbound.sendingdomain.com “
2. From outbound.sendingdomain.com to BorderMTA.MyDomain.com in the third line: “Received: from outbound.sendingdomain.com (outbound.sendingdomain.com [5.6.7.8]) by BorderMTA.MyDomain.com”
3. And finally it's delivered from BorderMTA.MyDomain.com to mail.MyDomain.com in the last (topmost) line: “Received: from unknown (HELO BorderMTA.MyDomain.com) (9.10.11.12) by mail.MyDomain.com”

You can tell that the sender has an outbound email scanner (outbound.sendingdomain.com) after their mail server and that you have a border MTA before your main mail server.

Now that we know what we're reading, it's trivial to troubleshoot routing issues or delays that may not have been permanent errors or delayed enough to cause a DSN, but delayed nonetheless. Let's take the above headers and play with te timestamps a bit

Received: from unknown (HELO BorderMTA.MyDomain.com) (9.10.11.12) by mail.MyDomain.com with SMTP; 14 Feb 2009 00:53:47 +0200
Received: from outbound.sendingdomain.com (outbound.sendingdomain.com [5.6.7.8]) by BorderMTA.MyDomain.com (8.14.1/8.14.1) with ESMTP id n1DLNSS7000915 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Fri, 13 Feb 2009 18:53:13 -0500
Received: from mail.sendingdomain.com (mail.sendingdomain.com [1.2.3.4]) by outbound.sendingdomain.com (8.14.1/8.14.1) with ESMTP id n1DLNQgX009122 for ; Fri, 13 Feb 2009 16:23:26 -0500
Received: from 10.0.0.100 ([10.0.0.100]) by mail.sendingdomain.com ([10.0.0.1]) with Microsoft Exchange Server HTTP-DAV ; Fri, 13 Feb 2009 21:23:21 +0000

Can you find the delay? I included several time zone changes, so don't get duped by them ;-)

If you said it's from outbound.sendingdomain.com to BorderMTA.MyDomain.com you're correct!

It took about 2.5 hours for outbound.sendingdomain.com to successfully deliver to BorderMTA.MyDomain.com. Now that we've got the location of the delay we know where to start looking for the issue. Although the headers simply show where the delay occurred, not the actual cause of the delay, you can save a lot of time just quickly looking at them and isolating the possible trouble areas.

You might have noticed that each intermediary server is listed twice in the headers. When the server receives the message it says “by some.server.com” and then when it sends it on it says “Received: from some.server.com” in the next line. Often this is enough to tell where the delay occurred. If a server accepted the message and didn’t deliver it for many minutes, hours, days, etc. it frequently indicates that the recipient’s mail server was unavailable, deferring connections or “greylisting” the sending server. In that case, the logs you’ll want to read are on the recipient’s mail server (or possibly their firewall especially if it’s a connection issue or if it has a MTA built in). If this server does not have pertinent log files, then contact the sending server’s administrator to see if their logs show any additional information.

Sometimes the sending server may have issues sending an email that are outside of the recipient mail server’s control. A few examples would be retrieving the DNS information of the domain or recipient server, issues with connectivity or specific routing due to ISP issues, or any number of other issues. In these situations you’ll have to contact the sending server to determine the issue. Because the message never left the sending server, there will not be any logs on the recipient mail server. This is often the best place to start when troubleshooting home built or custom email applications such as bulk mailing software/vendors, website forms etc.

In general, if there's a delay you can start looking at the recipient server logs first. These will be the most verbose if in fact the issue is there. If the recipient server is a "border MTA" or another server that is responsible for all of your incoming email, and you're not experiencing issues across the board, you'll probably want to check the sending server first. If there was an issue on the border MTA it's likely (unless it's a rate limit/spam filter/etc issue) that the issue is on the sender's side. Either way, the issue must be on one of these two servers, so there's no reason to troubleshoot elsewhere unless you see additional delays in the headers (which is unlikely).

In summary, The headers go in order from bottom to top. They should all have timestamps, although they may be in different time zones, or the server clocks could be off. If you have a delay reported to you, isolate the servers that were responible for the transmission at that moment, determine what services are running on the servers (anti spam, rate limits, etc) and which is more likely to relate to the issue at hand (if the issue is all of your email, it might be your border MTA, if it's only this sender then it's likely on their end) then read the logs on this server. Once you have the error in hand from the logs, Google it. You're not the first admin to run into this issue, and most often someone has posted it to a forum, etc and gotten an answer.

TEA

Forwarding to users home addresses

2010-06-01T23:34:00.001-04:00

In today's busy information age more people are connected to their office email at all times than ever before. Smart phones, webmail or even VPN connections to their office network are more and more common.

One unfortunately common idea to help facilitate connectivity to work email during vacations or for off site users is to set up forwarding rules. So, if I'm on vacation (like that'll ever happen ;-) ) you could email me at my email address NotA.RealAddress1@theemailauthority.com (that's not a real address, so spammers can't harvest it) and I'd just set that address to forward to MyHome.Address2@PersonalAddress.com. Then when I check my home email, I'll be able to see the work I have to respond to as well.

Seems like a very convenient way to give people access to their email while on vacation or for offsite users in general, huh? Think again.

If I had a penny for all the reasons I'm about to list I'd have, well a few more cents, but you know what I mean.

I. Habituation

Social Engineering is often used in hacker terminology to mean gaining access to restricted material by direct human contact. Although almost the exact opposite of this usage, it's still the admin's responsibility to "engineer" their end users into acceptable behavior. It's not only our job to facilitate technological requests, it's our job to explain why some things shouldn't be accomplished a certain way, if at all. If a user said "this whole logging in thing, it's a pain to deal with. Let's get rid of it" you'd look at them as if they had two heads. Similarly, there are times when we must take an almost mentor type role and explain the repercussions of user's decisions and guide them to become used to a more sound solution or adapt the solution in a way that's secure and doesn't create even more issues down the road.

Here are some examples of how users can create issues with email forwarding.

1. Reply/Reply All

OK, so a client's email gets sent to HomeUser@gmail.com or whatever. Now, if your user has to respond, where does the email come from? Their home address, that's where. This causes confusion to the sender since now they have a new address, and it can take weeks for the senders to stop just clicking "reply" or finding the old thread and "replying all".

2. Address books

In addition to the threads going back and forth to the user's home address, how many times do your users bother to look at the actual email address they're sending to? Often they rely on their address book entry, which can now be compromised with the new entry for "John Doe", but "John Doe's" home address. This can indefinitely prolong the above "replying all" issue as each new thread will be sent to whatever address "John Doe" has in the remote address book.

3. Multiple mailboxes

Skip forward a few days. Now the user is back in the office and looking at their normal email client. Where are the messages? In their home email, that's where. Now they have two places to check, again sometimes for weeks until the thread dies, offering the possibility of lost email, delayed responses, or any number of other issues that make your company look bad.

4. Misplaced blame

Ok, so your user receives a spam message (or a message they think is spam). They click the "this is spam" button in their gmail/hotmail/etc. If the message is being forwarded from your server, their home email provider often simply sees the connecting IP which if you're forwarding the email from your mail server is now you. Now the user's home provider views you as a spammer, and the legitimate emails you're trying to get to your user get caught in the spam trap as well as the spam. I've seen AOL for instance do this several times causing admins no end to headache (try getting a decent AOL support tech on the phone to fix a problem, I dare you).

II. Technology issues

In addition to the user based issues above, there are obvious technical issues as well.

1. Routing of complaints

Although mentioned above, I cannot stress this enough. If you do this with the average user, sooner or later you take a strong risk of getting blacklisted. Depending on a few factors this can have tremendous impact on your infrastructure. If your user is sending spam reports to their provider, and you have many other customers on the same provider, your email will end up not only in your user's spam box, but any other customer who uses the same provider. For instance, if you forward to AOL, and your user gets you blacklisted by AOL then you try to send a legitimate piece of email to another AOL user, you will not get it through. I simply cannot stress this enough.

2. Bad practices

Anyone remember the post about the "border MTA's" role? If not, click that link and read it right now. Now you've got your email passing through your border MTA with all of it's antispam, antivirus, DDOS protection, etc, then you're passing it to ANOTHER border MTA to handle the same things again. You can see above for one instance of how this can cause headaches, but there are so many more. Rate limiting, spam complaints, automated processes, greylisting, availability, the list goes on and on.

3. Backscatter

I'll have a whole post at some point on this topic if you're not familiar with the term. In the meantime you can click here for more info. Including multiple border MTAs will almost always result in some type of backscatter. There are RBLs that focus exclusively on the source of backscatter, and many include backscatter sources at least incidentally in their lists. If you've ever tried to get off of some of these RBLs you'll probably agree you don't want to do it again anytime soon. Additionally, I guarantee if you've ever been the victim of a spoofing attack you'll understand how frustrating backscatter can be, and you'll never want to inflict it on anyone.

4. Archiving and other requirements

Many companies are required by law to archive email sent on the company's behalf. How are you going to archive messages that were sent from the user's gmail/hotmail/yahoo account? How are you going to archive messages that were replied back to those addresses? Is it more or less likely that someone on vacation would respond in an unfavorable (and therefore liable) manner when they're sucking down Pina Coladas on a beach somewhere than when they're in the office? Don't you think these messages above all others should be archived?

5. "too many hops"

Many mail servers have rules in place for how many other servers can be listed in the headers. If there are too many, you can generate bounces that claim there were "too many hops". Since this is entirely out of your control once it leaves your network it's something to take very seriously unless you enjoy talking to other support reps all day while troubleshooting this error. Although a couple additional hops should be fine, you have no idea how badly mangled your user's home email routes are. Maybe they've had the same email address for years, but they tried Hotmail first, then used AOL, then gmail. If they set forwarding rules (which are often the home user's only option outside of changing their email address) you'll have several hops per provider, not including yours. You can see how this would quickly add up. Also, there are outbound routes too, so it could be that some of your senders place several hops before it even reaches your network. Therefore any testing you do would be moot, and those senders will still get "too many hops" errors.

III. Easy solutions

So, your user needs remote access. This is an understandable need. There are literally too many options to mention, but here are a few

1. Give them access.

Look, we have to be security conscious. There are simply too many threats out there to leave your firewall wide open for all email traffic. However, there are plenty of ways to mitigate the risk without making your users' lives harder. If you don't want to open POP/IMAP/SMTP ports, dont. But use other ports and NAT them to the correct ports on your server. Of course this doesn't stop port scans, but it will lessen the number of available threats to the point you can sleep most nights ;-)

2. Require encryption

If you feel safe logging in to your bank account via the web, then you trust SSL. Require SSL/TLS connections at least from your outside users. This will both change the default port (which you should again change), and also prevent snoops from intercepting potentially sensitive information. Sure, they could have a keylogger, etc at home, but they might in your organization too.

3. Control the hardware

Provide them with a laptop if possible that contains all of the security software you use at the office and require them to return it when they're back home. This way you know at least as of the day they left they have up-to-date virus definitions, patches, and all of the other things users forget to do at home, but it's your job to do at work.

4. Use webmail, or get a vendor

Quite simply, webmail is a must. If you don't want to offer the webmail service directly on your server, offer it through a vendor and grant them access to your server. Zimbra for instance allows you to sync multiple mailboxes via IMAP in their webmail interface. as long as you grant access from the vendor to your IMAP port they can interface with their email just fine. Zimbra also offers the option of smart hosting to your mail server for outbound communication to help with your archiving/discovery software.

5. Give them a smart phone

If it's a power user who is always on the go, give them a smart phone. Personally, I prefer typing emails on many smart phones. Maybe it's all the years of gaming, but I actually find it more convenient to type on some of the keyboards with my thumbs than on my desktop with my whole hands/wrists/etc. Then, depending on the method of delivery to/from your mail server you can enforce policies to only allow connections from the specific devices, or their carrier will often offer (in the case of Blackberries) an option for them to deliver messages for you. Just be sure to allow the carrier to send from your domain in your SPF record.

To summarize, there are just too many simple solutions that offer more security and audit trails, as well as much less risk of headache. If you're an end user who has to deal with whatever your provider gives you, feel free to use email forwarders. If you're an admin you have the power to handle these issues the "right way", or really in any of the "right ways" you see fit. Remember the K.I.S.S. principle applies to your email's path, not your setup. Think of the slight inconvenience of doing it right as an investment on all the time you won't have to spend troubleshooting later.

TEA

How to read a bounce message

2010-05-18T17:19:00.002-04:00

So, you or your user just sent an email, and it bounced back. One thing to notice or ask the user is "how long did it take for the bounce to come back?". Although not always important it can help isolate the difference (especially if the user already deleted the bounce) between a routing error and a "hard bounce" if the message was actually rejected (rejections are often instant).

If you're lucky enough to actually have the bounce, you usually have all the information you need to identify the server that had the issue and at least narrow down the cause (if not outright fix the issue). Unfortunately, many antispam services have taken to responding with generic status codes instead of explaining that they blocked the message and at least giving some sort of explanation. This can make the status message (which I'll explain shortly) worthless, but generally it'll be a good place to start.

When you're looking at a bounce (and by "bounce" I mean both a DSN and an NDR) there are three main things to look for.

The sending server
The server that encountered the problem
What the issue was

Now, if the email was going from one user to another on the same domain, 1 and 2 will be the same server. In that case you can skip to section three. If the sender was sending the message out to the internet, 1 and 2 MAY be the same server, but not necessarily, and not nearly as often.

According to RFC 2821 (my clarifications in italics):

"If (a relay server) accepts the task (of relaying an email), it then becomes an SMTP client, establishes a transmission channel to the next SMTP server specified in the DNS ... and sends it the mail"

More information on client/server can be found on the main site, but basically the "client" is the sending server (#1 above), and the "server" is the #2 server above.

1. The Client (sending) server. Also called the "Reporting-MTA".

All email has to have a sender. Even in the event of a "Null sender" the sender exists, it's just Null :-). Generally mail servers will send bounces as a null sender, but forge their name into the headers like "root@sending.server.com".

One thing to note about the "from" string is that Exchange 2010 has a new "feature" (quotes very much intended) where it forges itself as the "from" string in the headers on incoming bounces regardless of whether or not it generated the bounce. This is outrageously annoying as it removes one of the first places end users and novice admins look to see where the error occurred.

It's often presumed that the server where the bounce originates from is the server that had the issue. Although this is possible if it's a local error, most often it is the NEXT server in the route that caused the problem. More below...

You can find out what server is sending the message by looking in a few common places:

The "From" header - Often the "From" header in a NDR is "@sending.server.com", this shows that "sending.server.com" is the sending server.
The "Generating server" - If the body says "Generating server: sending.server.com", this is the server sending the bounce.
The "Reporting-MTA" - The "Reporting-MTA" may be listed in the bounce. If so, this is the sending server.

2. The "Server" server, or the "Remote-MTA"

Remember the section from RFC 2821 before where it says "If (a relay server) accepts the task (of relaying an email)"? Well, that bolded part is important because that's where 99% of your bounces will come from on a daily basis. Most bounces come from a server NOT "accepting the task", and instead delivering a 400 level response (which is like a "not right now, maybe later") or a 500 level response (which is like an outright "absolutely not").

The Remote-MTA is often the only place to find logs of WHY the message was refused. The Reporting-MTA will have logs THAT the message was refused, but will sometimes have less information even than the bounce message itself.

A few common places to find who is the "Remote-MTA" are:

The "Remote-MTA:" field in the bounce
The "Remote Server:" field in the bounce
Where it says "While talking to:" in the bounce

3. What the issue was

If you don't have access to the Remote-MTA's logs (or if it's an internal bounce and there is no Remote-MTA), the bounce will usually have some type of diagnostic text. It won't be as much as the server's logs, but it's a whole lot easier to read a bounce than read through logs. I recommend starting with the bounce :-)

Here are some places to look/things to look for:

If the bounce says "while talking to: receiving.server.com" it will say after that "receiving.server.com said:"
The bounce may say in the body "Reason:", "Error:" or "SMTP 5.x.x [...diagnostic text...]"
The bounce may have text that is configurable by the Remote-MTA such as links explaining the error or general diagnostics. For instance "Rejected due to abuse. See www.some-website-or-other.com/bounce.html for more information"

There can be other information in a bounce like the "Received-From-MTA" or additional headers so you can see the route, but reading headers is often extraneous at this point, and probably deserves an article all to itself. If you're super excited, read RFC 2821 for information on SMTP, and then RFC 3464. That'll cool your heels :-)

Regards,

TEA

What is a Border MTA?

2010-05-17T17:21:00.002-04:00

A Border MTA (Mail Transfer Agent) is the first mail server responsible for handling your email. It operates at the "border" of your network, hence the name.

This is commonly an antispam service or appliance, or sometimes your firewall will have an "SMTP Proxy" built in (common with SonicWall, Tumbleweed, Symantec and Cisco router/firewall combos). Sometimes all the packets are sent directly to the mail server, then it is the "border MTA".

The easiest way to determine your Border MTA (although not always 100% accurate) is to see what server returns its name in response to the SMTP HELO command. In the event of an SMTP proxy on the gateway, it can proxy even these responses, but that's uncommon enough to ignore for most troubleshooting.

The reason an MTA in this location has a specific name is that there are some services that should run ONLY on the Border MTA, and if they are run anywhere else in your email's route can cause disastrous issues.

For instance, SPF authentication should ALWAYS be run on the Border MTA. Microsoft, in their infinite wisdom, decided to include SPF (under a proprietary name, surprise surprise) as a default option in their IMF settings (which stands for "Intelligent Message Filtering", please keep your guffaws to yourself) in Exchange 2003 SP2 through 2010. Keeping in mind that many of their enterprise level customers would have several layers of protection, and therefore probably several MTAs in front of theirs, this was a less than educated decision IMHO.

SPF (or Sender Policy Framework) is a relatively new standard (spearheaded by MS oddly enough) that adds DNS records for the domain advertised in the "mail from:" command. More information can be found here, but essentially if the connecting IP does not itself exist or resolve to a record that does exist in the SPF record the email is bounced or flagged. Let's see how it's supposed to work:

Sender: me@domain.com
Sending server's IP: 1.2.3.4
domain.com's SPF record: "IP4:1.2.3.4 -all"
recipient's border MTA's IP: 10.0.0.1
recipient's mail server: 10.0.0.2

The recipient's Border MTA does an SPF lookup on domain.com since that's in my email address. My DNS server says the SPF record is "IP4:1.2.3.4 -all", which means that all email should be coming from a server with the IP of 1.2.3.4. Since the connecting IP is in fact 1.2.3.4 it passes this check and is either passed to the mail server or is at least allowed to continue being processed.

Now, how it DOESN'T work.

The recipient's Border MTA does an SPF lookup, and it passes the SPF check due to the reasons above. Then, the Border MTA passes the message to the mail server which now does another SPF lookup on it. This time however, the IP of the connecting server is actually 10.0.0.1 NOT 1.2.3.4. So, since this IP is not authenticated by the SPF record, the message bounces unless other steps are taken to prevent this (such as whitelisting your Border MTA's IP).

Other examples of tests that should NEVER be run after the border MTA include RBL checks, reverse DNS tests (and "Forward Confirmed Reverse DNS"), greylisting, or really any IP based authentication techniques.

Of course, all anti spam tests should also be completed at the Border MTA, so greylisting, antispam, antivirus, and other tests should be run there as well to prevent backscatter. Only in special circumstances should even rate limiting be done after the border since this can cause unforeseen bottlenecks that can take an essential route out of commission.

In conclusion, if you only read one paragraph this should be it. Run ALL tests, especially IP based tests, at the border. The only thing that should happen past the border MTA is routing/load balancing to different destination MTAs (where the email is delivered to the user). Ignoring this rule is guaranteed to add to your headaches, and to your employer's dissatisfaction.

Regards,

TEA