We’ll be speaking at the Deliverability Summit in Amsterdam on April 24th and 25th.

If you have your tickets, come say hi!

If you don’t have tickets yet there are still a few in-person tickets available, as well as on-line access.

If a message has alignment between DKIM and the 5322.from address but there is no DMARC record for that domain published in DNS, Google gives a warning that the domain doesn’t align.

Clearly the domains do match and the message is aligned. However, there is no DMARC record published for blighty.com.

My speculation is that the alignment message is generated from the Authentication-Results header. When you pull up “show original” google grovels through the “Authentication-Results” header to populate all of the special fields. If there is a DMARC=pass stamped in that header field Google reports “Pass”. If there’s not a DMARC=pass in the header field, Google looks for the DKIM d= value and the From header and puts those tokens into the Alignment message.

What appears to be happening here is that Google only reports alignment in the Authentication-Results header if there is DMARC record published in DNS. If there is no record, they don’t report DMARC=Pass and therefore the default Alignment message shows up with the domain names.

We can look at the raw headers and see all of this happening in the messages – ones with the incorrect Alignment message don’t have a DMARC=pass stamped in the headers.

I kinda want to talk about how Google isn’t using SPF here but every time I start that paragraph my science brain kicks in and goes “but you need to test that first”. Right now we can say that our tests show that a SPF pass with DKIM unaligned (but passing) is enough to get “Alignment=Pass” if you have a DMARC record but not if you don’t. I can’t help wondering if you get a DMARC=pass with DKIM but not SPF if you still get a warning. I don’t easily have a way to send mail that fails SPF but passes DKIM so I can’t do the tests I want, nor am I sure if I could that it would give us more insight into Google’s inner workings.

I can say I’m extremely pleased that our brand new mailserver in IPv6 space is successfully sending mail to Google and reaching the inbox even after just a few messages. It’s nice to know small mailservers can still work for small senders without a penalty from the big mailbox providers.

Yesterday I answered a question about whether or not spamtraps existed. My answer was that they do exist in the sense that they’re real email addresses but that they don’t exist in that they’re not used by individual people.

Overall, they’re real email addresses that are not read like normal mail. They’re simply used as an indicator of whether or not a sender is really sending permission based mail. The interesting thing about spamtraps is the type of trap can tell the trap owner a lot about what poor mailing practices. They can also tell the sender about where the problems in their data collection process lie.

Different Kinds of Spamtraps

In 2011, I wrote a post called “A Brief Guide to Spamtraps” where I talked about a bunch of categories of spamtraps. In the time since I wrote that, a few other terms and types of spamtraps have entered into the public lexicon. I’m talking, of course, of Pristine, Recycled and Typo traps.

Pristine Traps

Pristine traps are addresses that have never existed. Usually they’re at domains that have never existed for email either. We have a couple domains here that we don’t use for mail, but if we were to start accepting mail to those domains, they’d be pristine traps.

These traps generally only get on lists because addresses are being created by someone. They simply are not addresses anyone would use. One example is the “cold-outreach” sender who sends to laura-atkins@. That’s a pristine trap, the spam was the first message that address ever received. Now, did the spammer make it up? I actually think so because it gets no other spam, but these addresses can also be purchased.

Pristine traps tell the trap owner that the sender is making up addresses or buying them from a list seller that is making up the addresses. Overall, pristine traps are a sign that the sender is not building their list through opt-in processes.

Recycled Traps

Recycled traps are domains or email addresses that received legitimate mail at one point, but were decommissioned, allowed to bounce for a minimum of 12 months and then reopened.

Mails to these traps most likely indicates that the sender is either not properly bounce handling or they found an old list and started mailing to it. Recycled traps can also show up in purchased lists. Overall, though, we treat recycled traps as a sign of poor list hygiene.

Typo Traps

Typos happen whenever someone inputs what they intend to be their own email address but mess it up somehow and typo the address. Most of the time when we’re talking about typo traps, we’re talking about typos on the domain side of the email address. Over the years security and anti-spam organizations have bought some of these domains and turned them into spamtraps.

The term typo trap is a descriptor many of us use to indicate that we think, for the most part, a particular sender is only collecting email addresses through opt in forms. The problem is, the sender is trying to be opt-in, but they’re not taking care to verify the data is correct.

Spamtraps Are a Signal

Over the past 20 some odd years I’ve dealt with a lot of folks who come to me wanting help fixing their spamtrap problem. Just to be clear, spamtraps are NOT the problem; they’re a signal. Spamtraps tell us that there are problems with something about our data. No one really cares if email addresses that don’t belong to people get mail. But what they do care about is the fact that if your list processes allow spamtraps on the list, it’s likely you’re also sending mail to actual people who don’t want it.

Now that we know what spamtraps tell us, let’s talk about how to deal with them, particularly when they’re resulting in problems with your email sending.

Dealing with Spamtraps

In my line of work, so many clients come to me and part of their opening brief is “we want to remove spamtraps from our list.” I reframe that as “removing non-opt-in addresses off the list” and make sure that we’re clear I don’t have a list of spamtrap domains to simply remove. What I’m doing here is actually working out why there are non-opt-in addresses on a list and fixing that so that they’re no longer on the list, whether they’re spamtraps or not.

Here’s an abbreviated summary of my process.

• Identify what type of bad addresses are on the list.
  • I use a variety of means to do this. We have some internal tools, some ESPs and clients have access to public spamtrap data, and often Spamhaus will give me information about the type of trap involved.
  • Pristine traps suggest a purchased list
  • Recycled traps suggest a reactivation of old addresses
  • Typo traps suggest a problem with data entry on a website
• Implement changes to address that problem
  • Purchased addresses –
    • Does anyone know when they were purchased? If not, are there indications in data patterns that indicate when the purchase was added to the list?
    • If we can identify when the purchase happened, the simplest thing to do is just remove the purchased addresses. For some clients, they’re unwilling to give up data, so we see how we can address those concerns. For instance, we can treat a purchase as an implicit opt-in and keep those addresses. Or we can send a confirmation emails to some or all of the purchased emails. In any case, the goal here is to remove addresses of people who never asked for and who don’t want mail from the client.
    • The next step is to look at who at the company decided to buy a list and work out how to stop this from happening in the future.
  • Reactivate old addresses
    • Why were the addresses reactivated? Who authorized this? Who did the work internally to do this? Can we deactivate those addresses and remove them from future sends? Did someone actively decide this was a good idea or was there an oops with the database?
    • Once we’ve identified the how and why, I work with the client to find ways to stop this from happening in the future. This should never happen accidentally, so what technical changes will stop that from occurring again? If it was an internal decision, what was the thought process? Does there need to be additional training or some approval process to ensure this doesn’t happen for the wrong reasons. And, yes, there are good reasons to reactivate addresses (not many, but a few) and so that should be documented and it should be a responsibility belonging to someone at the client.
  • Data entry on the website.
    • Identify if the addresses are accidentally being subscribed, through some sort of Non-Human Interaction (NHI) or if people are giving bad data during the signup process.
    • There’s no one way to mitigate NHI or bots on the website. Al Iverson wrote a good blog post on this recently and I suggest checking out his post Signup Best Practices: Banning Bots and NHI recently and it covers the steps in more detail than I can on this post which is already too long.
    • If the issue is people deliberately giving fake or false addresses, then we start to ask why? What makes users distrust the sender so much? The next set of questions is what can we do to mitigate the bad data. The good news these days is most users are used to “check your email for a code” or other 2FA style

What doesn’t work to remove spamtraps is to use a data hygiene service. They might remove some of the commercial sensor networks but none of the data hygiene services have reliably identified spamtraps that cause your mail to be blocked.

Avoiding Spamtraps

It’s always better to keep spamtraps off your list in the first place. That means:

• Don’t buy lists. This includes avoiding buying B2B addresses.
• Don’t send to old addresses you find in the back of a desk drawer or in an old restore of your database.
• Take some step to verify addresses that are entered into forms on your website. You don’t have to go full COI to verify data entered into forms, but you do need to implement something that means you know there is a connection between the person who owns the address and the person who gave you the address.

More Info on Spamtraps

I’ve written in the past about different kinds of traps for almost as long as the blog has been around. I’m listing a few of the posts here. One of the interesting things is seeing how we write about spamtraps has both changed and not changed over the years. I think that reflects both how our understanding has changed and how spamtraps are used for filtering.

• 2010: Spamtraps
• 2011: A brief guide to spamtraps
• 2011: Spamtraps: should you care?
• 2012: Spamtraps are not the problem
• 2012: Spamtraps mean your list is bad
• 2012: Equivocating about spamtraps
• 2015: Only spamtraps matter, or do they?
• 2019: Recycled addresses, spamtraps and sensors
• 2019: Spamtraps on the brain
• 2019: Spamtraps are overblown… by senders
• 2019: Purging to prevent spamtraps
• 2019: Myths about spamtraps

Overall, spamtraps are a tool used by filtering companies and anti-spam organizations to tell them who has bad email practices and what those bad practices are. Those who end up on lists, or who have access to commercial sensor network data, can also use the trap information to understand what the underlying problems with their data are. These insights indicate what needs to be fixed.

Using spamtrap information to improve data collection and hygiene means a healthier email program and better overall delivery.

Do spamtraps really exist? Like, in the sense of being a real mailbox? They actually exist just like any other mailbox, yes? Otherwise they can’t be monitored and reported on. This includes typo spamtraps, correct?

Do spamtraps exist?

A spam trap exists in the sense that it is a legitimate RFC5322 email address and has a valid MX record (or A record). It’s a real email address. Most spamtraps accept mail for delivery, although there are a few that reject after the full SMTP transaction. In those cases the mail doesn’t “deliver” but the data is still captured by the trap owner.

What most of us mean when we say “spamtraps don’t exist” is that there is not a human that actually uses that address for email. The domain owner has not assigned that address to any individual for use. This means that no one sending mail to that address has permission to send mail from the user of the email address. There’s simply no person to give that permission to the sender. In that sense, the address “doesn’t exist.”

Are they read like any other mailbox?

In terms of existing like “any other mailbox,” again we’re in nuanced territory. For normal email delivery, the MX accepts the mail and then hands it off to a local delivery agent. Here, we have postfix as our MTA server and dovecot as our IMAP server . All of our mail clients (phone, desktop, tablets) then connect IMAP so we can read, respond to, forward and save mail.

But that’s not the only way we accept mail here. We have a system for clients where ever client gets a subdomain and can send mail to it. That mail never goes anywhere near a IMAP server, it’s dropped into a database and displayed on a website. I can read mail, but I can’t reply to it or forward it or anything. We also have aboutmy.email, which also drops mail into a database and displays it on a website but there’s no way to “read” mail if you don’t have the specific link for the message or do anything more with it.

The way our internal tools and aboutmy.email handle mail are much closer to how large spamtrap feeds handle mail. They accept the message (or just read through data and reject the mail), extract the data they need into a database and query it later. For large feeds, they physically can’t read the mail, it just arrives too fast. Some trap feeds are dozens or hundreds of messages a second. It’s also so much data they don’t keep it for long. They record important things in their database (IPs, domains, dates, headers) and delete the message after a few hours to conserve storage space.

So the feeds themselves are mostly mechanically ‘read’. The data is put into a database and dealt with as ‘big data’ with tools and reports. They usually don’t have any way to connect a mail client with them and individual messages aren’t read in any way that normal people read mail.

Do spamtraps interact with mail like real recipients?

For a long time many folks, including myself, reassured clients and the general public that spamtrap addresses were unlikely to show opens or clicks in email. And, statistically, we were correct. The folks running traps didn’t open or read or click on the vast majority of mail coming into their traps. However, there was always a chance that a spamtrap would open or click on a mail. Statistically it was unlikely. but I was always aware that we could be missing spamtrap focusing on engagement. Over time I got a little more refined in my recommendations to address these concerns.

More recently, I don’t spend a lot of time on opens or clicks as a filtering criteria to identify real addresses. The details depend the particular client, but I focus more on metrics that are actually the result of a human getting a message and taking an action. Opens and clicks are not interactions that indicate a human is reading a message and we cannot treat them as reliable human signals.

Overall, the answer to the original question is that spamtraps are real email addresses but they’re not addresses used by real people to sign up for mail. If they weren’t used as part of spam filtering and spam blocking systems, we wouldn’t care about spam traps. But they are used as a data source and too many spamtraps on a list can be a sign that there are issues with the sender.

Tomorrow I’ll be posting some more detailed information about spamtraps and what you can do about them.

One thing that some folks were seeing is a message that says:

This “Alignment” description replaced the DMARC verdict in the header. The interesting thing here is that while DKIM doesn’t align, SPF does pass and so the message technically passed DMARC.

Mike J. on the emailgeeks slack channel mentioned that he noticed that this only seemed to happen when there wasn’t a DMARC policy published for the domain. Well, I can test that! I have multiple domains that do align with SPF, don’t align with DKIM and don’t have current DMARC policies.

We published a DMARC p=none policy for carrotcafe.com and I repeated the test send.

So, yeah, that’s pretty definitive. The “alignment” warning pops up when DKIM doesn’t align and when there is no DMARC record published in DNS. If there is a DMARC record published in DNS, then the DMARC results take precedence.

Of course, as a scientist I would be remiss if I didn’t point out what I didn’t test. The conditions I don’t have the ability to test right now are DKIM aligned and passing with no DMARC record. My hypothesis / gut feel is that it would say DMARC pass, but without running the test I can’t say that’s what is happening.

Word of caution, though. These displays and reports do seem to be a bit buggy. Another email geek posted a screenshot that showed DKIM passing and aligned but also with the Alignment warning. In this case the alignment warning said “The From header of @email.example.com does not match DKIM domain email.example.com.” Which is clearly wrong. That message was double DKIM signed, by the customer and by the ESP domain, so it’s possible that there is a bug that needs to be fixed by the developers.

Overall, I think Google is testing how they’re displaying things specifically to the email deliverability space. Most folks don’t look at the “original display” for their emails. I’d even wager the vast majority of folks who do look at this are in deliverability, email, security or some other technology adjacent field. This is something they’re working out how best to show information.

One important thing to remember: the actual headers of these messages show the messages are correctly authenticated. Also, there seem to be no deliverability consequences (yet!) to the lack of alignment. Currently this is a display issue only. I think it does indicate that Google are serious about expecting folks to have DMARC records, even if they’re p=none. I also think it’s telling that they are putting much more value on DKIM passing and they’re ignoring SPF passing in the instance of no DMARC record.

I’ve been saying for more than a year that deliverability is in an era of upheaval and change and I think this is another example of it. We’re not sure what Google is doing, nor what it means. We just need to be a bit patient and keep our eyes open for what’s going on. I do expect it’s going to be a little longer before things settle down. But that’s OK, we’ve done this before, we can do it again.

Our current smarthost was configured in 2018 and hasn’t really been touched much since. We’re moving it to a different hosting location, in the EU rather than the US, mostly so we can make stronger privacy statements about all customer data being handled on EU located servers.

So we’re starting from scratch. I thought I’d document the process.

Basics

It’s a nice little VPS from Mythic Beasts. It’s just being used for outbound email, so I don’t need much storage. It’ll only have a few users – some human, some apps – so it’s not going to need much CPU either. It’s mostly just going to be running postfix and some helper services.

I paid the extra £20 to get an IPv4 address as well as the default IPv6 range. Email is not ready for IPv6-only yet.

Mythic Beasts have an excellent reputation for not tolerating bad actors on their network. That’s part of the reason I chose them for this – IP reputation isn’t as important for delivery as it used to be, but you still don’t want to be on a network provider known as a source of spam. They actually had a member of staff manually review the order before spinning up the new VPS, which is a level of due diligence I’ve not seen in years.

The normal new server setup path: sudo, ssh keys. I add it to our tailscale VPN and set up ssh to only listen on the VPN. Then I do the tailscale-wait-for-ip dance that’s needed to make services that bind to addresses at startup happy. If I’d done that in a different order I wouldn’t have locked myself out and had to use a virtual serial console to get back in. Oops. Lock down the packet filters so only port 587 can come in from the public Internet.

The hostname is preconfigured to be mail.turscar.ie – that’s what it will HELO as, and what it’ll stamp into trace headers, so it’s important to get right. Mythic beasts set up reverse DNS for both its IPv4 and IPv6 addresses when the VM was created, and I’ve added A and AAAA records for it to the turscar.ie nameserver.

TLS Certificates

We’re going to be submitting mail over the ESMTP SUBMIT port, 587. That requires authentication, so we need to support STARTTLS. That means that we need to acquire a TLS certificate, and it’s 2025 so we want a real, publicly trusted certificate, not a self-signed one.

There are several certificate authorities that’ll issue free, domain authenticated certificates using ACME clients, but I’m going to stick with Lets Encrypt.

Lets Encrypt certificates are only valid for a few months, so need to be renewed every couple of months, which means that requesting and deploying them needs to be automated. The most commonly used infrastructure for that automation is aimed at certificates for web servers, and requires a webserver to be running at the hostname that a certificate is requested for.

We don’t have a webserver for mail.turscar.ie. Fortunately the ACME protocol supports domain validation in other ways. The client I use, LEGO, has great support for authentication via DNS, supporting the APIs of a 150 or so different providers. I could use it’s native PowerDNS API support, but that would mean setting up and exposing that API on my primary DNS server, and handling all the authentication for that. Instead I use RFC2136 dynamic DNS updates. These are handled over DNS, and just require a shared secret between the DNS server and the ACME client.

(DNS is core to a lot of security, so if someone were to get hold of that shared secret it would be bad if they could use it to modify the DNS for my zones. So I create a subdomain _acme-challenge.mail.turscar.ie and allow only that subdomain to be updated via dnsupdate, and only from the IP address of the new server. The shared secret can’t be used to modify anything other than ACME challenge records, and it can’t be accessed from anywhere else. That’s an annoying amount of work to do by hand, but I wrote the script to do it once and now it’s a single command.)

There are a lot of certificate authorities out there, and not all of them are as careful about issuing certificates as they should be. We can mitigate that by publishing CAA records in DNS.

A CAA record contains the name of a certificate authority. Any certificate authority that’s about to issue a certificate is required to look up the CAA records for the domain. If it finds any CAA records, and it doesn’t find one with the CA’s name in it, it mustn’t issue the certificate.

We’ve already published CAA records for turscar.ie, so we’re good to go.

LEGO lets you run a script once it’s acquired a certificate and a private key. I mostly use that to write the certificate and private key to the normal places they live (/etc/ssl/certs and /etc/ssl/private) and reload any services that use them. Since I last set up postfix they’ve moved to recommending that the private key and certificate are stored in a single file, rather than separate ones, so I modify the hook script to concatenate the key and certificate and put that chain file where postfix (and only postfix) can read it.

I run LEGO once by hand, to get my certificate, then add it to root’s crontab to run nightly. When it runs it’ll check the expiry date of all the certificates it manages, and renew any that are close to expiring. That’ll keep running forever, and I’ll never need to look at it again.

A TLS certificate is part of a chain of certificates. The one you’re issued is authenticated by a certificate owned by the certificate authority. That one may be authenticated by an intermediate certificate and then that one is authenticated by a trusted root certificate. The list of trusted root certificates is compiled manually, and distributed with your operating system or embedded into your web browser. It’s good to know what certificates are in that chain, and make them all available to your server to offer to clients. If, say, you don’t offer an intermediate certificate then your certificate might not be accepted by a client. Or it might, depending on what certificates the client trusts. And if I wanted to publish TLSA records in DNS to strengthen trust in the certificate I’d need to serve the whole chain back to the root certificate

Running openssl storeutl -noout -text -certs mail.turscar.ie.crt prints out the chain of certificates in that file:

0: Certificate
Certificate:
    Data:
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C=US, O=Let's Encrypt, CN=E5
        Subject: CN=mail.turscar.ie
1: Certificate
Certificate:
    Data:
        Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
        Subject: C=US, O=Let's Encrypt, CN=E5

I’ve trimmed out a whole lot of information, leaving just the bits I want to talk about. You can see there are two certificates in this chain. The first one is the one I was just issued. The Subject: field is who it belongs to, mail.turscar.ie¹. The Issuer is the entity that vouched for this certificate, Let’s Encrypt’s E5 certificate.

The second certificate belongs to Let’s Encrypt, it’s their intermediate E5 certificate that they use for issuing many of their ECDSA certificates. The issuer is the ISRG Root X1 certificate.

And the ISRG X1 Root is widely recognised, so this certificate chain is good enough, at least until I want to publish TLSA records.

The Signature Algorithm: field shows that this is an ECDSA (elliptic curve) certificate rather than the older RSA type. Client support for ECDSA certificates isn’t universal, so the postfix documentation suggests offering RSA alongside ECDSA. If this were a public smarthost I’d do that, but we’ve been using ECDSA only on our existing smarthost and haven’t seen any problems, so I’m not going to bother.

Postfix

I’ve been using postfix for decades. It’s stable, fairly feature-rich, well supported and well suited for a small business mailserver, inbound or outbound. If I were building something bigger, or more suited to bulk mail usage I’d probably consider KumoMTA instead, but for this setup Postfix is fine. And I’m familiar with how it’s configured and how all the pieces go together.

The most recent release of Postfix is 3.10.0. The Debian repo gives me Postfix 3.7.11. I’d like some of the new features and security mitigations that have been added since then, but I’d like a clean, easily upgradeable install on Debian stable more.

apt install postfix asks me what sort of installation I want. I tell it I want a normal mailserver and it installs and configures postfix, and sets it running via systemd.

Time to configure things.

In /etc/postfix/main.cf I setup TLS

# TLS parameters
smtpd_tls_chain_files = /etc/postfix/ecdsa.pem
smtpd_tls_security_level=may
smtpd_tls_auth_only = yes

The default setup is for a mailserver that listens on port 25. We don’t want that, we want just a submission server on port 587, so off to /etc/postfix/master.cf we go. The default settings are fine, so I just comment out the smtp service and uncomment submission. Restart the service and we’re listening on port 587.

Client Authentication – SASL

We need to be able to authenticate clients, but we don’t need anything sophisticated. Simple username and password will be fine.

So we’ll use Cyrus SASL (simple authentication and security layer) with a PAM provider. Postfix will talk SASL to Cyrus, then Cyrus will use PAM to validate usernames and passwords against the operating systems usernames and passwords. This wouldn’t scale well, but for a few users it’s fine (particularly if those users are also going to want to log in to check the mail delivery logs occasionally).

sudo apt install libsasl2-modules sasl2-bin

This installs saslauthd and friends. The default setup is almost what we need, but not quite. Postfix communicates with saslauthd via a unix socket, but postfix is configured to run in a chroot, so can’t access the default socket location. The needed change is well documented in comments in the /etc/default/saslauthd configuration file.

Skipping over a lot of debugging here. Some days I hate being a sysadmin.

Unfortunately, the Debian 12 packaging of saslauthd just doesn’t work. There’s a bad interaction with SysV startup scripts and systemd meaning that it just silently fails, whatever you do. There are a smattering of open bug reports about it, and it’s supposedly fixed in what will become Debian 13. So I create a proper systemd service file for “saslauthd2” and that runs fine.

/usr/sbin/testsaslauthd -f /var/spool/postfix/var/run/saslauthd/mux \
 -u steve -p <password>

This gives me a nice OK "Success." response for a valid password and NO "authentication failed" for an invalid one.

Client Authentication – postfix

Now we tell postfix to use it by creating /etc/postfix/sasl/smtpd.conf

pwcheck_method: saslauthd
mech_list: PLAIN LOGIN

PLAIN and LOGIN are the two plain text authentication styles, which we need to be able to support them with this setup. They’re tunneled over TLS, so that’s fine.

The submission service in master.cf already has smtpd_sasl_auth_enable=yes and main.cf has smtpd_recipient_restrictions=permit_sasl_authenticated,reject so after reloading the postfix service we should be good to go.

Testing with SWAKS, finally

Time for a smoke test, using swaks.

swaks --to steve@blighty.com --server mail.turscar.ie \
  --protocol ESMTPS -p 587 --auth-user steve --auth-password mypwd

The SMTP transaction scrolls by, swaks uses STARTTLS to switch to using TLS, then sends AUTH LOGIN to authenticate. The mail is accepted, then sent on to the final destination.

Where it ends up in my spam folder. Nothing’s authenticated yet (and we have no Message-ID, because SWAKS, and it’s 5321.From and 5322.From domains don’t have an MX, because SWAKS), so no great surprise. We’ll do a more realistic test once authentication is set up.

SMTP Authentication – DKIM

A common way – for postfix and sendmail, at least – to plug most sorts of processing into a mailserver is to use a milter. That’s a portmanteau for mail filter, and it’s an API that allows postfix to pass an email to an external process. It’s been around for a couple of decades or more, so it’s pretty stable.

That external process, the milter, can make decisions about how the mail should be routed, and it can modify the header or content.

The commonly used DKIM milter is OpenDKIM. It can both verify inbound mail and sign outbound mail, and can be convinced to sign mail from different senders with different domains and keys, making it possible to have dkim authentication align with the 822.From header.

But OpenDKIM has some open bugs – nothing serious, but some unexpected behaviour in grubby corners. Its last production release dates from 2015. Debian seem to be distributing the most recent beta release from 2018. Don’t get me wrong, it’s good enough to use in production, but it’s not a codebase I’m particularly comfortable diagnosing or modifying. So what else is available?

The usual code I use for authentication is go-msgauth. It’s solid, understandable code and it does include a milter implementation. But it’s very basic, and can only sign with a single domain, configured as a commandline flag.

Rspamd’s dkim_signing module might be another option, but that’s a huge “intended for inbound mail filtering” dependency to pull in for a smarthost. And there’s a few pieces of very nice rust code for authentication, and a milter based on one, but I’d need to build and package those myself.

I guess I’m using OpenDKIM for now.

OpenDKIM

sudo apt install opendkim opendkim-tools

This installs a basic configuration file in /etc/opendkim.conf. Just like saslauthd it communicates with postfix via a unix socket, so again I need to configure it to create that socket inside the directory postfix is chrooted to.

I’m using opendkim because I want to have aligned dkim signatures, and that means I need to sign with different keys² depending on the domain in the From: address. That’s configured using a SigningTable and a KeyTable, so I add a couple of lines to /etc/opendkim.conf:

KeyTable                /etc/opendkim/KeyTable
SigningTable            refile:/etc/opendkim/SigningTable

KeyTable is just a text file with one record per line – starting with an arbitrary name (I’m using the full hostname where the TXT record will be published, because it’s easy to remember) followed by the DKIM selector, the DKIM SDID (“d=”) and the private key to sign with, all separated by colons:

blueberry._domainkey.wordtothewise.com wordtothewise.com:blueberry:/etc/opendkim/keys/wordtothewise.com/blueberry.private

SigningTable is an “refile”, which is used to map the value in the From: header to one of the entries in the KeyTable. “refile” stands for “regular expression file”. Despite that, it does not contain regular expressions, just simple wildcards using an asterisk. Again, it has one record per line, starting with a wildcard match for the From: header, followed by the arbitrary name for the key data from the KeyTable.

*@wordtothewise.com blueberry._domainkey.wordtothewise.com

Next up, signing keys.

First we create somewhere secure to store them, as root:

# mkdir -p /etc/opendkim/keys/wordtothewise.com
# chown -R opendkim:opendkim /etc/opendkim/keys
# chmod go-rwx /etc/opendkim/keys

Then we create a key pair, a private key for signing outbound mail and a public key to publish in DNS. We have a few choices to make – what selector to use, how strong a key we want, whether the key should be explicitly only used for email:

# opendkim-genkey --append-domain \
  --bits=2048 \
  --directory=/etc/opendkim/keys/wordtothewise.com \
  --domain=wordtothewise.com \
  --restrict \
  --selector=blueberry
# chown -R opendkim:opendkim /etc/opendkim/keys

That creates two files for us. One is blueberry.private – the signing key, generated in the place we’ve already told opendkim to find it. The other is blueberry.txt – the public key, in bind format ready to add to our nameserver:

blueberry._domainkey.wordtothewise.com.	IN	TXT	( "v=DKIM1; h=sha256; k=rsa; s=email; "
	  "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0FOof0FrJBTMxNm/3KbLSnUgBwX5jVkRILFEJJltBbaH0ZqduoUau/NcjioYgDSO8ktF6f4YnUem7VjARTzkl7mnQA9qlhF0Ix0W72oL5cDd6EptuoNn88ws9nBRvDBkeSjFNo/ftvrr6wEMet93EC0mxXKZXT9jgPTAii+cXl1Jg7QkO64DySFUDAodmaBMN9mVtr8P6drO0P"
	  "sG8RxH9KfvEtLS4L1a42TB7CtydMeIGQJKW51C55cIRhLVXzZ8emdTpZ067tdYNdeHFX7WsSEa5XBIJoDE8LI8RHrYSIdUbtGqkWncy9U0yYjVPJj369Q7yBgWZiwGrjbUQBr3XQIDAQAB" )  ; ----- DKIM key blueberry for wordtothewise.com

I add that to our main nameservers, and we’ve published our DKIM public key.

There are tools for testing a milter directly, but let’s hope we can skip that and just send some email.

Plugging opendkim-milter in to postfix is just a few lines in /etc/postfix/main.cf:

smtpd_milters = unix:opendkim/opendkim.sock
non_smtpd_milters = $smtpd_milters
milter_default_action = accept

I add postfix to the opendkim group, so they can both access the socket inside postfix’ chroot, make sure the directory /var/spool/postfix/opendkim is owned by opendkim and has reasonable permissions, then restart both postfix and opendkim.

Testing with SWAKS, again

This time we have to provide an email address for the From: header, to trigger the DKIM signing.

swaks --to steve@blighty.com --from steve@wordtothewise.com \
  --server mail.turscar.ie --protocol ESMTPS -p 587 \
  --auth-user steve --auth-password mypwd

The first attempt is deferred by the smarthost. Checking the opendkim logs the ownership of the private key file is wrong, so it can’t sign the mail. Fixing that, the second attempt delivers successfully, but DKIM fails because it can’t find the public key in DNS…

Authentication-Results: mx.turscar.ie;
	dkim=fail reason="key not found in DNS" header.d=blueberry header.i=@blueberry header.a=rsa-sha256 header.s=wordtothewise.com header.b=umSfgt6s;
	dkim-atps=neutral

header.s = wordtothewise.com, header.d=blueberry? I got the two columns in KeyTable the wrong way around. Easy fix. (I’ve fixed it in this post too, as I really don’t want to leave bad configuration where someone might copy and paste it).

Now it works.

Authentication-Results: mx.turscar.ie;
	dkim=pass (2048-bit key; secure) header.d=wordtothewise.com header.i=@wordtothewise.com header.a=rsa-sha256 header.s=blueberry header.b=AJgmgJCN;
	dkim-atps=neutral

Final test, send an email to aboutmy.email – the results look good.

SPF and DKIM are aligned, DMARC passes. Mail is being sent via TLS 1.3, over an IPv6 connection. Round trip reverse DNS looks good.

What’s next?

I need to add the other domains we use for email to the signing tables, and update their SPF records to acknowledge the new server.

I need to do something about key rotation. There’s not really a great way to automate that around opendkim. It would be possible, but would require generating new keys, pushing them in to the DNS, updating the opendkim signing tables (either by editing the files or, more likely, by moving to use database backed tables). I’m going to think about that, and see if anyone has written anything to help with it.

Conclusions

You can configure a perfectly solid, best practices compliant smarthost using off-the-shelf open source components.

But documentation is sparse, contradictory or wrong. Packaging, at least in the Debian world, is a wreck. The default configuration isn’t going to provide best practices, and the bits of configuration you need to do so aren’t on the simple path.

You should strongly consider having someone else run it for you, or using a commercial (supported, good practices by default) smarthost.

And next time I’m going to try KumoMTA instead, even for a small business smarthost.

1. This is a huge oversimplification of what’s in a TLS certificate – you should really be looking at the X509v3 Subject Alternative Name – but it’s good enough for this. ︎
2. Technically I could sign with the same private key, publish the single public key under multiple domains and still use a d= that matches the domain in the From: header. ︎

(The SEO slop tends to do things like sign up for free accounts at a bunch of ESPs, send three emails to probe accounts and see how many reach the inbox at mailbox providers served by the probe account service. If you’re a reader of this blog you can probably already see why that’s meaningless, but I could write several thousand words just on Why These People Are Wrong and Why Their SEO Slop Is Bad.)

Could we do the same sort of testing but … better? Not really. You’d need to be sending real content to a real audience – not probe accounts – consistently from each ESP long enough for metrics to stabilise to get a fair rough measure. And moving that audience to another ESP would change things enough that you couldn’t really compare with the first one, let alone by the tenth ESP. And even if you did all that it would, at best, give a vague comparison that’d be valid for folks sending the same sort of content to the same sort of audience.

So, what can you do? And how much does your ESP affect your deliverability?

If you have dedicated everything

First, is your reputation going to be isolated from their other customers?

If you’re going to be sending from a dedicated IP address, and all your reverse DNS, SPF and DKIM is using your domain rather than one owned by the ESP then your sending infrastructure is pretty well isolated. If you’re using your own domain in link tracking and image loading URLs, and your own domain in any CDNs used for images, rather than one owned by the ESP then your content is pretty well isolated too.

If everything you’re doing is white-labeled to your domain then you are going to get the deliverability your content and practices deserve. The direct impact of your ESP choice is going to be minimal.

If you have shared anything

If any of the identifiers mail filters use to detect bad actors – spam, malware, phishing – are shared between your mail and mail from other customers of the ESP then things get a little trickier.

At it’s simplest, if one customer is sending bad mail from an IP address, all mail from that IP address will be treated as suspicious (and, fairly commonly, all blocked at some mailbox providers). If there’s a lot of unwanted mail being sent with using a particular hostname in links in the message then any other mail using those same links will be viewed with suspicion.

Sending from a shared IP pool is the most obvious shared identifier, but sharing SPF and DKIM identifiers are also important. And an often overlooked aspect of sharing is the URLs in the content of the message – the ESP will wrap links for click tracking, add hosted images for “open” tracking and host images on a CDN.

In this setup you are sharing to some extent the reputation of all the other ESP customers using the same identifiers.

Sometimes the result of that can be catastrophic – if another customer on the same shared pool you’re using is sending bad content, or if other customers consistently send bad content, then a mailbox provider may outright block the IP address of that shared pool. Suddenly none of the mail from that shared pool is delivered to that mailbox provider, including yours.

Sometimes it can be less obvious. If a significant fraction of mail being sent using the ESPs shared click tracking domain is unwanted, then use of that click tracking domain will be recognised as a sign of unwanted email. Real spam filtering is usually more complex than a SpamAssassin score, but if you think of it as “sends from this ESP” adding three to your emails SpamAssassin score that leaves you a lot less slack to avoid hitting a score of five and ending up in the spam folder.

Most of this filtering tracks behaviour over time. A single bad send, or even a single bad customer is unlikely to be enough to impact delivery for other ESP customers (though it could cause IP blocking of the shared pool in some cases, at some providers). But an ongoing pattern of low quality mail from an ESP will affect delivery of all their shared customers.

Most large consumer mailbox providers are pretty good at separating mail streams from shared IP pools, so that even if you’re using shared IPs and shared domains they can mostly distinguish your email stream from those of your ESPs other customers. If you’re a B2C sender this isn’t really something you need to care about at the Gmails, the Yahoos and the Microsofts. It’s something you’re more likely to see issues with at smaller providers, or with enterprise filters.

So what should I be looking for?

Some folks may, at this point, say “Well, obviously I should be demanding a dedicated IP!”.

Dedicated IPs are expensive resources. If you’re sending huge volumes of mail then you need one, for network engineering reasons. But if you’re sending less…

There’s probably no real threshold of volume below which you absolutely shouldn’t be using a dedicated IP, but if you’re sending small amounts of mail – less than a million a week, perhaps? – building initial good deliverability on a dedicated IP may be harder.

Also, shared reputation across customers isn’t always a bad thing. If you’re one of a hundred customers in a pool and you do a bad thing, like sending to your suppression list, then the impact of that will be mitigated by the other customers using the shared identifiers. As long as they’re all good customers, mostly sending good email they’ll be supporting each other.

If you’re a smaller sender, you’re probably happy on your ESPs shared pools and – as long as most of your ESPs customers are good senders – will be able to build and maintain excellent deliverability.

You might still want to use your own domains for authentication and click tracking. That’ll isolate your reputation somewhat from other customers using the pool, and it’ll make it easier to move the good reputation you’ve built to another ESP. So if the ESP offers white label DKIM and SPF authentication that’s good. White label click tracking is nice to have too.

Are my ESPs other customers good?

This is an important thing to find out, but it’s really hard to measure directly. Spam filter operators will have a fairly shrewd idea, but that’s not data they’ll share (and it’ll change over time). You could check some of their IP addresses or domains on one of the multi-blacklist lookup sites – but most of the lists they check aren’t at all useful, and even for the few that are they only provide a snapshot of whether the ESP is listed now – and even some of the best ESPs may occasionally get listed.

But we can get some hints as to whether they allow customers to send spam from their network. Check their webpage. Is there a link to their acceptable use policy or terms of service? Does it forbid users from spamming? That might be phrased as requiring evidence of consent, requiring permission, an explicit ban on purchased lists, banning sending unsolicited messages, or banning use of third-party lists.

If you can easily find their AUP, and it forbids their customers from sending spam that’s a good sign that they don’t encourage spammers on their network. We maintain a list of ESPs that require good practices if you’d like somewhere to start. If you can’t find it, or it allows spam then that’s a bad sign.

Clearly stated policies are a good start, but whether they’re enforced matters too. That’s hard to measure, but a critical part of policy enforcement is having staff to do the work, and the infrastructure needed to do it.

If you’re a potential customer you might send an email to abuse@your-new-esp.com explaining that you’re evaluating them, and can they send you a link to their acceptable use policy.

If the email bounces, the ESP doesn’t have a functional abuse desk. If it returns an autoresponse saying the mail has not been read, they don’t have a functional abuse desk. If it requires you to fill out a web form, they don’t have a functional abuse desk.

550-5.7.1 Please do not reply to this email. If you wish to reply, please click
   550-5.7.1 on this <link> to update your
550-5.7.1 ticket, track the status, or close your ticket directly via the
550 5.7.1 console. – gcdp 5614622812f47-3f3af4a1d55si1924262b6e.281

This is really not a good sign

If it just vanishes and you never hear back, or you get an automatic response and nothing more that’s not a great sign, but it might just mean their abuse desk is overwhelmed and doesn’t have the capacity to reply to email, or is forbidden by policy from doing so. And sometimes it means that they accept and discard all mail sent to their abuse alias (yes, I’ve been told by abuse staff at very large providers whose name you’d recognise that they do this). It’s hard to distinguish between those cases, unfortunately.

If you get a response with useful information, whether it be boilerplate or from a human being that suggests there’s a functional abuse desk there, with the resources and policies to do something. And that’s a really, really good sign that their customers are probably, on average, over time sending mostly wanted email. And that’s what you want your co-customers on an ESP to be doing.

Indirect ESP effects on deliverability

If we’re concerned about deliverabilty there are some things we’d really like an ESP to do for us that aren’t directly connected to their ability to deliver our email to the inbox.

Most of the time the reason we’re having problems delivering email is because of something we’re doing. Identifying why, and fixing it, requires some resources from the ESP.

Do they have in-house delivery expertise to help? This is expensive to offer, and likely not available to small customers but can be extremely valuable.

Do they provide access to any third party delivery monitoring services, such as probe accounts? The value of these is arguable, but if there’s easy access to that additional data that’s good to know.

Does their reporting of delivery data provide what you need to diagnose a problem? Does it let you track metrics over time? Does it let you break down metrics by recipient mailbox provider (even at the level of gmail / google apps vs office 365 vs yahoo vs everyone else)? Does it let you get the actual rejection messages for bounced email?

It’s not all about deliverability

While deliverability is a critical aspect for an email campaign – if the mail doesn’t make it to the recipients inbox then every other aspect you worked on is wasted – it’s definitely not the most important measure for an ESP. Not least because deliverability is mostly driven by your behaviour, not the ESPs.

Ease of use, decent automation, segmentation, responsive customer support, great templating, A/B testing, reporting, integration with other tools you use and just “are they going to be a pleasant company to work with” are all at least as important.

The Public Suffix List is a manually maintained list of domains that’s mostly used to help web browsers prevent data stored by one web site being seen or modified by another. As most websites use cookies to secure user logins that’s critically important.

Loosely, the Public Suffix List allows a browser to know that login.blighty.com¹ and images.blighty.com are operated by the same owner, and should potentially have access to cookies set by the other. And, conversely, that login.example.com and login.example.org shouldn’t have access to each others cookies.

As well as traditional “top level domains” like .com or .ac.uk the public suffix list also accepts submissions from domain owners who want browsers to enforce this sort of segregation between their subdomains. A common example is a service where customer maintained web content is available at customer specific subdomains, perhaps for personal blogs where “https://steves-blog.example.com/” is a more saleable service than “https://example.com/~steves-blog”.

If you’re offering this sort of service you should think about registering the domain you host your customer content under with the public suffix list. It adds significant security between your users.

But

But. If you add your domain you might make it unusable for use in email. Perhaps forever.

How so? DMARC uses the public suffix list as part of its algorithm to decide whether two hostnames (such as the one in the From: headers and the d= in the DKIM signature) are aligned. If they’re not aligned, DMARC will fail.

And if you add example.com to the public suffix list, example.com will not be considered aligned with anything. And immediate subdomains, such as a.example.com and b.example.com won’t be considered aligned with each other.

That means there’s no way to get SPF alignment if you have “From: <hello@example.com>” as your from address. Normally you’d use a subdomain (e.g. bounces.example.com) in your return path, allowing your ESP to handle bounces and still get DMARC alignment. But that won’t work now. Oops.

DKIM is only going to be considered aligned if the d= domain is identical to the domain in the From: header, so use of d= subdomains to identify different mail streams is right out. And, depending on how the recipients’ DMARC checker is implemented, even mail with identical From and d= domains might not be treated as aligned.

As DMARC moves towards a required part of email this gets worse.

Forever?

You can remove a domain from the public suffix list – but it’s not a list that’s queried in real time. Rather it’s downloaded once by a software developer, compiled into an efficiently searchable data structure. That’s then compiled in to a piece of software and sent to users.

Maybe the developer will update the copy of the PSL they use for each release. Maybe they won’t. They’re probably using an upstream library rather than implementing it themselves, and due to dependency management even if the upstream library is updated the application using it may still use the older version of it. Perhaps for years, perhaps forever.

And even if the application is updated, that doesn’t mean it’ll be updated on ISPs machines immediately.

This isn’t theoretical. I’m seeing DMARC failures at Gmail that I can’t explain any other way for a domain that was removed from the public suffix list nearly a year ago.

This seems bad

It does. Piggybacking email authentication onto a browser cookie security infrastructure may not have been the best idea.

There is work being done to move DMARC away from using the PSL, rather using records added to DNS. It’s not clear when, or if, it’ll be universally supported but it’s likely to be used by large mailbox providers, at least, eventually.

The nice folks at the public suffix list use git to track changes to the list, so if you ever need to check whether a domain has ever been on the public suffix list you can do this to see all the changes:

$ git clone https://github.com/publicsuffix/list.git
$ cd list
$ git log --follow -p -- public_suffix_list.dat | less

1. I use blighty.com as an example domain a lot, as it’s a domain I own and it’s a bit more interesting than using example.com everywhere. Maybe I should let friends volunteer their domains for tuckerization? ︎

My ESP provides a dashboard of spam complaints. How should I be looking at the data? Are some complaints more important than others?

The spam complaint dashboard is a record of the feed back loop messages (FBLs) that an ESP has received about a message. Those messages are sent by the ISP when the recipient marks the message as spam in the user interface of that provider.

Complaints should be viewed as a percentage of the messages that were delivered to the inbox at that provider. So if you send 50,000 emails in total and 1000 go to libero.it, and you get 5 complaints from libero.it, then your complaint rate is 0.5% not 0.01%, assuming 100% inbox delivery.

Complaints are a part of the filtering and reputation system used by the recipient ISP to filter mail and determine how future mail will be delivered. The ISPs only care about complaints for the mail they see, they don’t care about complaints from any other provider.

You should have almost zero gmail.com complaints, because gmail doesn’t send complaints back to anyone. Sometimes you will occasionally see gmail.com complaints from Yahoo when mail is forwarded due to the ways Yahoo manages their complaint feedback loop. The only complaint data Gmail provides is the percentage of complaints and, sometimes, a identifier string in the Google Postmaster Tool interface.

ESPs use complaints to determine if their customers are violating their AUP and if they need to have action taken against them. In that respect, the “weighting” depends on the different policies of the ESPs (and they’re often not public).

It’s generally accepted that complaint rates over 0.3% are bad and that complaint rates below 0.1% are acceptable.

A couple things to note about complaints:

• Not all ISPs provide feedback loop emails to senders including some of the major broadband providers outside the US, Apple Mail and Gmail.
• FBLs are solely for mail to consumer domains. Microsoft has a full FBL infrastructure built into O365 and their consumer but only send FBLs for mail to their consumer domains.
• Not all “this is spam” buttons are tied to a FBL. Apple mail users, for instance, have a “junk” button but it only affects filters for that user and does not trigger a FBL complaint to go back to the ESP.
• Not all ESPs pay Validity the exorbitant amount they’re charging for FBL feeds.
• Complaint rates as viewed by the ISPs are different than the complaint rates as viewed by the ESPs. ISPs will always have a more accurate view of complaints.
• Users are not permitted to report mail in the bulk folder, so a lack of reports for senders may mean there are already delivery problems.
• Complaints are very noisy for small senders as users can sometimes report spam by mistake or incorrectly use the spam button instead of delete or trash.

Overall, complaints are a great way to monitor what your recipients think about the email you’re sending them. For ESPs and compliance desks they’re a good way to monitor which customers may have issues that need to be addressed before their mail is spam foldered or blocked at the ISPs.

When you’re diagnosing and mitigating hard bounce rates the first thing you’ll see is your ESPs dashboard or reporting. It’ll tell you how many emails you sent bounced, and how that number has changed over time. A useful start.

But a hard bounce can be caused by any of a dozen or more reasons, and they’ll imply different causes for the issue, and very different approaches to mitigation.

The first thing you want to do is to find out details of what’s causing your bounces, by getting hold of the actual rejection messages the recipient ISP returned. They might be available somewhere in your ESP reporting, or you might need to reach out to your support contacts to get them. But an ESP will have access to them (if they don’t … think about looking for a better ESP).

They’re easy to recognise – they’ll start with a three digit number, typically starting with a 5 (or maybe a 4). After that will be some human readable text, and maybe a link to a web page.

The human readable text will describe why the email was rejected. If there’s a link that page will give you more information about the recipient ISPs expectations, and what needs to be improved.

That might be enough for you to understand what’s going on, and if it’s not then those rejection messages are what any expert you ask for help is going to want to see.