Postmark Blog

Upgrading and Rotating DKIM Keys

cnagele — Fri, 24 May 2013 10:12:01 -0400

As many of our customers know, email delivery is the core of our business. A big part of ensuring that your emails get to the inbox is email authentication, such as DKIM and SPF. A recent change in Gmail/Google now requires that DKIM keys are at least 1024-bit in length. When Postmark first launched a few years ago, we started with 768-bit keys, which now results in Gmail showing DKIM as “weak” and displaying a “via” tag. Fortunately inbox rates are not affected.

As a temporary solution, we’ve been slowly upgrading keys manually for customers. It’s been a painful and very manual process. In an effort to make it easy to upgrade your DKIM keys, we’ve created a new process that allows you to automatically generate a new (and stronger) key pair for your domains. We worked hard to make the process as easy as possible.

Generating a new key pair

If your account contains “weak” keys you already received an email from us with instructions on how to generate a new key pair for each domain. If not, the process is pretty easy. Here are the steps:

1. Go to your Sender Signatures page and look for the “Weak DKIM” badge. Click on the “Setup” link.

2. Click on “Generate a new key” below the existing DKIM key

3. Create a new subdomain host and insert the TXT value in your DNS. Note: The selector is different.

After that, Postmark will check if the new key is visible in DNS. When it is, the new key will be active and your messages will be signed with the new, stronger DKIM key. The process does not require downtime or interruptions with DKIM signing.

What about newly created sender signatures?

If you created a sender signature in the last four months there is no need to update to a stronger key. Any new sender signatures already have the strong 1024-bit keys. The only domains you should worry about are the ones listed as “Weak DKIM” in your Sender Signatures page.

Should I delete the old DKIM record in DNS?

DKIM keys are just like any other key based authentication, so we do recommend revoking old keys. After a few days of the new key being active, it is safe to delete the old record from DNS or simply delete anything after the “p=” in the TXT value.

Rotate your keys quarterly

When we decided that customers needed to upgrade their keys we could have simply generated new keys and left it at that. However, according to the DKIM standard, it is recommend to rotate your keys quarterly. This new process in Postmark allows you to easily rotate keys with zero downtime in message signing. We didn’t want to just fix the problem, we wanted to improve the process.

Updating the official Ruby libraries

achistyakov — Wed, 22 May 2013 09:42:00 -0400

We’re happy to announce a big update for both of our official Ruby libraries: postmark 1.0 — Ruby developer’s little helper — and postmark-rails 0.5.0, specifically crafted for Ruby on Rails and ActionMailer.

Keeping up with the latest improvements

Recently, while writing documentation for the Postmark Heroku Add-on, I had an opportunity to play with Postmark libraries for different programming languages and frameworks, built by our awesome community. It didn’t take too long to realise that we might learn a lot from them. When we first released the gems in 2009, our goals were pragmatic: by integrating with the TMail library we targeted Ruby on Rails projects and the other software already running in production. Later we added more features and support for the Ruby Mail library, which now finally replaced TMail. Still it was clear that at some point we’d need to rethink what our customers want from the libraries. At that same time, the Ruby community evolved peculiarly fast: new tools and gems appeared, became popular and faded out quickly. So we took time to focus on Postmark development, while only issuing bug fixes for the libraries.

So, what has changed in Ruby since 2009? First, the tools and the whole infrastructure have significantly matured. Bundler, RSpec 2 and the new language features have brought the developer’s experience to a new level. Thanks to the amazing Travic-CI continuous integration service, the value of automated testing has grown even more, although it had always been particularly high in the Ruby community. Practicality of Ruby development has moved further with automated monitoring of projects’ code quality provided by Code Climate for free to all open source projects. The other trend is the uprising demand for concurrency support, popularised by JRuby’s real threading, Celluloid and Sidekiq. I’m glad to see how the influence of functional programming languages is affecting the Ruby community, which now favours pure data structures and tries to avoid unneeded abstractions and reduce the amount of mutable state in code.

The goal was not to just bring all those amazing new things to the libraries, but to see how it affects the real value, how it makes our customers’ happy. Whenever you think about the architectural changes, backward compatibility is always a concern. We were able to keep the public interfaces fully compatible with the legacy versions, while changing almost everything behind it. The new architecture is ready to be used by concurrent applications: whenever the old version failed or used memory locking, the new one allows you to take the most from the multi-threaded approach.

Better integration and more features

While Postmark as a service has always tried to simplify the way you send your application’s emails, the postmark gem required you to additionally install the Ruby Mail library, which was designed to work with traditional email interfaces. It’s no longer true: the gem now allows you to send emails right away, as easy as using the pure ruby hashes.

While making sure the new libraries are architecturally tidy, we introduced the support for some lesser-known features of the Postmark API. Now you can send emails in batches, work with bounces and change a server’s info easily. We also added a way to give Postmark’s inbound JSON some Ruby flavour, so the keys are underscored symbols instead of camelized strings. Check out the README file for more examples.

The biggest challenge was to properly test the new libraries and take advantage of the Travis-CI service. After adding a sufficient unit test suite written in RSpec 2.0, I took advantage of the postmark test api key, which allowed me to interact with the real API in the integration tests. This way it doesn’t unveil any sensitive information or require fake HTTP requests to be used. As a finishing touch I added Code Climate monitoring for the libs, and they both received the highest score of 4.0. Of course, that result can not be fully trusted, and personally I see a lot of room for improvement, but it definitely makes us proud of the achieved results.

We’re sorry it took so long, but we sincerely hope you’ll love the updates now that they’re released. Please file issues on Github or email support if you face any problems with the updated libraries. We’re always open for contributions, and if you’re willing to add something to the libraries, we’ll be happy to accept your pull requests.

Automated Testing at Postmark

ibalosh — Tue, 16 Apr 2013 10:47:00 -0400

With Postmark, a day doesn’t go by that I don’t think to myself how simple it is to use it and how fast you can start sending emails.

Simple, easy, fast - three words which are our top priority goals. Achieving all three, especially sending emails fast comes with a price: reliability. From day one, our goal has been to deliver email to you reliably. Losing emails or emails not reaching your inbox has never been an option.

We have been working very hard to maintain this. My main goal today is to share with you what I do, as a tester, to make sure Postmark is doing its job.

The tools we use

Before I go into further detail, let me share with you which tools we use to test Postmark:

Selenium (Selenium Grid, Selenium WebDriver + RSpec)
Jenkins
Statsd/Librato

Writing the first tests

I have been writing automated tests for Postmark for a while now. The first tests I wrote are the ones which we run most frequently.

When creating the test suite, I decided to work first on the most important test cases, from which myself and the team will benefit the most.

For Postmark, this was a no brainer. The most important thing to test is sending. First I wrote the tests which covered sending email by SMTP and the API. These tests are sending emails and check whether they reach the inbox (or not).

We are tracking how long it takes to send and receive the email. If we see any slowdown, the tests notify us right away. These sending tests run every 5 minutes.

Extending the test suite

Once I was sure that we covered all important test scenarios for basic email sending, I started extending our test suite with Inbound tests, Bounce API, and UI tests.

Running the tests

As soon as I wrote the first tests, I wanted to have a way to run them as frequently as possible. One of the options we considered was running them with SauceLabs. SauceLabs is a great service to run your tests in the cloud with rich reports and the option to run the tests in most of the Browsers/Platforms you can think of.

The only problem with this is we found SauceLabs would not be cost effective for us, since we have to run parts of the test on one of our machines frequently. It made the most sense to have a dedicated machine for testing.

Russ, our Sys Admin setup a dedicated Linux machine on which we installed Jenkins to run Selenium tests. Jenkins is a great CI tool, used a lot these days by other software testers too. We decided to use Jenkins on this machine solely for my tests.

Execute tests as frequently as possible

The main idea was to run the tests as frequently as possible so we are running Selenium tests all throughout the day. This allowed us to gain much more feedback from Postmark compared to running the tests only when there are changes being introduced.

Why do we run the tests all the time? Our developers have created a lot of tools that monitor Postmark’s health all of the time, but its always better to have two sets of eyes looking at our web application, and Selenium tests act like real users more than any other part of our test suite.

We can sleep soundly knowing that emails are being sent, activity pages are working, and our customers can use all of the features in the UI. As a bonus, the Selenium tests generate useful data in the test account we are using, which can help in manual testing.

To allow faster running of our tests, and continuous running of top priority tests, we use Selenium Grid and Jenkins nodes to be able to run tests in parallel. The sending tests run on separate nodes in Jenkins. The tests that check the UI are running on separate selenium ports.

Statistics

Since we are running the test all of the time, there is a possibility to track much more information about our tests than whether they are passing or failing.

My tests are checking sending, search, activity, bounce api, and inbound processing. So why not have statistics for all these tests?

Thanks to Chris, I found out about Librato and we decided to integrate the Selenium Tests with Librato. This allowed us to have a complete performance history of search, bounce api calls, and inbound processing.

Summary

The regression tests are working hard every day on our testing machine. They are a third eye, watching over Postmark constantly. They helped us out numerous times in preventing issues and even foreseeing issues.

Of course, there is always a room for improvement, and I am eager to hear what do you think, what is the process/workflow you are using to test software.

Designing a modern email

efedorenko — Thu, 11 Apr 2013 15:32:00 -0400

It’s not unusual to have a flashback to the Netscape Navigator 4 and Internet Explorer 5 days when working on an HTML email. The quality of rendering engines is totally inconsistent, most modern development techniques are unavailable, and even images – an essential element of many emails – are turned off by default in many clients. This can feel like 1998, but the web development community has learned a lot since then. Strategies like progressive enhancement and modern tools like Litmus can help us build HTML emails suited for today’s Babylon of inconsistent desktop clients, various web clients, tablets, smartphones, and high resolution displays.

Last month we decided to start sending educational emails to new Beanstalk customers to familiarize them with our features. The plan was to send a total of three emails with a set interval between them. After designing these emails I decided to explain my process and go through the tools and techniques I used.

Working on layout

We decided that each educational email will be written by a member of our team. Chris, founder of Beanstalk, wrote about deployments, Ilya, one of our lead developers, introduced users to advanced tools, and Russ, our systems engineer, explained our approach to security.

I started with a simple design but wasn’t really happy with it:

Emails written by our team members were personal, but the design didn’t reflect this. It looked as if the email was coming from a company, not the person who actually wrote it. So we decided to put the authors in the front seat while still referencing Beanstalk, and metadata is the perfect place to do this:

Now that its clear that the email is from Beanstalk I simplified the look, added the author’s photo and removed the logo:

This looks and feels much more like a personal letter written on the product’s stationery paper rather than a corporate brochure – exactly what I tried to achieve.

Bringing responsiveness to emails

To figure out which email clients are the most popular between our users I checked Campaign Monitor statistics from our latest newsletter. The results are impressive, but not too surprising for a geeky product like Beanstalk – 30% used Apple Mail, 23% iPhone, 19% Gmail, 6% Sparrow, 5% iPad, 3% Android and the remaining 14% is all other combined. This means that a third of our recipients use devices with smaller screens and our email may not be readable to them.

In normal view our text column width is set to 600 pixels, which limits the number of characters per line to about 85-90 – a top boundary for good readability. I started with adding the @media query for screens narrower than 780 pixels to allow for some whitespace and the green border around the text:

@media only screen and (max-width: 780px) {
    div[class="bs-email"] {
        padding-right: 20px !important;
        padding-left:  20px !important;
    }
    div[class="bs-email"] .bs-wrap {
        width: 100% !important;
    }
    div[class="bs-email"] .bs-content img {
        max-width: 100% !important;
        height: auto !important;
    }
    ...
}

Here I decreased whitespace around the text, let the text column and images take all of the available space and moved the sender photo inside the column:

Next is a version for small screens, like the ones on smartphones. I added another @media query for max-width: 380px after the first one, where I decreased whitespace even more, removed the green border around the email and set a smaller line height to fit more content on the screen:

Using @media queries in emails with old table-based layout still feels unreal to me, but they‘re well supported in new devices and old clients just ignore them. The only exception I can think of is Yahoo! Mail, which ignores the @media declaration itself but applies all included styles to your normal view. To fix this I use attribute selectors like div[class="className"] or h1[id="unique"], which it ignores too. Speaking of hacks, I can’t recommend HTML Email Boilerplate enough as a starting point in email coding – it’s a great collection of workarounds and fixes to rendering problems in major clients.

High resolution displays

What do people checking emails on iPhones, iPads and Androids have in common? Most of them look at high resolution displays, like Apple‘s Retina. Adding hi-res graphics to email is tricky as many of these people may be on a slow and expensive 3G connection, and nobody will wait a minute for the email to load.

My first take was to replace img elements with divs and use original images as backgrounds, while providing a hi-res version through the @media query only when it is actually needed. This is a hack from any standpoint – divs have no semantic value, they don’t provide alt text when images are disabled, and some old clients just won’t display background images. Even worse is that Gmail and other clients without @media support will show original image on hi-res devices, like MacBook Pro with Retina screen.

This is when I decided to go with the simplest available solution – just create all images twice as big and downsize them with width and height attributes of the img element. This solution works everywhere from Gmail to iPhone, but the downside is increased file size. I used ImageOptim and ImageAlpha rigorously to trim all extra bytes from our graphics and screenshots, and in some cases went with overcompressed JPEG instead of PNG. The total size of an email with 3-5 big images is about 100-150 KB, which is a lot but still not terrible, especially considering we don’t use any graphics in the design (except of logo in the bottom) and images appear only after a few paragraphs of text.

Testing emails

Any testing starts with sending an HTML email, and it’s not as easy as it may sound. First, you need to host all images and assets somewhere so they can be linked from HTML. Second, you need to turn your HTML file into an email and send it. One option is to create a free account on Campaign Monitor or MailChimp and send email previews to your own email address – this works well, but the process can be a bit slow. My new choice and a recent discovery is the Direct Mail app for Mac – it’s free for up to 50 emails per month, beautifully designed and works really well.

Litmus is another favorite of mine. I checked it before but wasn’t sold on the idea of testing and debugging emails based on screenshots, so I always maintained my own library of email clients and accounts. Things changed when they released Interactive Testing which allows you to test, troubleshoot and preview emails in real-time. Not every client in their library has this tool yet, but even the limited selection saved me an enormous amount of time.

Gathering data

I’m risking sounding like a Litmus promoter, but another feature we used for the first time in our transactional emails and totally loved is Litmus Analytics. We generated unique tracking codes for every type of email we send, so now we have a solid set of data on our recipients and their reaction to each of the emails we send. The way they collect this data is some sort of magic, but seeing engagement reports, email clients statistics and even the number of forwards is really empowering. With real data at our hands we can see which emails our users like the most and change our communication accordingly.

Hopefully you found something new in this walk through of my design and development process for emails. Know a better way to do something or have a tool recommendation? Let me know in comments!

Postmark Heroku Add-on (Beta)

cnagele — Wed, 03 Apr 2013 11:37:00 -0400

You can now easily send and parse emails in Heroku using Postmark. We’ve been wanting to do this for a long time, and thanks to the efforts of Artem on our team, we finally put it together.

If you use Heroku, or want to use Heroku, there are only three simple steps to using Postmark’s Heroku Add-on:

Sign up for the Heroku beta program.
Add Postmark on the add-on page or run in the command line:
$heroku addons:add postmark:test
Follow the documentation to get started.

Once the Postmark add-on is installed, you will be able to send 10,000 emails for free each month through your Heroku application. During the beta, you will not be able to send more than 10,000 emails per month. Once the add-on is out of beta, we will announce the pricing plans on Heroku that will take effect once you exceed the 10,000 limit.

If you use Heroku, or want to, we’d love your help testing the new add-on. Let us know what you think or if you need help getting started.

Globally distributed SMTP servers. Lower latency, more redundancy.

cnagele — Tue, 02 Apr 2013 15:30:00 -0400

First and foremost, Postmark is an infrastructure product. It’s our job to send your emails and get them to the inbox, with minimal or no downtime so you can rely on us. After that, everything is secondary. This past Saturday, we launched SMTP servers in four new data centers around the world in addition to our Chicago data center, including:

Sydney, Australia
Dublin, Ireland
San Jose, California
Ashburn, Virginia

Focused on lower latency and more redundancy

When your app talks to Postmark, latency can cause problems for both background processes and your customer experience. For instance, imagine that your app is hosted in Australia (a bunch of our customers are). When your server connects to our SMTP servers in the US, the latency can be up to three seconds for a single email due to the long trip and chattiness of the SMTP protocol. Three seconds is a lot, especially if email is not being sent in the background.

The only way to solve this is to connect to a server that is closest to your application server. In the example above, if the app was connecting to a server in Australia, the latency would be less than 400ms, including all of the auth and chattiness of SMTP. So this is exactly what we did.

Since this weekend, when your application connects to our SMTP servers, it will automatically hit the closest server to you. If your servers are in California or the west coast, you will hit our SMTP servers in San Jose. If they are in Europe, it will hit our SMTP servers in Ireland. And the same goes for the midwest US, east coast US, and Asia-Pacific.

Even better, since we have so many servers distributed globally now it provides more redundancy. For instance, If our servers in Australia go down, it will automatically failover to our US servers again. By having servers in every region, we can balance load and determine our heaviest hitting regions. Not surprisingly, we can already see that US East Coast is a heavy hitter due to so many apps being hosted in AWS East. That’s actually great, since the new SMTP servers we host are also in AWS, meaning even lower latency.

How did we do it?

It’s pure DNS magic. Our DNS will detect the source of your app servers and automatically route it to the closest region. In the event of a failure in a region, it will failover to the US to continue accepting emails. We were able to get all of this done in about 6 days of work by using AWS and Chef to quickly spin up new instances.

What about some API love?

Global support for the API is right around the corner as well. SMTP was just easier, but since we have more people who use our API than SMTP, we want to launch global support for the API very soon.

We hope you like it! I’ve been wanting to do this for a long time and it’s nice it all came together so fast. Big thanks to Russ and Igor on the Wildbit team for getting all of the servers ready and tested.

Our Recent Activity Issues

jpcancook — Tue, 19 Mar 2013 17:21:00 -0400

Over the past week and a half, Postmark has suffered numerous outages and delays on our front-end activity feed. While these issues have never negatively affected our API, sending, or the delivery of messages, it has caused a lot of confusion for our customers and for that we are profoundly sorry.

About 10 days ago we performed a fairly routine move of our physical Cloudant (Big Couch) servers at ServerCentral to a new cabinet. We have four total servers which are managed by Cloudant so it’s perfectly feasible to move them, one at at time, and then bring them back into a cluster. Meanwhile, the front-end activity feed is powered by ElasticSearch which uses the couchdb-river plugin to send user message data directly to ElasticSearch for indexing. 99% of the time this works extremely well.

After breaking apart the Cloudant cluster and putting it back together, however, the river plugin seemed to stop working and we could no longer update our front-end activity effectively. After several attempts to repair the river and existing ElasticSearch index, we opted to rebuild the index from scratch using a more reliable ElasticSearch setup.

Another issue that manifested at the same time was duplicate statuses showing up in activity. About 10% of the time, messages would show in activity with both “queued” and “sent” statuses. While this didn’t mean users’ messages were being delayed, it did mean confusion for anyone trying to decide if their messages were actually being sent.

Since that time, we’ve been working hard to restore all services back to their normal working condition. We found out that the duplicate statuses are being caused by our application trying to update the status of messages from “queued” to “sent” before the “queued” message was finished being written to disk on the Cloudant server. It’s important to have a quorum of writes across the cluster so that we can guarantee redundancy for message data. Unfortunately the writes haven’t been happening quickly enough and the result has been messages marked as both “queued” and “sent”.

To rectify the duplicate status issue, we began bonding the network interfaces in our servers to provide more bandwidth. As well, we’ve been updating the application code to guarantee documents are written to Cloudant before we try to update them. During the bonding process, however, we had some hiccups in the cluster which again caused our ElasticSearch index to fall out of sync.

We’re currently rebuilding the message index and expect to have that finished and working with 100% accuracy inside of 24 hours.

Regardless of the technical issues we’ve had with our front-end activity recently, we consider our recent delays and data inaccuracy to be completely unacceptable and wildly outside of the performance standard we strive for each day. The problems have gone on much longer than we anticipated and frankly, we find it a bit embarrassing.

We expect to have the problems resolved permanently over the next 24 hours and would like to offer our sincerest apologies to our users for the haphazard issues over the last 10 days.

- The Postmark Team

Spamhaus reminds us to keep our lists clean

natalienagele — Tue, 12 Mar 2013 13:48:00 -0400

A few days ago The Spamhaus Project published a response to some observations and assumptions made by large ESPs over what they felt were new ways of handling spam traps. Right around the holiday, big senders noticed that transactional emails were causing their IPs to get listed with Spamhaus.

What’s the Spamhaus Project?

For those unfamiliar, Spamhaus is a very well respected and far-reaching Spam monitoring organization. They monitor IPs and domains for sending spam and provide this data to ISPs as well as law enforcement. Years ago when we still ran Newsberry we got listed once. Our emails started bouncing with Gmail, Hotmail, and almost all other senders (large and small). It was a terrible situation, one I hope never to have to deal with again. Spamhaus is always right, though. Our customer wasn’t behaving and we got blocked for it.

How Spamhaus catches bad senders

Spamhaus uses spam traps as one of it’s ways to catch any sender who doesn’t practice good list hygiene. A spam trap is an email address that exists for the sole purpose of catching spammers. It has either been retired by someone, or it can be a commonly mistyped domain (gmil.com) that has been picked up by spam services like Spamhaus. You should strive for never having one of these in your list because they never asked to hear from you. If you send to many spam traps, there is something really wrong with your acquisition practices, or you’re not properly retiring emails.

How Postmark ensures good list hygiene

When we started running Postmark, we always knew that we would be strict with our handling of bounces and spam complaints. There are some mixed opinions on how many times you should let an email hard bounce before you deactivate it. We always took the stance that if an email hard bounced once, we don’t send to it again. Postmark will automatically disable the email and we won’t let you send to it again, unless you manually intervene.

Of course there can be false positives, but for the most part, ISPs and mail servers respond properly with hard bounces. If an email doesn’t exist, why send to it two more times? In the three years of running Postmark, we have often had to explain or defend ourselves to customers why we do it this way. We know it can be annoying reactivating false positives but it pays off in the long run.

In marketing emails there are tons of best practices on how to keep your list clean of spam traps. The trend is quickly moving towards measuring engagement. This means keeping track of how users interact with your emails and continuing to email those that actually open or click links in your email. In transactional email there are far fewer options for those kinds of measures. So what you HAVE to pay attention to is bounces and spam complaints. If you’re sending a transactional email to an email that is bouncing, disable it.

Summary

It’s really enlightening to read the response from Spamhaus. All of the senders they listed were sending transactional email to spam traps, over and over. Even more crazy is that these email traps actually responded with a bounce, and still they were never looking at the SMTP responses or engagement. It really is fantastic proof that you should pay very close attention to all of your mailings. And it also shows how important it is to separate marketing and transactional email. You’ll see even Spamhaus reacts differently to spam traps hit by marketing emails vs. transactional email. They are more lenient to transactional mailings because they understand that typos happen.

Last thing I want to mention is that as a Postmark user, you can and should fix these issues. Use our Bounces API to keep track of bounces. Use our web hooks to notify customers in your own applications to fix their typos so they receive your email. Transactional emails are your lifeblood, let us help you get them where they need to go.

Bye, MongoDB. Hello, Cloudant!

cnagele — Thu, 06 Dec 2012 11:51:00 -0500

The past seven days have been a whirlwind. I know a lot of our customers have noticed some issues with activity and sending the last few days after our migration. I want to give an update on what happened and what we have to look forward to.

First, why we moved to Cloudant

We’ve been using MongoDB ever since we released the activity feed, which was only a few months after we launched Postmark. It’s been a long and rough relationship to say the least. Initially the problem was basic instability back before MongoDB had replicasets. After that, it was search issues and disk access. Our dataset is pretty big and constantly growing. We solved that by moving all search and queries to Elastic Search (ES). It worked incredibly well, even though ES itself is young, we had to build our own oplog to ES follower, and ES has some of its own quirkiness. Over time, we got MongoDB to perform well by throwing a lot of SSDs at it and everything became pretty stable. At that point we still had some issues and bugs that would appear in MongoDB and we were not big fans of the master/slave architecture, so we decided to move on.

We evaluated a lot of options, such as Riak, CouchDB, Cassandra, and even MySQL. We were looking for a solution that would allow us to scale easily (by just adding a node) and something that fit in with our existing infrastructure (Elastic Search). About a year and a half ago we had talked to Cloudant (a Postmark customer). Even though at that point we were not ready to migrate (this was before ES), we really liked the offering. This time around, we decided that Cloudant was a perfect fit. Not only did it perform well in our tests, it was a drop-in replacement when it comes to Elastic Search and our data model.

On top of that, nosql and the shift to new data stores is overwhelming and pretty unpredictable. They are all fairly new and far from the stability people are used to from MySQL and other traditional relational databases. This is exactly why we chose to go with Cloudant. Their level of support is incredible and their expertise cannot be matched. When we were in the thick of it, we had a team of people from their engineering group in chat helping us. And not just with cluster health, but how to interact with the cluster in our code as well.

With Cloudant we were able to achieve a high level of customization. This allowed us to splurge on hardware like 8TB of SSDs and Intel Ivy-Bridge-EP processors. The servers are hosted at a Dupont Fabros facility outside of Chicago, one of the most advanced in the country, through Server Central.

A huge rewrite

At first the migration was pretty simple. To put it simply, we essentially just replaced how we interact with MongoDB and put Cloudant in its place. As we got into it though, the performance was not what we required and we noticed a lot of areas for improvement. For instance, in our previous architecture if MongoDB had high load, the entire sending process would slow down and queues would go through the roof. Postmark is kind of like a firehose in that respect. Even if the app is down our servers will still capture emails offline. The problem here is that MongoDB was not part of the core sending process, it was just for viewing email history in the Web App. While a lot of it was in the background, we still had some work to do.

The new system, launched on Sunday, allows us to put all non-critical aspects of the system in the background. This way, even if our Cloudant cluster were to go offline completely, Postmark would still send out your valuable emails and keep churning. While the sending history is important, getting your emails to the inbox is our first priority.

The migration

The migration on Sunday was not just a data migration from MongoDB to Cloudant. It was a huge refactor of code and processes. While we tend to work in smaller releases, we had to do it this way. After the initial migration everything looked great. Once records stopped going to MongoDB we had some scripts ready to start back filling the old data into Cloudant. Sometime in the afteroon on Sunday, all hell broke loose. Our RabbitMQ queues went through the roof and records stopped getting processed into our Cloudant cluster. At first we thought it was a mix of regular production activity combined with the back fill imports. We managed to get through it, stopped the imports, and things went back to normal. We decided to regroup.

The next day, Monday, it happened again around the same time. We spent a lot of time with Cloudant to analyze the cluster performance, but everything looked normal. The problem is that we had to delay our back fill imports, so while you saw your recent activity, anything before Sunday was missing. We were still a little nervous to run the imports again. It wasn’t until Tuesday, when it happened again at the same time, that we realized the formatting of certain customer emails (tens of thousands of them) was breaking our RabbitMQ workers. Thankfully, once Milan patched it, everything went back to normal.

Where we are now

We finished with the back fill imports yesterday. As of now, all previous bounce, sent and inbound history should be in the activity. The last step is to correct some incorrect “In queue” records that are there from the RabbitMQ issues. We should have them settled today and tomorrow.

Overall, a gigantic success

While it was exhausting and stressful, this was a gigantic success for us. The Cloudant cluster is humming along nicely. We just had a customer slam us with emails for a really large brand giveaway - our charts and benchmarking tools hardly even noticed it. There are many, many positive things to take away from this migration, but let me list a few:

A consistent and much more stable Elastic Search indexer (The CouchDB river is much better than our custom written tool for Mongo)
Successfully moved most of our processes in the background, decoupling critical sending from email activity and search.
A highly proficient team at Cloudant to help manage our cluster.
Ability to store many more days of activity history.

Overall, the real success is moving forward. With our Cloudant migration out of the way, we can now continue on some much needed features like Open Tracking, Link Tracking and some other really awesome ideas we have in the works.

I really want to thank you for putting up with some of the mess over the last few days. We expect better from ourselves and you deserve better. We’re looking forward to the next steps of Postmark. It’s going to be fun.

New! Invite your team and clients to Postmark

achistyakov — Tue, 13 Nov 2012 10:11:00 -0500

While you trust Postmark to send your emails, you also rely on our stats and digests to keep you updated on how it’s going. Email sending stats are something that can involve multiple people you’re working with. For a long time you’ve been asking for a way to share digests or server stats with your team members and clients. Now it’s easy to do, just invite them to your Postmark account.

You can manage people in your account on our new Users page. We took the liberty of personalizing them with Gravatar, so you can actually see their faces. Each person can view reports, email activity and receive email digests. They don’t have access to the account section, sender signatures, server settings or credentials. Those sections are accessible for only the owner.

Also don’t miss the new Permissions tab on the server page. To make permission management really simple we allow you to configure users access both per person and per server. When you give people access to a server for the first time, they will receive an email notification about the new server available to them.

To bring this all to life we had to switch our authentication system to use usernames. This way one person could belong to more than one account. If you’re an existing Postmark user, your email has become your username. You can change your username in your profile at anytime if need.

A few more notes on usernames and email addresses:

A username should be unique Postmark-wide.
Emails should be unique Account-wide.
You can own only a single account using a single email address.

This is a big update for Postmark and we would really love to hear your feedback. If you have any questions please leave a comment below or contact support.

Weekly Email Digests

efedorenko — Thu, 30 Aug 2012 11:54:00 -0400

At Postmark, our goal is to be an extension of your infrastructre – you set it up once and then forget about it. This can cause problems though – while we’re sending your emails with no issues, you aren’t really keeping an eye on problems (like bounces, spam, etc). Yesterday we launched a new major feature which will help you keep track of your servers’ activity without needing to login – weekly digests.

You can subscribe to digests on our new Email Notifications page under the Account tab. Once subscribed we’ll send you an email (one for each server you’ve subscribed to) every Monday with statistics, recommendations and hints. Here is what you can expect to see:

The total number of sent outbound and processed inbound emails, and how it compares to previous week.
If you use tags, we’ll show a list of the tags used most. This could be an easy way to gather some statistics about your app, like how many people signed up (if you’re sending a welcome email), cancelled accounts (if you’re sending a farewell email) or reset their password.
The number of spam complaints received during the last week. We’ll warn you when your spam rate is high or you’re risking suspension.
The number of bounced emails with a list of the most popular bounce reasons.
Processing failures for inbound emails with a list of recent errors.
Sending volume for each of your signatures. We’ll also provide you with recommendations on how to improve your delivery rate by setting up email authentication in your DNS records.

We’re looking forward to providing you with even more data in the future. In the meantime, go to the Email Notifications page and subscribe to all of your important servers!

Live chat support will be closed this week. We will reopen live chat Monday, July 23rd at 9am EST....

natalienagele — Mon, 16 Jul 2012 10:43:16 -0400

Live chat support will be closed this week. We will reopen live chat Monday, July 23rd at 9am EST. Email support is still very active. Thanks!

Winner of the Postmark Inbound Contest

cnagele — Tue, 26 Jun 2012 13:04:06 -0400

A few weeks ago we did a little contest for Postmark inbound. It was originally just a little script we wanted to write for ourselves, but we figured we would offer some credits and give people the opportunity to build something around our tools.

The goal: build a simple app that sends Google Alerts to Campfire using Postmark’s inbound tools.

The winner, Matt Gillooly, submitted a rails app to do just that. It will easily run on something like Heroku, where you can insert the keywords for the Google Alerts along with the campfire room to post them. We’ve been testing it internally and it works great!

Thanks Matt! And, if anyone else wants to try it, grab the code on Github and contribute back.

Using MessageID to Retrieve Bounces

jpcancook — Tue, 12 Jun 2012 15:57:00 -0400

Today we’re releasing a change to the Postmark Bounce API that we think will make retrieving bounces from the system a whole lot easier!

Previously, in order to retrieve a specific bounce, a user would have to query using a filter similar to the following:

https://api.postmarkapp.com/bounces?count=10&offset=0&emailfilter=wildbit.com

A filter query like the above would return a list of bounces that were originally sent to the wildbit.com domain. Users would then have to sort the list for the specific bounce they were looking for. Once a user found the bounce, they’d have to re-query the API using the bounceID to retrieve the full content of the bounce, if needed:

https://api.postmarkapp.com/bounces/389296145

As of today, users can use the messageID field on the Bounces API to retrieve a specific bounce without first filtering by other criteria. The messageID is the string returned for each successfully sent email through the API. A typical response from the Email API looks like this:

{
“To”:”someone@example.com”,
“SubmittedAt”:”2012-06-12T15:29:45.9688186-04:00”,
“MessageID”:”bdd0ad3c-5128-4ad2-8b1d-223b1d3521dd”,
“ErrorCode”:0,
“Message”:”OK”
}

Because users can save the messageIDs returned when the emails are first sent, they can use that information to query the Bounce API directly:

https://api.postmarkapp.com/bounces?messageID=bdd0ad3c-5128-4ad2-8b1d-223b1d3521dd

Note, when querying by messageID, the count and offset fields are not required.

To retrieve the full content of the bounce, users will still have to use the single bounce retrieval API endpoint with the bounceID.

We think this change will go a long way to help speed up bounce querying for a lot of end users!

Contest! Bring Google Alerts into Campfire using Inbound Processing

natalienagele — Tue, 05 Jun 2012 12:01:00 -0400

Want to get ~~25,000~~ 75,000 free Postmark credits? Well here’s the chance! Our team wanted to build ourselves a little tool to get Google Alerts into our Campfire chat room whenever someone talked about any of our products. While we were discussing it we thought instead it would be an awesome litte contest to get to meet some of our developer community and also see what someone else could do with Postmark’s Inbound processing.

How it should work

Our team spends a lot of time in 37signals’ Campfire chat client. A few of us have Google Alerts set to keep track of basic buzz around the internet about any of our products. What we’d like is for those alerts to be posted directly into our Campfire. This way, the whole team knows if something is going on or if there’s some great press going around.

Google Alerts come as an email to anyone subscribed. Using Postmark’s Inbound processing, you can have that emailed alert get processed and posted to Campfire using the Campfire API. How you do it is up to you.

Rules

Some ground rules to qualify

You can use any language you want. Whatever makes you happy.
The code needs to be open sourced and hosted somewhere like Github.
Should work with Google Alerts and 37signals’ Campfire chat client.
It should work!
The code needs to have tests built in.
Campfire rooms and keywords must be configurable.

Submission

We’ll be taking submissions until June 11th 12pm EST. Just email us with a link to where it’s hosted so we can review it and take it for a spin. Make sure to include your account email address for us to add credits. If it works and follows the rules, we’ll add 25,000 credits to your account.

We’re excited to see what everyone comes up with!

New Feature: Postmark Inbound Auto-Retries

mgornik — Tue, 01 May 2012 14:43:30 -0400

New inbound feature - retries

Since launching Postmark Inbound, one of the features that our customers requested the most is the ability to retry inbound posts. Lot of things can happen that can disrupt an inbound JSON post coming from Postmark to the inbound hook server. The web application receiving emails or the web server hosting it can go down due to various reasons. Or, networking issues can render your inbound hook unreachable from Postmark.

Whatever the reason for the failed POST, you shouldn’t ever miss an email. That’s why we are adding automatic retries to Inbound, to ensure your web app receives the processed inbound emails.

How does it work?

When an email is sent to your inbound mailbox, Postmark will retrieve it and process it. Immediately, it will try to post the generated JSON to the inbound hook URL you defined. If the post was successful, it will be listed as Processed. The post is considered successful when the inbound hook returns one of the HTTP 200 codes. This is the same behavior we had until now and the presentation in the UI is the same.

However, if the hook returns a non-200 code, we will now schedule the JSON post for a retry. The entry in the activity screen will show how many retries were attempted, and will show the timestamp of the next pending retry.

You can even filter your emails by using the drop-down filter, looking for “Retrying” emails.

A total of 10 retries will be made, with growing intervals from 1 minute to 6 hours. This is done so that if your web app has longer downtime, it still has a chance to receive all of the posts when it is back online. Only if all of the retries have failed, your Inbound page will show the message as Inbound Error.

How much does retry cost?

Retries don’t cost you anything - a single credit is taken for processing inbound email and includes all of the potential retries.

What is the next thing we are working on?

We are improving our infrastructure to produce much faster processing times for inbound emails. Another idea we’re working on is manual retries initiated from UI. If you have any suggestions on what you would like to see with Inbound, please let us know! You can comment here, send us your thoughts over Twitter (@postmarkapp) or email support.

Inbound Improvements: Better Contacts JSON & New Date Fields

alexknows — Fri, 06 Apr 2012 12:35:00 -0400

We have introduced some changes to the posted JSON object.

In a nutshell, here are the improvements:

All contacts (From, To, Cc addresses) will contain both email and full name (display name). Before this change, we only extracted email address, but having the the full name will be a useful addition. Email and full name are separated into two JSON fields, so no additional parsing on your side is required.
Emails with multiple recipients will be parsed on our side and passed with the JSON object as JSON arrays. Now you can easily traverse through all recipients, without any parsing.

The Date property is added to JSON, holding the timestamp of the original message (presenting local time of the message sender). This can be useful to fetch local time zone of the people sending messages to your web app, among other things.

The specifics of these changes are included in the developer docs for handling contacts and send date.

Executing a non-breaking release

To be respectful of your apps, we are going to keep all existing fields in posted JSON and only add new ones.

We added new fields with similar information to some of the existing fields, but they offer more information and have additional formatting.

For example:

"From": "myUser@theirDomain.com",
"FromFull": {
  "Email": "myUser@theirDomain.com",
  "Name": "John Doe"
},

This will cause some duplicate data to appear in posted JSON but will allow you some time to change over to using the new format.

We’re looking for specific use cases where the legacy format is useful. If your app includes one of those cases, please reply to this email and let us know.

Finally, many thanks to the developers who have quickly updated their code examples for Postmark Inbound in response to this update. If you’re interested in providing examples in other languages/frameworks, we can offer free credits in return for your contributions!

We’re really happy to be hearing from customers about how they’ve been using Inbound, so if you’ve got stories you haven’t told yet, please share them with us!

DevOps Deals with Server Density

alexknows — Thu, 05 Apr 2012 13:45:00 -0400

Our friends at Server Density are running a series of deals in the coming months highlighting the useful sysadmin/development tools that they use and love. We’re thrilled to be their first featured deal.

In return, they’re offering a year of free server monitoring for 1 server to all Postmark customers!

What it does: Server Density provides hosted monitoring, analytics, and alerts for your servers and websites. You can easily start monitoring the status and response times of your websites which can be correlated with server metrics like CPU, memory, disk and other applications like Apache, MySQL and MongoDB.

How we use it: We use Server Density to monitor server health and statistics (on OS and DB levels). And we also publish custom metrics to Server Density which are used to monitor our own parameters in Postmark.

The deal: Server Density is offering Postmark customers 1 year of free server monitoring for 1 server to new accounts.

How to get the deal: E-mail hello@serverdensity.com with your Server Density account URL and mention this blog post.

Lessons learned from our recent outages

cnagele — Fri, 30 Mar 2012 11:34:00 -0400

I owe an explanation for the recent issues on Postmark. This week we had some pretty terrible performance on our MongoDB cluster, resulting in delayed emails, disabled activity and even disabled bounce API. No emails were lost during the outages, but I do want to explain what happened, what we did, and what we are doing about it.

Cause and effect

In short, one of our MongoDB clusters failed, forcing a failover to a secondary machine as a primary. Unfortunately this new primary did not have enough resources (Disk IO) to keep up with the write load that Postmark throws at it. Since the other replica failed, we basically had to wait until it rebuilt to bring it back, which took over a day with the amount of data we had on that cluster. Once we failed over again, everything instantly came back to normal.

The issue that occurred is not something new. We’ve had challenges with MongoDB in the past. I would say that it is not entirely about how MongoDB works, but how it fits our needs and how we use it. Postmark captures a lot of data each day, which we store both in MongoDB for archival and Elastic Search for searching.

We go through so much data that in order to maintain the performance that our customers expect, we need to purge data constantly. With a solution like MongoDB, this causes two problems.

First, we have to constantly delete data, which adds to the disk utilization on each primary in the replicaset. Second, when data is purged, it does not release the space on disk, meaning that we have to run constant compactions or resync to get it back.

As a result of these problems, our Bounce API was down for almost two days. We chose to disable that part of the API because it uses MongoDB queries to find emails, which puts additional load on the servers. We decided that it was better to have email sending services in good shape if it meant the Bounce API was offline.

While embarrassing, I still feel this was the right decision. The other side affect of the performance issues are inaccurate “In queue” messages in activity. While all emails were sent successfully (although possibly delayed in some cases), the activity statuses were still being pulled from MongoDB data which reflected some inconsistency while it was suffering. We are slowly updating the activity history to reflect the properly sent messages and you can expect them to be back to normal within the next day or so. All new messages are being shown accurately.

A silver lining

One really good thing that came out of the performance issues is an updated Bounce API. The bounce API has always gone directly to MongoDB when searching for email records. Since MongoDB is not really meant for searching, it was very slow, and if you use it you probably had timeouts in the past. One of our developers, JP Toto, was able to rebuild it to make calls directly to Elastic Search, which results in near instant responses. This change will also allow us to enhance how you search for bounce (and sent) records in the future.

Rethinking our architecture

We’ve been on MongoDB for a long time. At this point we are rethinking our architecture and have a plan for some very thorough benchmarking of other nosql (and sql) databases to see how they might fit our needs. The most important part of the results of this benchmarking is making sure that our email archive cannot affect the real-time sending that our customers rely on. This either means an eventually consistent solution or heavy background processing, or a combination of both. We still have not ruled out MongoDB, but we have some ideas on how we can use it differently that might fit us better.

Since most of our customers are also developers, we’ll make sure to post our benchmarking results once we are done.

More than anything, I want to sincerely apologize for the problems. For the most part, we know we are doing our job when you forget we exist because everything “just works”. We heard from many of you because that wasn‘t the case in the past few days.

We‘ll make sure that the next time we hear from you, it’s on a positive note. Thanks for your patience and support.

A really rough Saturday

natalienagele — Sat, 24 Mar 2012 15:38:00 -0400

Today we were going to have a really important, but low impact maintenance. This morning at 9am the team brought the application offline to migrate to a new, more powerful database. Email sending was not going to be affected during most of the process, we expected about 2-4 minutes of paused sending. Things did not happen as we expected and we had some long outages and delays in sending. I want to explain what happened in detail.

What happened

The process to move to the new database meant we had to transfer data. We were moving the database to our datacenter in Chicago (from Virginia). When we came back online we noticed significantly worse performance. While we investigated what the cause could be, emails were sending but at a much slower rate.

To combat the slowness, we added many more workers and about 20 minutes later sending was back to full speed. At this time we realized something strange was going on. Our test accounts that we created after the migration were reflecting inconsistent data. As we continued to investigate, we realized that the transfer of the old to new DB was incomplete. This was totally unexpected and took a while to realize. During this time, all accounts were not affected and sending chugged along very quickly.

We decided that the best way to fix the issue was to reimport the old DB dump. We had to put the app in emergency offline mode and run the import. After about 15 minutes it completed, but was still missing some records. We decided the only solution at that point was to revert back. About 10 minutes later the app was back online and sending was instant.

The damage

By reverting back we lost some data from today between 10am EST and 2:30pm EST. This includes a some new account sign ups as well as any credit deductions and sent email count. However, the sent records can still be seen in the activity page.

The good news is we didn’t lose any emails, and all emails are in Activity for searching.

What we’re doing now

Russ and the team are investigating what happened. In all our testing we weren’t able to predict that the sync would be incomplete. Until we know for sure, we will hold off on the migration. Our hope is to try again next weekend. Before we do, we will send an email notice about the maintance so you’re fully aware of any upcoming downtime.

We are very aware of the affect this kind of day has on our customers and their businesses. We use Postmark ourselves and rely on it’s uptime. We’ve gotten to a really good place testing and most days you never hear of any of our deployments or updates. On behalf of myself, Chris and our team, we’re very sorry. We will continue to work towards making sure these kinds of situations don’t happen again.