Mike O'Toole's Blog

Reading SOPA

2012-01-16T00:00:00-05:00

Yesterday, while appearing on Up with Chris Hayes, Richard Cotton, NBCUniversal Executive Vice President and General Counsel, repeatedly claimed the SOPA was exclusively dedicated to fighting websites outside the United States that are wholesale dedicated to the theft of intellectual property. At one point he said “This legislation would not effect a single site in the U.S.” I was curious to see how much truth there was to Cotton’s statement, so I gave the text of the bill a read.

The bill actually has several sections, all mostly related to preventing theft of U.S. Property online. The two sections of the bill that are primarily related to what everyone is talking about seem to be 102 “ACTION BY ATTORNEY GENERAL TO PROTECT U.S. CUSTOMERS AND PREVENT U.S. SUPPORT OF FOREIGN INFRINGING SITES,” and 103 “MARKET-BASED SYSTEM TO PROTECT U.S. CUSTOMERS AND PREVENT U.S. FUNDING OF SITES DEDICATED TO THEFT OF U.S. PROPERTY.”

While I’m certainly not a legal scholar, it would seem that Mr. Cotton did not read the bill terribly closely. His claim that it only applies to foreign sites that are wholesale involved in copyright infrigement conflates pieces of these two sections of the bill.

Section 102 deals with foreign sites involved in any sort of intelectual property infrindgement, not just sites that are dedicted wholesale to IP infindgement as Cotton suggests. The section outlines a set of powers that allow Attorney General to serve notice to search engines to remove links to these foreign sites, and payment network providers and advertising services to stop servicing the sites.

Section 103 applys to any “Internet site is dedicated to theft of U.S. property.” This appears to be the section of the bill where Cotton got the part about ‘site that are wholesale dedicated to piracy.’ However, this section of the bill appears to apply to any site that is used by users in the U.S., whether it’s foreign or domestic.

103 is also a particularly scary part of the bill. It grants powers to intelectual property holders to submit notices to payment service providers and internet advertising services, requiring that they cut off their services to sites ‘dedicated to theft of U.S. property.’

To clarify– a holder of intelectual property can claim that essentially any website is dedicated to the theft of U.S. property, and that they are being harmed by the site. The IP holder formalizes this claim into an official notice as outlined in 103(b)(4), and delivers it to any payment service providers or internet advertising service that are servicing the site. Those companies are then required to stop all services to site within 5 days (for exact details on what this entails checkout 103(b)(1) and 103(b)(2)).

No court judgement is involved in IP holders serving these notices. However, if one of the providers elects to ignore the notice, then the IP holder can get a court order to force them into halting their services to the site.

Claiming that SOPA will not effect a single U.S. based website is simply false.

Hosting @font-face fonts on Amazon S3

2011-10-19T00:00:00-04:00

At Nestio we host all of our static assets on an Amazon S3 bucket, http://static.nestio.com. Generally, this works great, we keep all requests to nestio.com limited strictly to things that require our app servers to respond and offload the work serving static assets to S3. Unfortunately, this comes with one limitation: Firefox won’t show your users fonts you specify with @font-face, unless they’re served off the same domain of the page they’re being displayed on (not even a different subdomain works) or the fonts are served up with a proper Access-Control-Allow-Origin header.

Sadly, right now S3 doesn’t allow you to specify the Access-Control-Allow-Origin header that your objects get served with. For a while, we sort of let this slide and Firefox users just saw the next standard font down on the font stack, but with our recent redesign, we decided to finally implement a fix.

We didn’t really want to make any major changes to our asset deploy pipeline just for fonts, so hosting the fonts somewhere besides S3 wasn’t really a good option.

Instead, we setup a simple nginx VirtualHost for fonts.nestio.com, with the following config:

server {
    listen 80;

    server_name fonts.nestio.com;

    location / {
        proxy_pass http://static.nestio.com/fonts/;
        add_header Access-Control-Allow-Origin http://nestio.com;
    }
}

Requests to fonts.nestio.com are proxied back to our Amazon S3 bucket, so the files themselves are still hosted and can be deployed through S3, but before sending a response to the user, nginx appends the necessary Access-Control-Allow-Origin header.

Effectively sending email from your web application

2011-05-28T00:00:00-04:00

Successfully sending email from a web application is a complex problem. Unfortunately, this complexity is often hidden until you have to spend hours or days trying to figure out why your email isn’t getting to your users.

Sending an email in PHP is seemingly as simple as this:

mail($to, $subject, $body);

Frameworks like Rails and Django have similar methods for sending email.

With the default configuration and the likely case that your web application server has a Mail Transfer Agent (MTA) like Sendmail or Postfix, running on it, PHP will happily execute this call for you. And it’ll return true, suggesting that the email will actually be delivered. Sometimes this will work beautifully. However, unless you’ve done a lot of work upfront to ensure that your web server is allowed to send email on behalf of your domain, you’ll start getting complaints from users that your email is ending up in their spam folder or not arriving at all.

Rather than spending the time focusing on making your web server a valid point of origin for your email, it’s much easier to punt the responsibility to a service that’s already been setup to send email on behalf of your domain, like an existing email provider. Or, better yet, you can use a 3rd party service that specifically focuses on sending large volumes of transactional email from web apps. Some options are Sendgrid, Postmark, or Amazon SES. Just follow the directions provided by any of these services for setting up DNS records to ensure that they are authorized to send email from your domain.

Note to GMail/Google Apps users: It’s not a great idea to have your web app send email using Google’s SMTP servers, as Google limits all accounts to sending a maximum of 500 emails per day. If you start sending significant numbers of messages, you’ll easily max that out. It’s likely that many other mail providers have similar limits, so check with yours before using their SMTP servers to send high volumes of email.

Remote Network Connections

However, sending mail via a remote server introduces a whole new problem: a remote network connection has to made whenever your app sends an email. So every time a user submits a request that triggers an email being sent, a connection has to be made to the remote SMTP server and the message data needs to be passed through it. All of this probably happens while the end-user is waiting for a response. If network latency is high then it can really slow down that response being returned to the user, or if latency is really bad the user’s connection to your site might timeout entirely. Even worse, the remote SMTP server might be unreachable because it’s down or there’s a network partition somewhere along the line. If that happens, depending on how your app is built, it may fail silently or an error will be be returned to the user, in both cases the email is never sent.

To prevent such problems, rather than attempting to connect to your SMTP server while the user is waiting for a response, it’s preferable to queue the message locally, return a response to the user, and then in the background connect to your SMTP server and send the message. If the SMTP server is unreachable, requeue the message and try to send it later.

You could build all that logic into your application. Maybe store messages to be sent in a database table, and have a background job that attempts to send them, but that really seems like a lot of work and overhead for something as simple as sending email. Low and behold, all this queuing and retry logic for sending messages to remote SMTP servers is already built into MTA software like Postfix. As we saw above, you can run Postfix locally on your web server, which virtually eliminates any latency or possibility of a network partition when your application is connecting to send an email.

But you’ve just spent all this money on a Sendgrid account so that your email you wouldn’t end up in a user’s spam box, why would you go back to using a local MTA!? Well it turns out that MTAs can be configured to relay all of the mail they receive to an upstream SMTP server (like Sendgrid, Postmark, or your email provider of choice), that server in turn will go ahead and relay the message to third party recipients.

So rather than connecting to a remote server to send email, your web application can connect to it’s local instance of Postfix. Postfix accepts the message, quickly adds to it to its message queue, and responds with a success code. Your web app can then go ahead return a response to the user, who can go about their business. In the meantime, Postfix will asynchronously attempt to connect to your upstream SMTP server and pass the message along. In the event that it can’t reach the upstream server, it’ll put the message back on the queue and try again later. Once the upstream SMTP server gets the message, it sends it to the recipient’s SMTP server; so to that server it looks like the email is coming from a valid point of origin: your email provider, rather than your web server.

Implementing the Solution

If your application server is running a *NIX system, I highly recommend setting up Postfix as a local MTA to relay messages to an upstream SMTP server. It’s fast and relatively easy to configure. If it’s not already installed, you should be available to install it with your system’s package manager. For example, using Debian or Ubuntu:

apt-get install postfix

If the install process asks you about configuring Postfix, simply select No Configuration, we’ll configure it ourselves in just a minute.

Once it’s installed, test to ensure that postfix is capable in sending mail in it’s default state, by running the following command:

echo "Oh hi there" | sendmail your_email_address@host.com

This will send an email using Postfix to your email address. If you don’t receive it, first check your spam folder, then check the Postfix log, usually stored at /var/log/mail.log, for errors.

The first thing to do whenever you setup an MTA, is to ensure you’re not creating an open relay. Open relays allow anyone on the Internet to connect to your MTA and send mail using it. This is really bad. Spammers will find this and use it to send lots of email through your server. Leading to many terrible things, including having your IP blacklisted as a server that sends spam. By default Postfix accepts all connections on port 25, so ideally you should deny all remote connections to port 25 via your firewall. In addition to that, you should to configure Postfix itself to only accept connections from localhost, by adding this line to the bottom of Postfix’s main.cf file (located at /etc/postfix/main.cf on Debian/Ubuntu):

inet_interfaces = loopback-only

Once you’ve added that, restart the Postfix daemon. On Debian or Ubuntu:

/etc/init.d/postfix restart

With that, only connections coming from the within the server itself will be able to send email through Postfix.

Note to Amazon SES users: While it is possible to relay mail from Postfix to Amazon SES, the configuration is slightly different than what is outlined beyond this point, for more information check out Amazon’s guide.

Your upstream SMTP server probably requires authentication, so you’ll also need to install some SASL libraries that Postfix relies on for authentication. On Dedian/Ubuntu:

apt-get install libsasl2-modules

Or a yum-based system:

yum install cyrus-sasl-plain

Postfix then needs to be configured to relay all the mail it receives to your upstream SMTP server. To do that, simply add the following lines to the bottom of main.cf file:

smtp_sasl_auth_enable = yes 
smtp_sasl_password_maps = static:your-SMTP-Username:your-SMTP-Password
smtp_sasl_security_options = noanonymous 
header_size_limit = 4096000
relayhost = [your.smtp.host]:25

Obviously replacing the auth credentials and host with those provided by your email provider.

Or, if your upstream SMTP server supports it, enable TLS encryption, so that your messages and authentication credentials aren’t sent in plaintext, use this configuration:

smtp_sasl_auth_enable = yes 
smtp_sasl_password_maps = static:your-SMTP-Username:your-SMTP-Password 
smtp_sasl_security_options = noanonymous 
smtp_tls_security_level = may
header_size_limit = 4096000
relayhost = [your.smtp.host]:587

After updating the configuration file, restart Postfix.

/etc/init.d/postfix restart

And configure your web application to send email through the SMTP server localhost, on port 25 with no authentication. For example in Rails:

ActionMailer::Base.smtp_settings = {
  :address => "localhost",
  :port => '25',
  :domain => "yourdomain.com"
}

Send some test emails from your web app, or from the command line:

echo "Oh hi there" | sendmail your_email_address@host.com

If you look at the headers of these messages, they should indicate that they’re being sent through your upstream SMTP server rather than directly from your web app server.

This gives you the advantages of sending mail via local MTA: minimal network latency, as well as having messages queued and retried when the upstream SMTP server is unavailable, combined with the deliverability advantages of using an SMTP server that’s properly setup to deliver mail on behalf of your domain. To the recipient mail servers, messages will look like it’s coming from your proper SMTP server rather than your web app server.