Mailinator(tm) Blog

Mailinator is for Humans

2013-05-01T19:49:00.003-07:00

Mailinator started a decade ago with the simple idea of providing disposable email addresses to anyone who needs one. It's done that ever since.

What has happened as time progressed however is that programming got easier. And people creating automated systems to access Mailinator became more common. Way more common. The problem became that programmatic access plus anonymity seemed to encourage trouble. Programmatic interfaces like POP3, designed to be an extra feature for humans, became an abuse channel for scripts.

Once dealing with abuse started becoming the primary maintenance activity of the site, it became clear that it was time to make some changes. POP3 got shut-off early largely because of a wide botnet attack (it was actually pretty impressive - and rate-limiters did nothing to slow it down).

With all that in mind, you can expect some changes in the near future if you access Mailinator programmatically. All changes are meant to diminish abuse on the site while improving user (i.e. human) experience. For the everyday user of Mailinator that accesses email through the web - nothing changes except the experience should be faster, better, slicker.

For developers who use Mailinator for testing - you're not forgotten and a real programmatic interface is in the works. One that will require you to "sign up" before getting such programmatic access however.

The changes that are happening are:

1) For the first time in many years, Weblog files are being turned on. Just like 99.9% of all other websites, we'll now keep weblogs. Not having them has been a hindrance to tracking abuse. Having logs will help us track usage patterns and hopefully slow down abuse altogether.

2) All current programmatic-esque interfaces will be phased out. POP is already gone, others to follow.

3) A sign-up version of Mailinator will appear with all the features you'd expect including pay tiers for users with bigger needs (we get some interesting requests).

4) A site redesign is in the works (I love it, you'll hate it, everyone hates change, but I promise, it's awesome and you'll love it soon too) that will change how the system is accessed. It's going to be cool - for humans I mean.

Mailinator's privacy policy is not changing. It has been and has always been:

Mailinator provides NO privacy

It can't - all inboxes are shared. All email is "public domain". None of that changes.

There are lots of improvements in store and this turned out to be a good time to clean up use ends. Thanks for using Mailinator !

Code Review for a String Lock Map

2013-04-16T14:14:00.000-07:00

I recently needed a mechanism to lock on given Strings in a particular piece of code. As maybe a weak example, say I was doing disk file updates to files each with unique filenames (i.e. read, update, write-back). I could have synchronized the methods that modified the files, but I preferred to not lock at that level - in other words, only one file update could occur at a time then.

I'd prefer to simply lock on that file - or more specifically, that filename. Its a dubious proposition to lock on String objects in Java as different objects can represent the same set of characters. You can intern() all your strings (which makes sure the objects you have are the "one" system-wide representation of that String) but that is rather global. I wanted to do things locally - where I can control the scoping of the lock.

(The filename example is just an example, I've needed this structure in many different scenarios - so don't look to just solve that problem).

I might have missed it, but I didn't find any facilities in java.util.concurrent to help me (and to be fair, I haven't kept up with the happenings there. Long story short, I wrote my own.

Of course the best advice when writing concurrent locking datastructures is - DON'T.

But I did. And I probably plan on using it eventually, so I put it here for review. I'll update it as comments come in (so if you see a comment below that makes little sense it's probably because I heeded it and changed the code).

The use case I want:

public static final StringLock stringLock = new StringLock();

...

try {

     stringLock.lock(filename);

     // load disk file

     // change its contents a bit

     // rewrite it

} finally {

     stringLock.unlock(filename);

}

So.. find below the code I've come with so far. Its a bit heavy under contention but the normal case (no contention) it syncs 3 times (writes on ConcurrentHashMap being a sync). I could also use Cliff Click's NonBlockingHashMap to remove two of the syncs but and I'd be happy to be convinced that'd be superior for some reason.

public class StringLock {

 private final ConcurrentHashMap lockedStrings = new ConcurrentHashMap();

 public void lock(String s) { 

   Object lockObject = null; 

   while (true) { 

     lockObject = lockedStrings.get(s); 

     if (lockObject != null) { 

       synchronized (lockObject) { 

          Object lockObject2 = lockedStrings.get(s); 

          if (lockObject2 == lockObject) { 

            try {
                            lockObject2.wait();
                        } catch (Exception e) {
                        } 

          } 

        } 

     } else { 

        lockObject = new Object(); 

        Object lockObject1 = lockedStrings.putIfAbsent(s, lockObject); 

        if (lockObject1 == null) { 

          break; 

        } 

     } 

   } 

 } 

 public void unlock(String s) { 

    Object lockObject = lockedStrings.get(s); 

    synchronized (lockObject) { 

      lockedStrings.remove(s); 

      lockObject.notifyAll(); 

    } 

  }

}

I Further Predict the Death of Your Web Framework

2012-08-21T09:19:00.002-07:00

In my last post, I philosophized how new technology is going to change the bottleneck of web (and other) systems (which despite everything else, have remained surprisingly stable for awhile).

This spelled the demise of many systems that relied on a given system bottleneck, specifically, slow runtime systems.

But there is another technological shift conspiring against many web frameworks that isn't focused on performance, but instead focused on "ease of use" - which in many cases may hit far closer to home.

That shift is the reorganization of MVC.

MVC stands for "model-view-controller" which loosely means you have a datastore/database (the model) which is retrieved and manipulated (by the controller) such that it can finally be shown to a user (the view).

That is, pretty much always the flow. Well - kinda. The first thing you might notice is that "MVC" is an out-of-order acronym per the dataflow. In that case it would be "MCV". And happily, given that dataflow is paramount to my story - I'll use that in the rest of this article (that might irk you if you're CDO (which is like "OCD", except in alphabetical order, like it SHOULD BE)).

A predecessor to MCV was a simpler idea of simply "client/server" where client was the view, server was the controller and model (or, some or all of the controller could be in the client too). However, the client in that case actually implied it was a real client - that is a program that received the data and showed it.

In the web, the browser is the client, but interestingly in things like Rails, Jails, Nails, Grails, Struts, Play!, PHP, ASP.Net, and many others the "view" is on the server which then renders HTML and sends that to the browser. As far as the programmer is concerned, the whole MCV is on the server. The browser is often just a dumb terminal.

In the last year or two however, the popularity of a new type of framework is changing all that.

That change is coming from libraries such as backbone.js and ember.js (and many, many others).

Those libraries allow you to render views (not just show, actually render) in the browser itself. In addition, they let you leverage a lot of javascript magic in the browser. This is pretty awesome for several reasons.

The computing power of rendering is moved to the client's machine. Rendering isn't probably your biggest computing expense, but take off that computing cost from your server (times every web request you get) and its measurable.

And as you can imagine, if the "V" of MCV actually migrates to the client, all that's left on the server is "MC" (to be fair, sometimes even part of the "C" goes to the client).

What thousands and thousands of Rails developers discovered upon moving to backbone is that they no longer needed their fancy template views. Their backend became a system that pushed JSON over HTTP.

Very clean and very simple. At my new company Refresh (we're hiring!), our backend pushes the exact same JSON to our webpage as it does to our IOS app. And that same system will someday seamlessly become our API too.

For me, using Rails for webapps over Java (where I spent plenty of time years ago) was a simple decision. ActiveRecord was beautiful and elegant (especially compared to things like Java's hibernate). Also, the view layer was simple, well-laid-out, and standardized. If anything, Java had too many choices.

But these days, I tend to use NoSQL on the backend. And ember on the front-end. All I need in the middle is something to manipulate and push JSON. Why was I paying the Rails tax? (again insert any language that is a multiple slower than Java in that sentence).

I'm not particularly picking on Rails - it is just a full MCV solution that I no longer need. There are plenty of those.

And if you're thinking this is a win for Node.js - you're probably right. With much more javascript coding entering your web framework as a whole, using Node on the backend is probably the winner of all this on the usability front. Javascript on the server isn't the fastest, but it's pretty darn good at manipulating JSON (and thank you to whoever it was that shot XML dead).

So my not-so-amazing prediction is that in a few short years time full web frameworks from any languages disappear. Node picks up some of that slack but so do less feature-ful frameworks (and maybe performant ones). Even non-frameworks altogether get more use.

There's surely no mourning required here. Web frameworks change every few years no matter how you slice it. But between this post and my last, I see two converging fronts out to kill some our most popular ones right now.

Personally, I'm hoping to never server-side render HTML again. I'll let your browser do my rendering while I sit back, chill, and push some JSON.

And yes, Mailinator is in rewrite now to use ember, much to the chagrin of web scraping programs everywhere! (but much to the happy of JSON receivers)

Your Bottleneck is Dead. Long Live Your Bottleneck.

2012-08-13T07:58:00.002-07:00

There's an old joke that, if you think about it, you can apply directly to system bottlenecks.

Two hikers are walking through the woods when they come face-to-face with a pack of wolves. One of the hikers immediately drops to the ground and hastily changes from his hiking boots to the running shoes he had in his backpack.

The 2nd hiker says, "What are you doing ! You can't outrun those wolves!"

The 1st replies, "I don't have to outrun those wolves. I just have to outrun you."

Web developers tend to know their system's biggest bottleneck, but how often do you know the 2nd biggest one? Right, in one sense it doesn't matter - because the 2nd bottleneck doesn't get to become a bottleneck unless it's the biggest one.

There is an economic model hidden within every complex system. This includes something as mundane as web system performance. Knuth famously said (or re-said) that "premature optimization is the root of all evil" which could be restated as - if you optimize before you know what needs it, you're optimizing (and probably breaking) the wrong thing.

Hence we don't (or aren't supposed to) optimize what doesn't need it. Seems obvious - but it has interesting ramifications.

When something doesn't need optimizing, we can afford to be (and often tend to be) lazy with it when it comes to performance. Concretely, if your code's database access is going to take a 15 milliseconds, worrying that processing that data will take 20 microseconds because of your sloppy n^2 algorithm probably isn't worth much thought.

If that statement raises your ire, feel free to sit in your chair and pout - because there are thousands of websites that were happily coded using notepad with interpreted and dynamic scripting languages that flagrantly use gotos and lists as if they were hashmaps. I've seen it. It's enough to turn your stomach. It's not pretty.

For the average, everyday web hardware ecosystem - we have CPU power to spare. And in the bigger business sense, if I can save time developing a website cutting performance-concerned corners with no ramifications, all the better.

Web development largely started in scripting languages (i.e. perl-cgi, php). Again for the same reason - CPU to spare as compared to other bottlenecks.

In fact, I'll go so far as to say that the popularity of scripting language web frameworks required the condition that disks be some order of magnitude slower than CPUs. That's right - I'm looking at you Rails, Grails, Nails, and Jails.. (ok, not Jails, it's a Java web framework but it rhymed).

Java web frameworks added a lot of structure, verbosity, and performance that simply wasn't needed (and eventually, amazing bloat). If your bottleneck was the database/disk - your web processing simply had to not add significantly to that - and regardless of the language, that wasn't hard.

A simple definition of latency is the time it takes to get data back after requesting it. Similarly to an program anyway, bandwidth could be viewed as how long it takes us to get all the data requested (once you start getting any).

Think of how that relates to code performance. If your latency is 3ms (a reasonable server harddisk seek time) - it doesn't matter if your code is hand-loved machine language or interpreted COBOL - it does nothing for that 3ms. In CPU time, 3ms is an eternity.

As a general tendency however, the more data you receive, the more processing that likely goes around it. Consider a few megabyte JSON message - at a minimum it will likely be parsed. Possibly shoved into a map or an object.

Said another way - lowering latency and increasing bandwidth will tend to put more pressure on processing data (i.e. requiring more CPU/code performance)

So all this time we're happily and harshly slowed down by slow things like spindly harddisk drives and networks. Then, in walk Solid State Drives. Prices and capacities are both heading in the normal directions for new technology (down and up, respectively).

Latency goes from standard spindle drive 3ms seeks to (varying reports) 100microsecond seeks.

Argue the specifics if you will, but for some number of existing systems, installing an SSD will remove the database as the primary bottleneck. In fact, this is probably the cheapest way to improve your system's performance today.

What happens to the bottleneck in those systems? It will shift somewhere else (i.e. the SSD put on its running shoes). In many cases, it will shift to the CPU (CPU in this case is a polite way of saying "your code").

Everyday across the world, there are meetings at companies complaining about the performance of their website. Today, many of those say "get the DBA in here".

In some of those meetings soon, the shift will be away from blaming the database. Some will push for code optimization (postmature), some for bigger hardware, and some for faster languages.

Keep in mind, this is a subtle, slow moving effect. Having your CPUs pegged all the time might not make you change anything today but may make you reconsider your architecture next time you build something.

Of course the network is a bottleneck too - at least for now. In places like Korea and Kansas City that's not so true. If you haven't heard, if you live in Kansas City you can get Google Fiber to your home. In other words, your internet speed will be 100 times faster than the average internet in the US. (In fact, if your machine has the common SATA2 disk drive interface, sending a file to your neighbor across town in Kansas City will only take about 3 times as long as storing it on your own disk just a few inches away).

Here's another prediction - in 5 years the phrase "downloading a movie" won't exist. (We used to say we were "downloading an image" which was preceded by us saying we were just "downloading").

If bandwidth drastically increases, it will change how we write code. We think nothing of loading a 1M webpage now which 10 years ago was offensive. In the future, we may think the same thing about a 100M webpage.

Given that data expands to fill available bandwidth (modified Parkinson's Law) our programs will tend to process much more data. Processing speed will matter more and more.

And the more often code becomes the bottleneck, the more often solutions to fix that will be considered.

Simply - your favorite bottlenecks might be changing. And for that to happen, your disk doesn't necessarily need to be able outrun your CPU - it just has to be able to outrun your code. (And it wouldn't hurt if it could also outrun, you know, wolves too).

My startup Refresh is looking for awesome IOS and front-end engineers. Join us! Email us at jobs@refresh.io De

The love and hate of Node.js

2012-05-16T07:48:00.001-07:00

I've been in the happy position lately of interviewing engineers for my new start-up. If you've read any of my previous articles or seen my talks (Talk Notes (warning: PDF download)) you know I love this stuff (Jobs page).

I'm always up for a spirited discussion about algorithms or languages with smart people, but I do consider too much technical religion to be a red flag and it seems to be a rather common affliction.

So when I started interviewing recently, I was immediately reminded of the blind loyalty some times given to pieces of technology. As if all other competing languages/frameworks/variable-naming-schemes are "crap" where "crap" is loosely defined as, well, I'm not sure - but something that the person saying "crap" sure doesn't like. It's probably safe to say that any popular technology has (or had) a useful purpose at one point, and also I think it's safe to say that same piece of technology is not always the right solution.

Along with the perpetual Hackers news debates, I right away ran into a "Node Guy". That is, a guy maybe just a tad overzealous about using Node.js for, you know, everything.

I asked him why he chose Node for his last project. The answer was "Because it scales super well".

I will say, the marketing hype around Node is pretty good. I am not saying his answer was wrong. It wasn't. But it's pretty similar to answering the question "Why do you want to buy a Porsche?" - with the answer "Because it's fast".

Likely true, but by no means a distinguishing feature.

It isn't hard to find discussions in the developer community defending the performance of Node. Node at its core, is a C++ server. That part is likely competitively fast. But after that, the focus is on the performance of the JavaScript callouts. Someone told me that "microbenchmarks" aren't fair to Node and don't show the true performance. I think they're right in both cases - because microbenchmarks likely involve only a small amount of JavaScript. Truth is, the more JavaScript your Node application runs, the more its going to lose against server frameworks built in faster languages. In other words, microbenchmarks are probably the best Node is ever going to look.

Google's V8 JavaScript engine literally enabled Node to exist at all.
There are of course another set of people (the "Node haters") that are nothing short of incensed by the idea of Node. To them, it feels rather silly to create a server framework in a language like JavaScript. I can relate to someone who has spent years eek'ing out all possible performance of a C++ server only to watch someone write one in JavaScript and claim speediness.

In the early days of any field of science - science, invention, and engineering must overlap. That is, folks think up science, try it, and piece it together to see if it works - however rickety. Eventually however, enough tools and best-practices exist to allow details to be abstracted.

When that happens, many more people can create cool things with existing and tested pieces (i.e. engineer them together). Simply, you need to worry less about the details of the science to get things done. People with no knowledge of the underlying science can glue widgets together to make something. Often amazing things - at that time, you might consider that that "science" is somewhat beginning its evolution towards being an "art".

Possibly the quintessential computer science course is something like "Algorithms & Data Structures". Do you need that course to develop apps these days? Again, by proof of existence - I think not. If you have a phone interview with me I will ask you the difference between an Growable Array (aka Vector, ArrayList, etc) and a Linked List. Both structures do arguably the same thing but with notably different performance characteristics.

It's quite hard to create any application without using some form of list, but as long you have a "list" you know you can get the job done. I promise you there are thousands of sites and apps using the wrong list at the wrong time and incurring nasty linear time searches or insertions in nasty places. Truth is, if that never shows up as a measurable bottleneck, then one could argue that despite the algorithmic transgression - that code is "good enough".

Happy or sad, "good enough" is getting "good enougher" all the time. CPUs are fast and getting faster covering the tracks of slow code. We've never lived in a better time for algorithmic indifference. Comparatively, disks are slow, which make databases slow, which make the performance of the algorithms and languages you choose in your app less important than ever (not to be confused with "unimportant"). In fact, I'd argue that the entire resurgence of interpreted, dynamic languages can be traced back to the lackadaisical 5ms seek times of spinning disk drives. That's a bold statement and probably a whole other article - but if disks/databases are basically always the bottleneck (rather true in most web apps) - who cares how fast your language runs.

(disclaimer: If you've read anything else I've ever wrote you know I'm merely an observer of this trend, not a subscriber)

The controversy over Node is that it implies that developers from the client are piercing into the server. A domain typically full of people that came up from the OS layer. Those people are asking does it really make sense to write servers in a historically (slow) client language?

Further, and possibly a bit more personally, should people who only know client languages be writing servers at all? Server code is a unique skill just like many other things. Dabbling in writing servers is like me dabbling in doing web design - trust me, it's not pretty. There's only so many lower levels left - would you want a JavaScript operating system?

On the notable other hand - People who only know or love that client language have been given a whole new freedom and ability. They'll argue (with a rather reasonable defense) that Node.js represents one of the easiest ways to create a server or web app. Even if they don't defend the performance, in many practical cases, they don't need to - like it or not at the right time it can be "good enough" (again, proof by existence). It's positively no wonder they defend Node. They are defending their newly found, wondrous abilities that can solve real problems. Wouldn't you?

So as my information-less friend said - Node will scale. But that is, indeed information-less. So can Ruby, Rails, Java, C++, and COBOL - architectures scale - languages and servers don't. Just like most web apps, a Node application will probably be bottle-necked at its database. You can fool yourself that Node itself is "insanely fast" but you'd be fooling yourself (Java/Vert.x vs. Node, Java/Jetty vs. Node, Node vs. lots) and rest assured that despite scaling, some portion of your latency is baked into your language/framework performance. Your users are paying milliseconds for your choice of an interpreted and/or dynamic language.

Should your start-up use Node? That depends on a lot of things. If history is a teacher however, massive success will likely push you to something statically typed. Facebook "fixed" PHP by compiling PHP to C++ and Twitter painfully (after years of fail whales) migrated from Ruby to Scala/Java. Square has migrated to JRuby to improve performance, I'll be interested to watch if its enough (I'm feeling yet another article on the nefarious demons upon drop-in replacing a global-interpreter-locked Ruby with a true multithreaded one).

The fight over Node is, in truth, one of the least truly technical developer fights I've seen. People on both sides are simply defending their abilities and maybe their livelihoods - the technical points are rather obvious. I'd say Node is definitely a possible solution for some non-trivial set of problems - then again, I can think of plenty of situations I'd also veer away from it. But of course - I'm not very technically religious - and I'm definitely not a "Node Guy".

All this being said - I am seriously hiring across the stack (including mobile) at my new start-up. If you have a desire to argue with me about this sort of stuff on a daily basis - email me at paul@refresh.io and attach a resume.

This article was spawned from my own comment at William Edwards blog

How to get your resume "Silicon Valley Ready" - Part I

2012-05-02T10:43:00.002-07:00

Per my last post, I've been given the opportunity to review a nice pile of resumes. As I am prone to, this got me to obsess a tad over how the resumes were put together and more importantly, what each told me.

What I perceived as issues are, in retrospect, my fault, not the resume owner's. That's because, per the entire point of my last post, the start-up environment is radically different than the corporate IT department. And the latter is where many of these resumes came from (which is exactly what I wanted and asked for).

In many cases, I received a resume from someone that included the regular set of data - experience, education, skills, etc. But the ones that got me excited were the ones where the person included in the email links to the websites or mobile-apps they had built. As I've said - the number one selling point for you as an engineer to get a job in Silicon Valley is that you love this stuff. There's an age-old conundrum of new grads who say "Employers want me to have experience before they'll hire me - but how do I get experience if I can't get a job?"

In our business I'm happy to say - that problem does not exist. Simply because you don't need anyone to give you a job to build something. A website. A mobile app. Heck, a program that finds smaller sets of strings in larger ones.

I realized that's probably the number one thing I'm looking for. You can show me, with no doubts, that as a software engineer - you can build something. Start to finish.

Interestingly, I like to think that I also don't put that much weight into whether a project was a commercial success. If it was, that's nice but and maybe it's because you are not only a great engineer but you have an awesome product sense - who knows (it just might mean you were lucky too). And unless it wasn't a commercial success because it was poorly engineered, that's not really the point. The point is that you built it. Or at least some non-trivial part of it.

With that in mind - this article sprang forth on what I like to see in resumes. I'll point out that this isn't very different from what I looked for when I was on a hiring committee at Google (so there are at least some current Google engineers that are partially there because of these thoughts).

First - I propose a new section to resumes - at least for software engineers. In addition to Experience, Education, Skills, Interest, and References (not suggesting we remove any of those) - I propose we add Cool Stuff I Have Built.

If your resume is going to go over one page (which, personally, I don't mind) - I'm hoping it's because of this section.

Any project you did solo or had a major hand in - whether paid or not paid, million users or just your mom, I'd love to know about. Websites, iphone apps, android apps, desktop apps, open-source projects, github accounts - you name it. Solo or as part of a team (indicate that). But it has to be, in one way or another "finished". Even if your iphone app was rejected by the app store - you can point me to a link to see it. It doesn't have to be a product either - maybe its an open-source project. The bottom line is it is something that you "finished". You executed. Your idea became a living breathing application or piece of code that in some way some how you could show to people.

The section might be broken up into individual projects with bullet points about each. For example:

Project Name: Mailinator

Technologies used: Java, tomcat, (no database)

Team Size:5'11", 175lbs (haha)

Implementation details/challenges:Custom server architecture built as an experimental test-bed for highly-multithreaded server design. Custom SMTP server. No database as emails are stored in RAM with a custom compression scheme.

Notable Metrics: up to 25MM emails per day, ~20k users per day, runs on a single server (on purpose as part of a personal experiment to optimize the system).

relevant links: www.mailinator.com, http://mailinator.blogspot.com/2012/02/how-mailinator-compresses-email-by-90.html

Surely you could add other bullet points too (and suggestions welcome, leave a comment to this post). But you get the idea.

My previous post resonated strongly with some people - that is, they were in "IT departments" feeling like they weren't growing technically. And as you can imagine, a resume telling me you did a payroll system is great - but it's not what (most) start-ups are building. But what if you haven't built anything? And your "Cool Stuff I've Built" section is empty?

Well .. fix that.

No one is stopping you from building something. No one said you're ready for a transition out of your current job today and as with much of life it's up to you to take yourself to the next level. But nearly any person that already writes software with a penchant for learning and some ambition can spend the next few months of nights and weekends learning and building. (And it's absolutely possible that your day job accomplishments belong in that cool stuff section too).

So if by day you're a payroll guy, but by night you're in iphone ninja - you've got my attention. Not only because you have the skills that I'm looking for - but that in your spare time, you're out doing great things. And instead of going out every night drinking with your friends - at least some of those nights, you chose to stay home and learn and build cool stuff. And why would you do something like that? Simple - you love this stuff.

(My start-up is located in Palo Alto and I am right now interviewing for the initial engineering team. We're well-funded, building cool stuff, and plan to change, ya know, the world. No matter where you are - if you're a software engineer, willing to relocate to San Francisco/Silicon Valley (and of course, love to build great things) send me your resume. paul@refresh.io or check out www.refresh.io/jobs)

Why you should join a start-up - and maybe why you shouldn't

2012-04-24T11:04:00.002-07:00

I've recently been interviewing engineers for my new start-up (fyi, this is wholly separate from Mailinator). We're well-funded, have a world-changing idea, and as you can imagine, I plan to build an awesome engineering team. (Regardless of where you are, if you're a passionate developer, I'd love to hear from you. Check out the Job Description here and email me your resume at paul@refresh.io ).

I've been talking to engineers from all over hearing their stories. There's really amazing talent everywhere and honestly, a non-trivial amount of it seems to be idling or even decaying in environments that aren't using its full potential. A bunch of moons ago I used to work for Dow Chemical in the dreaded "IT department". It was pretty clear to me then that I was not growing technically in that job. I left to start my Ph.D. but I always vowed from then on that if I was going to be a software guy, I was going to work for companies who's business was creating software. In other words, at Dow I was an expense, I'd much rather be an asset.

Eventually and with that goal in mind, I ended up at Google. Without reservation I can say it was a fantastic experience. I have said before, "if you're the smartest person at where you work - quit". And trust me, nothing makes you realize how smart people can actually be by working at a place like Google. (To avoid any implications - I did eventually quit Google, but rest heavily assured, it was not because I anywhere even close to being the smartest person - read on!).

I did over 200 interviews while at Google and it was actually a bit fun to interview someone who was coming from someplace where they were the smartest person (at least about tech). I could always tell. It's no surprise that if you're the smartest person somewhere for a long time, you get used to it. You get used to waiting for people to catch up to where you are.

By no fault of their own they walked into the interview with some attitude. An attitude of impatience if nothing else. After someone like that started at Google however, it didn't take long for them to realize the situation they were now in. It was humbling in many respects and I don't mean that negatively, simply they'd not recently (or ever) experienced a place where many of the people they met were at their level or better. Obviously, there are smart people everywhere but almost universally, smart people enjoy the company of others like them, the synergy makes them all better. This is why Silicon Valley is a magnet for them.

As I said, Google (and similarly Facebook, etc) are great places to work. At some point after working there however I thought to myself what a wonderfully steady and safe place to work it was. My responsibilities, expectations, and compensation package were well outlined. I was working with awesome people and learned a ton but I still felt it was far too big for me to have any real impact.

For a time, I worked on the Google Web Server which I could best describe to non-techies as "well, sort of the thing you interact with when you do a search" (this is a bad definition at best). A woman I was dating thought about that answer a moment and condescendingly replied - "what do you mean you work on that - isn't that done?"
In one sense she was right, I worked on that darn thing every day but to her it all worked the same. To her, I was having no impact.

It occurred to me that Google would be a fantastic place to work if what I wanted was a meaningful 9-5 job that after each day of work I could drive my minivan back to my home in the suburbs. But I didn't have a minivan. And I didn't own a home. And I didn't live anywhere near the suburbs. What the heck was I doing there? The smart-person environment was at start-ups too - I could get that there and even have some ownership of what I was building.

It's a relatively normal course of life in our sea of first-world problems that you'll have many chances to take risks early in life and those chances diminish as time goes on. Simply put, Google will always be there. And if Google isn't - the next big, awesome company will be. Every decade or so has a "company" (or two) where the greatest things and the greatest people are happening. At times it was Microsoft, Cisco, Apple, Google, Facebook, etc.

I left Google not because Google was in any way bad, but because I wasn't done swinging for the fence. And I still had the luxury of trying. If I ever got to the point where I wanted to realign my life's risk profile, Google (or Google-next) would be there. And this is a pretty common theme - places like Google and Facebook incubate some set of people into entrepreneurs who then go start their own start-ups. But with big ideas, agility, and impact. And they don't tend to fall far from the tree. You might think Google doesn't like this - but I doubt that's true. This is a constant stream of risk-takers that go try stuff for them that they can buy back if needed.

What gets me today is how vibrant Silicon Valley is right now. And even for Silicon Valley this place is on-fire. It seems cities around the world try to copy it but that's really hard to do. The start-ups are here because the investors are here, and the investors are here because the start-ups are here. Guy Kawasaki wrote a great article several years ago partially about why Silicon Valley is Silicon Valley.

I am fully aware that Silicon Valley has a nasty habit of simply not being able to darn well shut-up about Silicon Valley. Other cities are hotbeds for tech too (Austin, NYC, etc), but truth be told, you could find a cadre of smart engineers doing a great start-up in a Des Moines, but it's not easy. There's LOTS of great companies in Silicon Valley that can take you to the next level.

We're in the midst of a huge wave. Depending on your risk profile, joining a start-up or joining "a Google" is the best way to put your chips in the game. Regardless of where you are - if you're a crack-shot engineer looking to change the world, you could do worse than coming here. Again it's all about your risk profile and what's keeping you where you are (which may be great reasons). Start-ups will not only pay to relocate you, we'll put you up for a few months (in the corporate crash-pad) while you find your own place. Joining a start-up now will get you experience both technically and start-up-wise that you can't get anywhere else.

I'm not thinking the start-up life is for everyone. I can definitely see a point in my life or where I have life-constraints where I'll want my job to be a less important part of my life (probably because my life will be more about, well, you know - just "life"). But for me right now, and maybe for you - I'm swinging for the fence. And love or hate IT departments, I couldn't do that there.

Again, if you're a software engineer that loves what you do and lives in commuting distance to Palo Alto, CA or is willing to relocate, I'd really love to hear from you. We're well-funded and I'm literally building the first engineering team right now. It's a fantastic opportunity to get in on the ground-floor of a great start-up. Refresh.io jobs

Mailinator sponsors Wasteland 2

2012-04-05T16:01:00.002-07:00

Mailinator is a proud sponsor of the Kickstarter project Wasteland 2.

http://www.kickstarter.com/projects/inxile/wasteland-2

(Boy did I waste some weeks on Wasteland-1 a long time ago).

Anyway, we sponsored at the $1000 level giving us the ability to name an in-game weapon.

When you play - expect to find a (very) big gun called "The Mailinator". Let them eat spam!

(I hope I can get them to have it shoot spam!)

How Mailinator compresses email by 90%

2012-02-21T07:51:00.001-08:00

Given the title of this article, the first thing that should pop into your mind is probably - "well, use a compression algorithm - right?".

Right! Well, yes, well, not exactly. Read on.

Your second thought might also have been - "Why bother? Just buy more disks." Which in the big picture is also not a bad answer. But for Mailinator that doesn't work - if you have read previous Mailinator tech articles you might know that Mailinator stores all it's email in RAM.

There were good reasons for that when Mailinator started. One was the use case - which was always disposable email that lasts a few hours (rather longer nowadays). Secondly, when Mailinator started, disks and datastores weren't as sophisticated/fast as they are now.

Also, Mailinator is/was always a free service so keeping costs down was always important. To this day, Mailinator runs on a single server. It averages about 4-5Terabytes of bandwidth a month and the peak incoming email rate I've seen is about 3500 emails/sec (this is just a production observation, server limit is bandwidth, not CPU).

And finally - last but not least - to me, much of web and application development today is utterly devoid of any fun algorithms. I spend a non-trivial amount of time in interpreted/dynamic scripting languages that do a fantastic job of hiding (or at least lure me away from thinking about) algorithmic complexity. I've probably inadvertently written more n^3 algorithms than, um, (n^3)-for-some-large-value-of-n.

Mailinator has always been my test bed for trying fun ideas, algorithms, and datastructures. In other words - I probably didn't need to do all the work I'm writing about here - but I definitely did have fun doing it (probably should have been out talking to girls, but alas).

Compression

Ok - so back to 90% compression.

So to start testing, I grabbed a few hundred megs of the Mailinator stream and ran it through several compressors. Mostly just stuff I had on hand 7z, bzip, gzip, etc. Venerable zip reduced the file by 63%. Not bad. Then I tried the LZMA/2 algorithm (7z) which got it down by 85% !

Well. OK! Article is over! Everyone out! 85% is good enough.

Actually - there were two problems with that result. One was that, LZMA, like many compression algorithms build their dictionary based on a fixed dataset. As it compresses it builds a dictionary of common sequences and improves and uses that dictionary to compress everything thereafter.

That works great on static files - but Mailinator is not a static file. Its a big, honking, several gigabyte cache of ever changing email. If I compressed a million emails, and then some user wanted to read email #502,922 - I'd have to "seek" through the preceding half-million or so to build the dictionary in order to decompress it. That's probably not feasible. And, as I said, the Mailinator cache is constantly throwing out old emails and putting in new ones.

In other words, an algorithm that relies on previous entries to build a dictionary can't work given that we keep purging the front of the stream never to be seen again.

Hence, we cannot compress emails "together". But we can compress them individually. Sadly, this hurts our compression ratio - and by a lot. The algorithm now must start building a new dictionary with each email. And emails are small so the dictionary isn't very mature by the time we're done compressing in many cases.

We can help this situation by giving the compression algorithm a pre-built dictionary. That is, scan a typical piece of data to be compressed, find common sequences and create a list of them. Then we give that dictionary to the compressor/decompressor as it takes off.

Woopsie. Again, the Mailinator stream is a living and breathing entity that's always changing. One minute might be a few million viagra spams, the next minute might be all about fake rolex watches. In other words, there is no "typical piece of data" - a static dictionary built off a sample of emails will be obsolete in relatively short order.

So, the first idea was to build a sliding dictionary builder. Each email is scanned for string occurrences and we keep a count of them. Then every so often (minutes or hours), the compressor switches to using the most recently constructed dictionary. Every compressed email is given a reference to its dictionary so when/if it needs to be decompressed, it knows what dictionary to give the decompressor. Many thousands of emails share the same dictionary so RAM to store dictionaries isn't particularly significant.

Well, that's great and does restore LZMA back to about 60-70% but remember I mentioned I had another problem with LZMA? Speed.

The C++ version of LZMA by Igor Pavlov compresses at about 1.7MB/s per CPU core on my test machine. Um. no. Firstly, Mailinator can pull down tens of MB per sec at times. Secondly, no component of our processing pipeline can be allowed to take up this much CPU (my rule, not yours). We need our CPU for other things when large volumes of mail arrive. (The java version by the way was about the same speed).

Simply - LZMA is pretty awesome - but it's too slow for this purpose.

So the for the moment, I fell back to using a fast but simpler compression (zlib/LZW) on individual emails - and we sink down to about 40-50% savings from compression.

A Bigger Idea of a "Dictionary"

The next step for me was to think about email composition. We get lots of different types of email - but we get lots of the same types too. For example, we get lots of newsletters (people send them to Mailinator then read them via POP or RSS).

The nice thing for us is that a newsletter email blast could be 10,000 emails that are, all the same. Well, ok, not exactly - no two emails are ever the "same" because headers have times, dates, message-id's, etc. within them. But if we remove the headers, you can get 10,000 emails going into 10,000 different inboxes that all have the same message "body". Are you thinking what I'm thinking?

Right - store each email with it's own headers plus a pointer to ONE system-wide byte-array containing the newsletter body. What's the "compression" ratio of that? Well over 90%. And just to be a snot we can then apply compression to that byte array to eek out another few percent. We're reusing memory here so it's not exactly "compression" but we are reducing the size of the data sent to by some fantastic amount for this happy use case.

This isn't a revolutionary idea (online music libraries do the same thing) but it does fit pretty nicely in the Mailinator paradigm. Sadly apart from newsletters, not many other email sets, spam or otherwise have email bodies that are identical. In fact, spammers specifically change the subject line and destination url of every email they send for tracking and spam-detection-thwarting purposes. So what you get is something like this (headers omitted):

Email 1:
Buy vi4gra now!
http://rrr4.somerandomthing.com/?3jwow33oo
Happy man are you will be!

Email 2:
Buy vi4gra now!
http://1rr220.somerandomthing.com/?ajo200kko
Happy man are you will be!

So much for simply detecting identical email bodies. And this goes for less nefarious things too. Sign-up emails from websites will contain the same surrounding text with different names and validation urls inside.

What we could use here is a Longest Common Substring (LCS) algorithm. Basically, it would compare the two email bodies and be able to break them up as:

Common string 1:
Buy vi4gra now!\r\nhttp://

Disparate strings:
rrr4.somerandomthing.com/?3jwow33oo
1rr220.somerandomthing.com/?ajo200kko

Common string 2:
\r\nHappy man are you will be!

Nice .. each email is stored as 3 (compressed) byte arrays where 2 of those can be shared.

Unfortunately, classic LCS algorithms are expensive. Comparing two sequences is an O(nm) algorithm. And we're not interested in comparing two sequences, we're interested in comparing each new sequence (er.. each new email) with the few million that preceded it. Also, the LCS algorithm is also very memory expensive in the creation of trie datastructures - again, scaling to millions of emails just doesn't fit in our parameters.

Generally speaking, there are a lot of tricks I've noticed in analyzing algorithms. A few off the top of my head are: if you see an easy O(n^2) algorithm, it's rather likely there's an O(nlogn) one hiding in there somewhere. In contrast, if your dataset is small, you might be better off sticking to algorithms that make your CPU's cache and instruction pipeline happy instead of worrying about algorithmic running time (i.e. bubblesort > quicksort for small data). Lastly - if you can make assumptions about your data, you can often short-cut the classic algorithm with an good approximation.

Caching Lines

Cool, so let's assume something about the data. For emails, as it turns out, disparate parts of emails often occur on line boundaries (as you see in lines 1 & 3 above). A few same lines, a different one, a few more same. Instead of looking for common sequences based on individual characters, we can treat individual lines as units. Then we can attempt to find multiple occurrences of those lines. It cannot be as precise as LCS proper as in our above example (we would not find the identical portion "http://" in line 2) but we're basically settling for a greedy approximation, and one that works pretty well.

How do we store it though? LCS's tries would kill us. I know - let's use an LRU cache. Those darn things work for everything! We can use an LRU cache that caches full email-lines. It will inherently flush out old email lines as the spam stream evolves (nice!) and will provide quick look- ups to compares thousands of lines at once (happy!). Specifically in Java, an LRU-cache is a synchronized LinkedHashMap with true as the last constructor parameter and an overridden removeEldestEntry.

So we store a few 10's of thousands of email lines in an LRU cache and then as each new email comes in, we check to see if that line is in the cache. If it is, we reuse the one in the cache instead of creating new storage for this email. By assuming all common sequences are bounded at newlines, we remove the boundary-discovery work LCS must do. Strictly speaking, we're cheating and losing some opportunity, but it's a good enough guess for this type of data.

This had a dramatic effect on our "compression" (again, it's slighty dubious to call it compression but, as you consider the big picture, our entire machinery of the LRU cache and bastardized LCS-in-spirit algorithm is creating a reuse-dictionary, it might not actually be compression - but it goes through several of the motions).

Caching Multi-lines

Caching lines is great - but what about caching multi-lines? Say we have a few emails - for brevity, assume each character in the following examples are email "lines":

Email 1:
ABC1

Email 2:
ABC2

Email 3:
ABC3

Email 4:
ABC4

So the first 3 lines are all the same in each email (ABC), the 4th lines are numbers which are not the same. Our algorithm:

1) Load a LINE and see if it's in the cache (if no more lines, quit)
2) .. if it's not there, put LINE in the cache, and store LINE in the email - GOTO 1
3) .. If it IS there:
4) .... see if LINE + NEXT_LINE is in the cache
5) .... if its not there, put LINE + NEXT_LINE into the cache and store LINE (which is a cache hit) in our email - GOTO 1
6) .... if it IS there, LINE = LINE + NEXT_LINE, - GOTO 4;

So if we run our 4 emails above through this algorithm. We get the following:

Running through all of email 1 - we get:
- Cache HITS stored in email: none
- Cache MISSES stored in email: A,B,C,1
- Cache contents afterwards (lru order): 1,C,B,A

Running through all of email 2 - we get:
- Cache HITS stored in email: A,B,C
- Cache MISSES stored in email: 2
- Cache contents afterwards (lru order): 2,C2,C,BC,B,AB,A,1

(notice how '1' (which didn't cache hit) has worked itself to the end)

Running through all of email 3 - we get:
- Cache HITS stored in email: AB,C
- Cache MISSES stored in email: 3
- Cache contents afterwards (lru order): 3,C3,ABC,AB,A,2,C2,BC,B,1

Running through all of email 4 - we get:
- Cache HITS stored in email: ABC <-- very cool result, note coolness
- Cache MISSES stored in email: 4
- Cache contents afterwards (lru order): 4,ABC,AB,A,3,C3,C,2,C2,BC,B,1

So what happened? The system has realized that ABC is cacheable and is now pointing to that. All subsequent emails with the set-of-lines ABC will reuse the same memory. Note that the disparate lines 1,2,3, and 4 will always be stored separately, but the algorithm will then pick-up any common line-sets later in the email too (if there were any).

This elaborate system to find equal email lines and reuse them drags out compression of the entire flowing email stream down to about 80%. What about 90%? Well.. one more trick.

Back to LZMA

Remember LZMA from above that we abandoned because it was too slow to happen inline? As you'd guess, the biggest impact it had was on bigger emails. And although it's a CPU hog, we do actually have a few cores laying around. So let's give it one (but seriously, just one).

We setup one core (i.e. thread) to trail behind and scan incoming email for ones that are over some size (say 20k) and re-compress those using the sliding dictionary LZMA we mentioned earlier. While 3 of our cores average 5-10% utilization by receiving, analyzing, and storing incoming email - the 4th core sits at 100% re-compressing emails where it will find benefit. If it gets too far behind, it simply leaps ahead and leaves some compression on the table.

(Note that empirically, LZMA is an order of magnitude faster decompressing than compressing, otherwise that would have been a new problem as it could take too long when someone wanted to read an email)

Voila. 90%. (Two notes: 1: that's a reasonable average at least... sometimes better, sometimes worse and 2: I realize I'm not exactly sure what "Voila" means, looking that up now).

There are also some other important notes. Storing a byte array in Java costs something. The pointer alone (64bit) is 8bytes. Then there is the byte length field, padding, etc. In other words, I limited the system to never store email lines under 64 bytes. Small lines get concatenated together straight away.

Second, there are more email-idiomatic tweaks we can do to improve the situation. Base64-encoded attachments are effectively un-cacheable, so we pass over those.

Third, although from our cheeky example it may seem like we're finding optimal line sets (i.e. ABC). We're not. We could end up caching ABC and destroying an opportunity for a more optimal BCDXYZ or something. I'm guessing this doesn't happen often but would be an interesting future consideration.

Edit: Wow, sincere thanks to an Anonymous commenter for making me reconsider the above algorithm. I had originally stated it was O(n^2). My first version was indeed O(n^2) (which wasn't written about) and after a few changes it became O(n) and I failed to see that. I find its very easy to find tech reviewers once an article hits Hackers News, before then though - not so much. :) My apologies for the error.

So for the end-user, this whole diatribe simply means little except their emails are sticking around longer. They have no idea that when they click to read an email we may be LZW or LZMA decompressing tens of byte arrays shared by thousands of emails with a custom-sliding dictionary built by scanning emails that arrived hours ago and then catenating them together so they can be shown on their webpage all in a few milliseconds. And they likely don't care, they're probably too busy signing up for Minecraft or something.

But that's ok. I know.

And if you got this far, you know too.

Ok.. now back to real work. What was I doing again? Oh yeah, writing some slick one-liners in Ruby. No clue on the running times - probably like O(n^4) or something, but if I fiddle with it a bit more - I bet I can cut the character count of the code by half!

RickRolling 1000 people a day

2011-12-05T07:31:00.001-08:00

I wrote previously about the voracity of some folks to discover Mailinator's list of alternate domains. If you recall, the point of alternate domains are, to me anyway, to alleviate the worry that websites might 'ban' email addresses ending in the mailinator.com domain.

People use Mailinator when they want to use a website but not give out their primary email for fear of future spam. Preventing that use is asking the customer to reassess a decision they've already made (not wanting to give out their real email) which of course might result in their answer being "nah".

Or, of course they could decide to simply go create a fake yahoo account. That takes a few minutes and before Mailinator, when I tried that I often found myself losing interest in the whole idea before I was done.

In my last blog post I mentioned we get emails similar to this from time to time:

Hi! Love your service. Can you send me the exhaustive, comprehensive, and complete list of alternate domains so I can pick a nice one that suits my individual personal style? kThxBai

And when I say from "time to time", I mean like last week too. Giving out that list of course would be completely counterproductive to the point of having the alternate domains.

I did some minor front-page spruce-ups this weekend (like the dinosaur eating spam?) including giving a relatively real-time count of the amount of email Mailinator gets in the last day (15 million when I just glanced at the homepage), I made the randomly-generated names more readable, and matched them up with alternate domains as examples.

In addition, I added a bunch more alternate domains. Also if you recall, you can point your own spare domains at Mailinator and send email to those - then you have your own Mailinator alternate domain!

At this time there are literally several hundred (maybe over 1000, have to check) alternate domains.

It occurred to me though, that I might be able thwart some of the people emailing asking for a full alternate-domain list by actually giving them a link where they can see the only version of the full list I'm willing to publish. You'll see it on the homepage as:

Want to see a list of ALL Mailinator alternate domains?

I made the link small and didn't expect many people to click it. Maybe just the curious few. Well, to my surprise, the curious few turned out to be about 1000 a day. Nice. I surely underestimated the number of curious people out there.

Feel free to click yourself, but don't expect to actually see a list of any domains Oh, and enjoy the video. :)

How to get gmail.com banned - not that I did this

2011-07-01T16:40:00.000-07:00

When I started Mailinator, a LOT of people told me it wouldn't work because websites would ban it right away. Ban it with reckless abandon. Ban it like the new thing on the internet was to just sit around and ban Mailinator all darn day long.

As it turns out, that didn't happen. Sure, some sites do ban Mailinator and some are even really (really) excited about the idea, but in the grand scheme of things, it's not really very many. Thousands of people use Mailinator everyday, so clearly, its a useful tool that many sites accept.

Back in the day however, I sadly fell prey to the words of doom that I was being fed. I mean, holy mackerel - what if sites DO ban it? What then?

So I drew up a plan. A plan, that at this time I can say I may not be fully proud of. A plan that involved guile, wit, a few domain names, and some rate-limiting (thread-safe) data structures.

I write this now because, well, for the most part the war is over and Gotham has grown past needing Batman anymore. Mailinator is not really the rogue tool it once was. Heck, hotmail supports disposable email now. It's mainstream.

Typically there are two reasons people want to ban Mailinator. A few years ago, people really had some sort of notion that your email somehow equated to your identity. Given the radically insecure setup of email in general, that was really a ridiculous technical assumption. Nonetheless it was pervasive.

Secondly, people banned Mailinator for fear of people abusing their website. Now keep in mind, anything you can do with Mailinator, you can also do with YahooMail or Hotmail. Its just that Mailinator lets you do it faster, but Yahoo is plenty happy to let you sign-up for 100 email accounts.

I get occasional emails from people asking me to have Mailinator stop accepting email from their site. Usually for the reason of stopping abuse. If they're nice and it makes sense, I almost always do it. But in my experience, usually when the existence of Mailinator is pinpointed as a cause of abuse, it is in truth merely an avenue that is already inherent to the internet or your website. Even shutting Mailinator down wouldn't solve the problem. The bad-guys just go somewhere else and keep on abusing.

Any sort of abuse is needless to say, no fun for anyone. Mailinator has specific code built-in to detect scripts and stop them.

In truth, Mailinator's system for detecting and shutting-down scripts and abuse really only serves one purpose. Its like that silly metal bar people put on the steering wheels of their cars. Let's be real, if a thief really wants your car, some dinky metal bar on the steering wheel isn't going to do diddly to stop him.

Same with Mailinator's anti-abuse code - it won't stop a determined person - but it does make it more of a pain than simply using something else.

So -Dear hacker bad-guy abuser dudes - other disposable email sites probably don't have that sneaky anti-script pain-in-the-butt hacker-stop code like Mailinator. Go use them.

There....

1) Solved hacking and abuse on the internet ? --> Not even maybe
2) Solved a little of the hacking and abuse on the internet possibly for me? ---> DING!

Ok, back to the story - as I said, in the beginning, the idea of wide-spread Mailinator banning scared me a lot. So what did I do? I bought some additional domains for Mailinator.

To this day, you can email bob@thisisnotmyrealemail.com and it will end up at mailinator (in the bob inbox) just like bob@mailinator.com.

Cool. Alternate domains. Problem solved.

Wait a second. How exactly do I tell the world about the alternate domains without telling the people that want to ban them all?

Every few weeks I get an email like:

Hi! Love your service. Can you send me the exhaustive, comprehensive, and complete list of alternate domains so I can pick a nice one that suits my individual personal style? kThxBai

At first, I was like "Neat! People love Mailinator and want to...heeeyy.. waaiiit a second".

If I give them the whole list, then they will, um, have the whole list. And then they can ban the whole list.

Ok. I know. I'll list one random alternate domain on the homepage every time you visit. No one will have the whole list. Just one here, one there. Perfect !

There problem solved. Again. Well, sort of.

Soon after I put up this "one random alternate domain per homepage load" system - the scrapers started. Every now and then I'd notice several hundred homepage loads from the same IP in a very short period of time.

They were scripts; scripts that were loading the homepage over and over and scraping out the random alternate domain that was shown. Sneaky. By doing this they could eventually formulate the entire list of alternate domains.

Drat. Now what. For awhile, nothing. I just let them go. A few months later however, I got an email from a Russian guy (sorry Russian guy, I don't remember your name).

You are dumb. Your homepage is easy to scrape and doesn't change so its easy to scrape your alternate domain. You are dumb.

He was right. Well, I'm not sure about the dumb part, but my homepage was easy to scrape. Someone could probably write a script to scrape it in short order. Probably just took a few minutes.

Could I make it harder to scrape? Well, I could, but wouldn't really slow anyone down much.

It was then however, I had a flash. An idea of simply epic proportions. A thought so crazy - that dad-burn-it, it just might work.

Let's not make the page scraping harder - let's make it EASIER.

I removed the bit of code that displayed the alternate domain and put it in its own (teensy) webpage. That "webpage" had absolutely nothing in it, except the text for the randomly chosen alternate domain itself.

Then, I embedded my new tiny webpage into the homepage (so it showed as before). Basically, to the viewer of the homepage - nothing was different. You saw the homepage and a randomly generated alternate domain, just where it was.

But to the folks that had been scraping my site, things looked plenty different. In fact, I probably broke all their scrapers (Sorry nice people trying to get all my alternate domains just to ban them! (ok, not really)).

Now here is a finer point of semantics. If you go to the Mailinator homepage, there is some text that says "Here is an alternate domain" followed, by, well, a randomly chosen alternate domain.

However, now that I split off that tiny little webpage with JUST the alternate domain in it - you could go there too by typing in the url directly. And you'd see nothing BUT the alternate domain. No surrounding text. No text saying "this is an alternate domain". That little page showed a domain, but made no claim about what it was displaying.

For your browsing pleasure, here's the only direct link to that page that I know of: Go ahead, reload the page a few times. (You can see this also on the Mailinator homepage on the lower left).

After the script guys got over the minor annoyance of their scripts breaking because o f my new setup, I'm sure there were office parties across the nation. Mailinator! Now even easier to scrape!

Now for the record, the rest of this post is hypothetical. An unimplemented idea if you will. Who knows - I'll bet nothing you read here on out ever happened. Just random thoughts. Musings. One big theory. Consider it random daydreams of guy who runs a fun email service.

Remember all that script-detecting code from the anti-abuse system? Well, what if I put that in here too I thought. Let's "detect" when a script is hitting our weensy alternate-domain page.

And, what if we also detected when the little web page is being viewed but not "in" the homepage - but by itself (just like the link above). And what if after about 30 page hits from the same script (or so), stop displaying actual alternate domains and start sprinkling in some other things. Hmm... but what other things?

I know - how about "gmail.com". Or, um "hotmail.com". Or maybe, "yahoo.com".

What, in our completely and totally hypothetical situation, would that do?

Well, let's see. There are these folks out there running scripts against Mailinator collecting all my alternate domains. Those scripts probably put results in a database or something and connects to their website. When one of their users tries to sign-up on their site using one of my alternate domains, it's in their database as a banned site and its immediately rejected.

Now imagine the wacky fun if somehow, some way, (totally theoretically speaking) some silly person snuk "gmail.com" in that list. I'd guess banning your users trying sign-up with "gmail.com" addresses is probably not what you want.

And, hypothetically speaking if you had code that would sneak in these non-alternate-domains in the page they weren't supposed to accessing anyway, when would be the best time to set it into action?

Well, those scripts ran at many different times, but just after midnight seemed like a popular time-slot.

If such code existed, making it active Sunday morning from Midnight to 2am seems nice. I mean heck, if my website stopped accepting signups from "gmail.com" on some Sunday morning, I'm sure I'd be downright chipper to hop into the office and find out why.

Boy. If all that stuff happened - I wonder what kind of email conversations I'd have on that Sunday afternoon? I bet they'd be like:

Your alternate domain list displayed 'gmail.com'!
Hi Fred, no it doesn't. Just reloaded the homepage 10 times, nothing like that. all the best.

or I bet another would be like:

Yahoo.com? What is this some kind of joke?
Sorry, did you mean to email this to Carol Bartz? Not sure what you're talking about.

Phew. Well, that's surely a fun thought experiment. As you can see from the link above however, it surely doesn't do anything like that. Honestly these days, most of the scrapers are gone. I think simply that the internet evolved and more of them simply lost interest in the fight.

Every now and then I'm still asked what I think about banning Mailinator. I've mellowed a lot since the early days and I pretty much always give the same answer. If you think banning Mailinator going to solve your problem, go ahead. In my experience, it won't. And by asking I am guessing that you are making some assumptions in your site that will surface as issues in other ways.

And of course, script writers, you now have the direct link to the alternate domain page above. Scrape away. But keep in mind, the best way not to trigger any Mailinator abuse systems is to not do anything "too fast". Those script detectors are pretty fickle little beasts. It's not a bad idea to try and stay on their good side.

Why you see the Ads you see

2011-06-15T11:05:00.001-07:00

Since we've launched Clickochet (as in "Click Ricochet", an Ad Trading Network), I get a rather constant flow of questions about how we do ad targeting.

Honestly, I've never been a person to use an Ad-blocker and the reason is that if I go to a website that brings me value, it's a small price to pay to see ads in exchange. The value that Google search brings me EVERY day is immense - and all I have to do is view ads that once in awhile are actually what I was looking for anyway? Deal. Reddit actually thanks its users for not using ad-block.

The trick here however is to bring people "good", "quality" and "relevant" ads. And it is of course, our goal at Clickochet is to try to bring you the most relevant ads to you as we can.

So, after writing a few responses to nice Clickochet users about how our ad-targeting works, I've decided instead to simply - let you see for yourself. At least for now, let you into the belly of the beast.

After spending a good number of months staring at ads, I'm a bit obsessed with them now. It's rather rare that I don't go to a page with Clickochet ads and push my nose to the screen wondering "Why is THAT ad showing THERE?!"

I usually (usually) then continue that self-conversation with, "oh. oh. ok. that makes sense." But to be able to answer that for myself, Clickochet uses some diagnostic cookies to turn on extra information about why you see the ads you see. And actually - more important for me - it has facilities to tell me why you don't see the ads you don't see.

A few disclaimers about what you're about to see. Pretty much, 100% of this stuff will change. Ad targeting is an exercise in perpetual tweaking and change - and given the infancy of our network, you can count that statement as double. Also, keep in mind that Clickochet is a new network. We're working furiously to build the best network for our customers that we can - but our targeting mechanisms are relatively boilerplate at this time. Finally, this stuff isn't for general consumption - in other words, it's not meant to be "pretty" nor am I guaranteeing it works across browsers (so use firefox or chrome).

So to begin, in a separate browser window or tab, hop over to one of Clickochet's favorite yoga sites, Qi-Yoga.

http://www.qi-yoga.com

Nice ads huh? And chances are you'll see an ad like this:

I guess you'll see that particular ad because:

1) I chose "yoga" specifically because we have a few, but not too many yoga sites in the system (at the moment) so that determinism helps writing a semi-predictable blog post.
2) This ad is contextually relevant to qi-yoga and our contextual system realizes that. (the two sites are not otherwise related)
3) I don't know the fine folks over at HealthAndYogaRetreats.com but I slipped them a nice pile of extra impression credits so that their ad won't "run out" in the middle of my examples.

Ok - keep qi-yoga.com up in a browser window and open yet another browser window (welcome to my world). This time go here:

http://www.clickochet.com/pages/diag

You don't have to sign-up or be logged in. Again, this is arguably an internal page. Press the "Turn On" button for both options. Note: The 2nd option CAUSES A POPUP for every page you load that has Clickochet ads. Again, this is a diagnostic page no one sees unless the turn on that cookie. And incidentally, to follow along, you'll need to tell your browser to "allow popups" from qi-yoga.com I'd guess.

Great. Now refresh the qi-yoga.com page.

Welcome to the diagnostic view. Ad titles are now color coded and have been suffixed by a single character. That character signifies the targeting mechanism used that turned that ad "on".

The popup window is a list of most of the ads that were considered for showing on this page. (I say "most" because some might have been short-circuited once we found 3 acceptable ones.)

The popup you see should look something like this:

The first 3 ads in the diagnostic ad window are the ones that were shown. The rest (and sometimes the list is long, are the ones that were rejected). The lists will perpetually change as you hit refresh (the system knows when you hit refresh over and over and no one is charged or awarded credits for that behavior). Also, keep in mind that refreshing the page a few times will help our examples here but aren't a normal ad viewer use-case. People typically see one set of ads and then leave the page. So you'll likely start getting less relevant ads the more refresh.

So, what are the title suffixes you might see? The represent the different ad targeters and specifically which ad targeter decided to show you that particular ad.

(C) - A "contextual" ad, for some broad definition of "contextual". Back to my theme that "everything will change", the contextual engine is finding its sea legs. It sometimes expands its search and sometimes contracts it to find ads that could be relevant (it then keeps track and hopefully finds a happy place). Hence, you might see "contextual" ads you don't think are all that contextual. Don't worry - if that's true, they'll cycle themselves out eventually. Again, I chose the yoga theme here because of the median set size of what the contextual targeting finds.

(A) - A "site affinity" ad. Refresh the page a few times if needed to see HealthAndYogaRetreats again. You'll notice it is an affinity site. Yes its contextual, but more importantly, people have clicked from qi-yoga to healthAndYogaRetreats a few times. One click won't do much - but a pattern of clicking and the system takes notice. Those sites become (spiritually) linked in the system.

This is a very important feature beyond contextual advertising. The classic example is that "Budweiser" ads might do very well on a "Nascar" website. But contextual analysis might not determine that (in fact, drinking and driving might be a negative contextual correlation!). But of course let's hope the budweiser drinkers are watching the races, not participating. So when they click the budweiser ads, the system realizes "huh, for some reason these two very unrelated things have an affinity for each other".

Affinity relationships perpetually degrade and require, again, a pattern of clicking to keep them alive.

(U) - An underserved ad. Simply put, there are ads in the system that haven't found a solid niche to display in. These ads can be pretty non-relevant at times but the idea is that they won't individually show up all that often on any given site. And showing them serves two purposes -

1) to get underserved ads some impressions, and
2) remember site affinities from above? How do we 'discover' site affinities if we only show direct contextual ads. Showing ads from a larger pool gives the affinity system a chance to discover unexpected affinities.

You'll see more underserved ads if you refresh the page a lot - which again, isn't the "normal" use case.

(1) - actually, any numerical digit you see means we're running an experiment. Again, every last thing here is subject to change, but experiments are even more transient than that. One experiment we're working on is transitive relationships between site affinities. So if Budweiser ads are good on Nascar sites, and then we discover Nascar ads are good on fishing sites - are Budweiser ads good on fishing sites? (I'm guessing that's an affirmative)

If you'll notice, none of the targeting above has to do with YOU (or more specifically your browser since we never know about "you"). All those targeting mechanisms have to do with the sites, ads, and contexts involved. Site affinity is based off clicks but it's really about all clicks in the system - not really about you.

That being said - it's purely possible that although you like both budweiser and nascar - you might like yoga too. And although you are an awesome beer-drinking race-car-driving yogi - that's probably not a common combination. Showing a yoga ad on a nascar site as a general rule is probably wrong - but for you specifically, it might be right.

So, what we can do for you specifically is try to find things that you like. This is of course where people get worried about advertisers finding out private information about users. Clickochet does not keep personally identifiable information about anyone who views the ads (please see our privacy policy on the site). We keep a cookie noting where a browser goes (and note that if people share a browser, that browser becomes a collection of all browsing destinations, regardless of any given user).

The goal again is to show you the most relevant ads possible. That brings the most value to people viewing ads and the most value to people showing ads. Of course, Clickochet has an opt-out cookie (see the front page of Clickochet for a link or go directly to http://www.clickochet.com/pages/optout to turn it on). But! Don't turn it on just yet or the rest of this blog article won't work!

Ok. Let's go to another Clickochet ad site - oh, wait, you're already on one. This blog has come Clickochet ads on it in the right-hand side-bar. Ok, note the ads you see there and .. hit refresh. Here, I'll bookmark this blog entry for you so you can pick up reading here.

BOOKMARK FOR WHEN YOU REFRESH THIS BROWSER WINDOW TO SEE RETARGETING

Ok .. hit refresh.

BOOKMARK FOR WHEN YOU REFRESH THIS BROWSER WINDOW TO SEE RETARGETING

Ok.. great. Now.. what ads shown? There's a high probability (note: EVERYTHING involved here is a probability) that you'll see a red ad for Qi-yoga followed by the suffix: (R). (if not refresh the page once or twice)

(R) retargeting. It seems almost counter-intuitive but the ad industry has plenty of documentation that people will click on things they already have seen - and/or simply want to see again. The system now knows you went to qi-yoga.com (because I told you to earlier in this case). And thinks, "Hey.. this person likes that site" - so it gives a reasonable probability to show ads for qi-yoga to get you to go there again.

There's surprising advantages to this. If you liked qi-yoga and saw it last week, you might be interested in seeing if the site is updated. Qi-yoga likes this because as with all websites, there is a lot of bounce - people come, go, and never come again, This reminds you of that great site you saw. Clickochet likes this because it keeps you within our community of sites that are helping each other exchange traffic.

And then.. we can run experiments based off that correlation. Maybe there is a site affinity between this blog and qi-yoga. Or, maybe its simply a personal preference of yours and maybe we can help you discover other yoga sites. The possibilities are endless.

If this all seems odd that all this is happening in the background, I'll note that Clickochet is a newbie in this arena. Every ad system out there does this and more (goto zappos.com, browse some shoes and then browse around the web - see how many zappos ads you then see).

As I said - you can opt out of this type of browser-based targeting on our opt-out page if you like. But there's nothing more nefarious there than simply trying to show you ads for things you might be interested in.

Again, our goal is help our community of sites trade traffic for traffic. As we say in our FAQ - give an ad, get a few - the system really does work. And it will work better and better with the more value we can bring to everyone involved including publishers, advertisers, and web-passers-by.

Sign-up (it's free!) at http://www.clickochet.com

Introducing Clickochet: A (free) Ad trading community

2011-05-10T12:12:00.000-07:00

I have a lot of ideas for apps/websites/start-ups. No seriously - I mean a lot. Some of them aren't all that bad either. And some are too - really bad, like "waking up in the morning and thinking 'what the heck were you thinking'"-bad too. I mean really.

Alas, usually even with ones I think are awesome - I typically end up finding some fatal flaw. It could be business model or timing or ... who knows.

There's a lot of factors that make me consider an idea for more than a few minutes much less actually work on it:

1) Is it in my skillset
2) Is it the right scope - too small and its a weekend project, if its too big, am I the right guy to assemble the infrastructure (team, investors, etc) to bring it together.
3) How hard will customer acquisition be?
4) Does it have a viable business model?

There's a lot more subtle questions, but these are some of the bigger disqualifiers.

Once in awhile however I get one that sticks. One that I say "why didn't I think of that !" - and then I say "Oh. I did!". Mailinator was one. And this - this is another.

Last fall my girlfriend started a celebrity news blog. I'm not much into celebrities but she was pretty passionate about it and it was exciting to share her enthusiasm.

As with any new site, she was trying to gain traffic. So to help with that effort, I bought her $100 worth of adwords. I've bought adwords for many different companies and projects, but I was still rather surprised with the results. For $100 I got a 3-day (or so) ad campaign that yielded 100 visitors.

The interesting part was that all this time, she was running Adsense - that is, ads to make money on the site. Now, ask anyone trying to make money with Adsense why they don't "buy" ads to gain traffic and they'll tell you, it's losing proposition. And no wonder - that's the crux of how ad networks make money.

Sure, they sell ads. But for a moment, consider a website that does both - it's both an advertiser and a publisher. In that scenario, ad networks sell you ad impressions for money - and then trade you money for ad impressions. Simply put, there is an exchange rate on ad-impressions to money and just like the kiosk at the airport which changes currency, they take a "cut" in both directions.

And the cut can be pretty big. In the case of my girlfriend's site, I successfully managed to convert $100 into 15cents by changing it into ad-impressions and back again (Actually its even worse because the girlfriend gets to keep the 15cents, but we don't bring that up).

Now my assumption above is for sites that are both advertisers and publishers. Many small sites are publishers, but not so many are advertisers. So my assumption is, well, rather assumptive.

But why aren't they advertisers? They want traffic too right? Advertising is a way to get it right?

The answer is that those sites would do both, but its non-economical to do so. Buying ads to show ads makes no sense. Even just buying ads is a very expensive method of traffic acquisition - but what if ads were really (really) cheap. Then everyone would buy them wouldn't they? I mean, it would be a cheap method of traffic acquisition.

Ad networks (BTW, I don't mean to pick on Google above, I just had been using their service because they are one of the most accessible ad networks for small sites) provide a valuable service. Clearly - the ad industry is highly alive - they are bringing value to lots of people.

But it occurred to me - how do I make an ad system that brings value to smaller sites. The under-served billions of websites where buying ads makes no sense today. How about I make a system where you trade ad-impressions for ad-impressions. That simple. And, instead of making it where you give something and get a lot less back, what if we fliped that. That is, give one ad impression and get a few back?

I call it Clickochet. Like "Click Ricochet" because that's what it is.

So how do you get more back than you put in? Well, consider your normal banner text ad. In reality, its one ad impression on your website, but it contains ads for 3 different websites within. When you show one "ad", three other sites benefit.

Clickochet is built as a community. We connect it closely with social networks that encourage social interaction and membership. A Social Ad Trading Network. This is key - you're not trading money and ads with a big company, but you're trading ads with each other. Other website owners looking for traffic just like you. Of course we strictly filter out porn and malicious sites, but being a good community member becomes an important factor on how much you get out of the system.

So, in a perfect world, you'd give one banner (keep in mind smaller ad creatives have less ads inside) and get 3 ad impressions. But it became quickly apparent, the world isn't perfect. If someone puts your ad at the bottom of their page, I doubt you'd be happy giving them an ad at the top of your page in return. Or what if their ad quality was low? Or conversely, what if their website had a huge click-through profile? In other words, not all ad impressions are created equal.

All these factors matter. So instead converting ad-impressions to real currency, we convert them to virtual currency. An exchange rate exists, but its minimal and its based on a lot of things - much of it tied to people showing good ads. If you show a banner ad, you won't get 3 ad impressions for it, but that isn't a far off guess either. You'll very likely get more out than you put in.

You are the publisher - you show ads on your site and earn credits.

And you are the advertiser - you spend credits and your ads show all across the network.

You can spend your virtual "impression credits" the instant you make them. Or you could bank them and save them for your new site's impending launch 6 months later. Or take your impressions (from a site like Mailinator) and spend them on your friend's site (like my girlfriend's).

This isn't about replacing existing ad networks. There's no need for that. If you're making money running ads - don't stop.

But unlike buying an ad campaign which costs a lot and lasts a few days (which you surely can still do), you simply give up whatever amount of page real estate you want for Clickochet ads and leave them there. And every day, from here on out, all across the network your page views are giving your a multiple of ad impressions on other sites.

To "prime the pump" I'm giving this virtual economy a boost - an "economic stimulus" if you like. I'm injecting all of Mailinator's ad impressions into the economy without taking them back out. Active users (i.e. you must be earning your own credits somewhat) will get lots of free ad impressions. (note: when I say "all" ad impressions, I mean "most" as I plan to advertise Clickochet on the Clickochet network somewhat).

Of course, it's free to sign up. Free to use. We may introduce a premium service at some point, but you can count on being a member of this community staying free.

Notes from "The Programming Leap to Multithreading"

2011-05-01T08:09:00.000-07:00

I gave a (techinical) keynote at the ICIS conference in Shanghai a year or two ago. It was a fun talk and I was recently asked for the notes so I thought I'd post them.

When I was at Google I spent some of my 20% time simply surfing through the Java code base looking for race conditions. It was a lot of fun and often I was patching code that was stuff that hadn't been changed in forever or was completely out of my normal stomping grounds.

After submitting the code changes with the comment 'Fix a race condition', I can remember at least one code reviewer responding, "Approved. Cool. But, what race condition?"

This talk covers a bunch of the common cases I encountered.

LeapToMultithreading.pdf

How I sped up my server by a factor of 6

2010-06-02T08:57:00.000-07:00

(with one linux command)

Subtitle: IO-bound and CPU-bound applications are common - here's maybe a "memory contention bound" app

I've written a lot of servers. If you've read the architecture of mailinator or benchmarking talkinator or "blocking faster than non-blocking" you probably already have that idea.

I was working on a new server infrastructure recently I needed for a new product. The server has a novel internal architecture to me that seeks to never voluntarily block threads or cause context switches - but at the same time is highly multi-threaded and creates new ones whenever it needs to.

The server's nature isn't totally important but you can think of it along the lines of a twitter server. It gets messages from one place, collates them, and sends them to (potentially many) other places. I suppose it could be used directly as a "twitter server" but of course, they already have those and my start up focus is rather orthogonal to that.

I usually write servers in Java on linux so keep that in mind for the ideas raised here. At Google, I worked on servers in C++ and Java. And especially after that experience, I'll stay with Java. To hopefully avoid language wars, the #1 reason I write in Java is that I'm way better at Java than I am at C++ or Ruby or Python or whatever. In other words, it was a practical (and definitely reasonable) choice in this case.

Also, the benchmarking here isn't contrived - I'll save the details, but I truly am asking the server to do precisely what it would do in the wild except at a higher rate.

Now, given the nature of the server, I needed 2 things to help benchmark. A "producer" (a system that produces messages) and a "consumer" (a system that consumes them). The producer produces a sequenced message set, and the consumer verifies it receives every message intact *and* in-order.

I chose message sizes at somewhat less than twitter size (msg size = 100 bytes) to induce a reasonable amount of activity in the server (server purpose is not necessarily large messages and large messages tend to just start saturating bandwidth without causing contention or server cpu activity). My custom protocols add a few bytes and TCP has a 40byte header - so overall I'm guessing that I'm using an average 200 byte messages counting everything.

Producer -> Server -> Consumer

The server system is pretty flexible and in effect, location transient. That is, a server process might live on one machine today and find itself on another machine tomorrow. Or a few new server processes may join the mesh. Needless to say, the idea is to create a scalable, flexible system.

One ramification of that is that it's possible and even very likely that some producers, servers, and consumers could be on the exact same machine. Socket communication over localhost changes a lot of assumptions. Firstly, TCP fusion can occur to reduce overhead significantly. Secondly, there is not an effective bandwidth limitation - there isn't a real network involved, it's a virtual (and fast) one.

Running all on the same server, I would expect this benchmark to be CPU-bound given that I/O is now virtual and effectively a CPU operation. Like I said, loopback is a real scenario I need but I initially think I benchmarked on a single machine mostly as a matter of convenience.

So, with all 3 processes running on the same machine, I ran the test. The CPU was a Intel Core I7 920. It has 4 hyperthreaded cores that act like 8. (note: I've tested all of this on a Core2Quad (non-I7) cpu and CoreDuos and results are effectively the same)

Here's the image of htop during the test with all 3 processes running. (if you run linux and use top, upgrading to htop is highly recommended).

Don't get hung up on reading the numbers or text. The graph in the upper left gives you a sense of how busy the CPUs are. In this picture, they're all at least a little busy. That's no surprise given we're running many tens of threads.

Notice that none of the cores are "maxed". This benchmarked showed the server to receive and re-send about 120,000 messages per second (that's 120k from producer to server and the same 120k from server to consumer - so 240k "messages transmissions" but only 120k messages - this would be analogous to queries-per-second for a webserver).

Why aren't the active cores maxed?

It occurred to me that I was running 3 CPU-bound processes on the same machine and that the processes might be stepping on each other's toes. It's possible that if the server is running on core 4 one second and the producer is running there the next, the level 1 cache of that core could be ruined for the server the next time it ambles over there.

The simple solution was to assign cores to the processes. In linux, you do this with command: taskset

Taskset is followed by a bitmask value to assign CPUs - so I ran (in separate xterms):

taskset 0x3F java manyfeedServer
taskset 0x40 java theProducer
taskset 0x80 java theConsumer

and off they went. The server is highly multithreaded so I gave it 6 cores. The producer and consumer each got one.

The result? 270,000 messages/second! Wow. If my cache assumption was right (and I'm not claiming to know if it was) - it REALLY worked. One way or another though - something worked - we got better than a 2x speedup.

So you might be thinking the moral of this story is:

1) If you're in linux, install HTOP
2) If you're sharing a computer amongst CPU-bound processes, isolating the processes might (and very well "might not") be beneficial.

And ok, those are fine ideas. But what bugged me was that htop showed me that no CPUs were maxed yet. Again, what was slowing my application down ahead of CPU power?

I then tried limiting the server to 2 (hyperthreaded) cores. (I also tried keeping the producer and consumer on the same hyperthreaded "core" and given that I had cores to spare, also tried separating them, but the result was the same).

taskset 0x03 java manyfeedServer

Now, we get 530,000 messages/second. Nice. Reducing the cores from 6 to 2 nearly doubles our msgs/sec again. Here's the htop now:

You can see that cores 1 & 2 are plenty busy. Cores 3,4,5,6 are idle as expected. Core 7 (the producer) is pretty busy and so is Core 8 (the consumer).

530k msgs/second is nothing to sneeze at but.. um.. again, no core is maxed. Why - not? What's the bottleneck?

Obviously.. the last test is to throw the whole server on ONE cpu. Apart from the fact that I very purposely and meticulously coded this server be highly multi-threaded, fewer CPUs seem to make it happier. I am a conservative thread-safety-inducer. That is, I'm only really dangerous with firearms and synchronized blocks. But I'm by no means afraid to use the latter when needed.

taskset 0x01 java manyfeedServer

And finally, we hit 100% utilization on CPU 1 at 790,000 messages per second.

Here's the TL;DR of this blog post:

Some multi-threaded java applications apparently run faster on 1 core than on multiple cores.

Note the very non-committal phrasing in an attempt to make this a rather defensible statement. A graph showing the ManyFeed server's performance per number of cores:

So.. if you are a CPU or JVM guru and want to tell me your thoughts on what's going on, I'd love to hear it.

A long time ago, I benchmarked a bunch of different CPU types on doing contended memory stores. To my surprise, a little single-core Dothan processor did amazing in a few tests and I had no idea why. The discussion above is really the same idea. (I've retested that benchmark by the way on the Core I7 - and it has very different characteristics - probably a testament to the I7's new memory architecture).

Note that if you have a pre-I7 Intel multi-core cpu, you can reproduce my "1 core runs faster" results here using the code in that blog post (note: this code *is* a micro-benchmark, built to create a situation that causes extreme and unrealistic threading competition - for more details read that blog post). This gives a clearer picture of what's going on. My theory is that with 2 cores and 2 threads - each gets a core and every store operation competes for the memory bus in order to do its operation. Threads spend lots of time "competing" and less time doing real work. On one core (and 2 threads) - only one thread runs at a time - so every time it tries to store in memory, it can.

The Core-I7 is different (and maybe AMD cpus too) - it doesn't suffer from this memory contention problem at all. (Although - given that my server still runs (way way) better on 1 cpu even on I7, then maybe the contention is elsewhere in my server).

Here's that benchmark running for Static Volatile Store on a Core Duo CPU with 2 cores and again with 1 core. No performance loss on just one core, otherwise same exact run.

(numbers along the bottom are the number of threads in that run - it's not until we have 2 threads that contention creeps in and hurts us).

Oh.. and how does my little nifty manyfeed server do on a real network? Testing on 1 CPU on my local 1Gbps LAN I get about 300,000 messages per second.

With some quick (and hopefully not silly) math, this looks about right.

1Gbps = 1,000,000,000 bits/sec = 125,000,000 bytes/sec

125,000,000 bytes/sec divided by 200 bytes/message = 625,000 messages.

half for producer send and half for the server send (to consumer) = 312,500 each

On a real LAN, the server saturates bandwidth (i.e. becomes IO-bound) and that's no surprise (i.e. until I have access to 10Gbps networks, I don't need to speed up my server any more).

Interestingly I tested this idea of "1-core'ing-it" using apache tomcat and apache bench and saw no improvement. I also saw far fewer qps (loading a single, tiny, web-page over and over) and 100% core utilization even on multi-core. It'd be my guess that tomcat isn't contention-bound but truly cpu-bound.

In other words, don't follow this path without testing this yourself. The good news is that its extremely easy to try and you don't even have to change a line of code. Just try "1-core'ing" it.

Mailinator and (not) Death-by-Popularity

2010-03-01T09:11:00.000-08:00

As far as I know, Mailinator was the first website of its kind. A website to accept any email at all, no sign-up, no registration. Obviously, email websites surely existed before but not with this "no one owns any account" twist. And what a twist it turned out to be. Check out the original web page in 2003 at the Wayback Machine (including the "its like flicking a booger at spam!" original tagline!)

Needless to say, copycat websites showed up fast. And that's a decent indicator that a new idea is a good one. Or at least an interesting one. As I've (and many others have) said before ideas by themselves are basically worth nothing. And any good idea is destined be copied, stolen, and taken credit for. Execution is the key.

If you don't believe me - here is part of Mailinator's terms of service present on every inbox page:

"You agree to hold ManyBrain, Inc./Mailinator harmless from any damages caused by loss of emails, content within emails, damage to your computer or innocence from viewing emails, direct or indirect use of this system, or anything else you can think of. Use at your own risk."

I wrote those words a bunch of years ago. I'm no lawyer so I tried to come up some simple words to get the message across.

Now. Go to google and search on (quoted) "computer or innocence". Here's a screen shot of what I got.

Almost all entries on this page are Mailinator copycat services that not only "borrowed" the idea of Mailinator - a few even "borrowed" its terms of service !! Talk about uncreative copying. :)

For the public record - again, I'm no lawyer so if you unauthorizedly copied my terms of service and get in trouble - you can't blame me.

Now keep in mind here - theft of ideas is all part of the game. It is, for better or worse, part of human nature. And as I outlined here - the execution and evolution of Mailinator was mine, but the idea of no-sign-in email wasn't. It was my old roommate's (who at times, has helped out on Mailinator copy and such).

Google didn't invent the idea of internet search. Facebook didn't start the social web. They just made it better or more usable - or something. But they won. Sometimes taking a great idea and twisting it just a little, turns it into something great.

Mailinator however is an interesting beast. It is a hard business to monetize. And that's just fine, so long of course, it doesn't cost a ton to run.

But therein lies the rub.

There's a strong sentiment in the web industry that performance, at many levels, doesn't matter. That you simply can "buy another server" and solve many performance problems. And that's true. And generally that's a good idea. I mean, wasting a lot of developer time (who could instead be creating new features) on performance optimization is dubious. Especially if you can throw down 2 or 3 thousand dollars and simply buy another machine to solve the problem.

I wrote here that the initial incarnation of Mailinator started to die at around 800,000 emails per day. If you want to make a Mailinator copy - its really not very hard. You can do it with almost all off-the-shelf (and free) software. Sendmail to receive and some sort of webmail front-end to view.

That's it. And that's largely what Mailinator was when it started and it took about a weekend to setup.

But then... I ran into that 800,000 a day problem. This was a performance problem of epic proportions. That is, the site was perpetually crashing. I needed a solution.

One solution, as I pointed out was to "buy another server" but that would have cost the 2 grand plus the ongoing monthly cost to pay for it. And Mailinator was not making enough to cover that at all. I would have had to start paying for Mailinator out-of-pocket to keep it going.

And maybe worse - that would have just solved the problem temporarily until I needed yet another server.

So instead, I did the wrong thing. I rewrote the system from scratch. Threw out the off-the-shelf stuff and built a software system that was customized for Mailinator. Again, financially this was a poor decision (i.e. cost of my development time) but luckily Mailinator is a hobby too so I wrote it off as just that.

As soon as I brought up the new software email jumped to 3,000,000 a day. The old system was not only choking at 800,000 it was refusing connections. I then upgraded the network at the time from 10mbs to 100mbs and the email again leaped to 6,000,000 a day. That is, the new software was fast enough to expose that we were now saturating bandwidth.

(Mailinator has now seen >25,000,000 emails per day)

So that's all fun stuff - but remember those copycat services? Over the years, I've seen 3 of them rise in popularity - and then die (as in literally, site gone, blog explaining it became too expensive to run).

That's what I meant when I said Mailinator is an interesting beast. Its a paradigm that's extremely easy to reproduce - so long as you don't actually get popular (or you find a great way to monetize).

Seems like the web has a lot of opportunity for services like this. Cool ideas that in many senses are just too costly to keep running. Then again, the price of tech comes down - and performance increases.

A Sendmail/Webmail Mailinator today on a modest machine could probably handle a fair bit more than 800,000 a day.

Every few years I try to re-evaluate every old idea I dropped because I felt it wasn't technically/financially feasible. There's a perpetual convergence of faster tech and cheaper prices.

New things become feasible every day that weren't feasible yesterday.

Got any ideas?

Mailinator for Testing

2009-11-11T11:10:00.001-08:00

Recently I implemented a captcha system in Mailinator.

If you're a normal user - you've probably never seen it. That's because it doesn't get activated until emails with the same subject get read more than like 10 times in a minute. Needless to say, if you're using Mailinator to sign-up for something here and there, that's not a normal use case.

But if you're a script - or a person signing up for some website over and over and over (and over) - you hit this pretty fast. It has done a fantastic job of stopping scripts pretty succinctly and slowing down humans zealously working over some site. The main problem with the latter is that many of those sites don't like that and may then ban Mailinator. That's not good for anyone.

What surprised me about the captcha system is how many people emailed me that I broke their test scripts. That is, they had email system tests (i.e. "Thanks for registering!") that use Mailinator as an end point. Their test then (with variation per user of course) checks the Mailinator inbox that their system correctly sent the email.

ManyBrain, Inc. (the company that owns Mailinator) has offered for a long time testing packages that completely bypass Mailinator's abuse system for high-volume testing or other emailing.

I'd be interested in however in improving this service and making it more mainstream. If you use Mailinator for testing - or would like to - I'd like to hear from you.

How would you use it? What kind of volume? I can't believe anyone is excited about scraping HTML (that wasn't particularly designed to be scraped) to get test results. Mailinator already has a (secret, shhh) JSON interface thats not public. That would seem to be the way to go.

My thoughts is a JSON-based SOAP/REST/whatever API that allows testing scripts to read emails. Whats a reasonable method and threshold of use to start charging? Possibly a sister site to Mailinator itself would be a better home for the testing service.

If there's enough interest to support the development, I'd be excited to get this up and running.

Email me at paul@manybrain.com

More Alternate Domains !

2009-09-21T00:45:00.000-07:00

I've added a few more alternate domains. If you've never used them, they are simply other domains (for example, sogetthis.com) that forward all email to Mailinator.

Simply, if you send email to fred@mailinator.com - or - fred@sogetthis.com it works exactly the same way and your email will arrive in the "fred" inbox at Mailinator both ways.

You can find alternate domains (including the new ones mixed in) on the front page of Mailinator on the left column.

I'll be sneaking more into the rotation soon!

A Beautiful Race Condition

2009-06-09T05:53:00.000-07:00

I recently gave a keynote at the ICIS 2009 conference in Shanghai. The topic was why multithreaded programming seemed so easy, yet turned out to be so hard. The fun part was I investigated (per my last post and this one) several old, personal concurrency demons I knew existed but wanted to know more about.

One of those was, indeed, my favorite race condition. It doesn't escape me that its probably wholly unhealthy to even *have* a favorite race condition (akin to having a favorite pimple or something) - but nonetheless, the elegance of this one still makes my heart aflutter.

The scenario of this race is that we assume, not terribly inaccurately, that race conditions at times, can cause corrupted data. However, what if we have a situation where we sort of don't mind some corrupted data? A "good enough" application as it were.

The dangerous part of all this is if we assume (without digging in) what kind of data corruption can happen. As you'll see, you might just not get the type of data corruption you were hoping for (which is one of the sillier sentences I've ever written).

The particular instance of this kind of happy racing I've encountered is where someone uses a java.util.HashMap as a cache. I've never done such a thing myself, but I heard about this race and thus this analysis. They may use it with a linked-list or maybe just raw, but the baseline is that they figure a synchronized HashMap will be expensive - and in their case, a race condition inside the HashMap will just lose (or double up on) an entry now and then.

That is - a race condition between two (or more) threads might accidentally drop an entry causing an extra cache miss - no biggie. Or, it may cause one thread to re-cache an entry that didn't need it. Also no biggie. In other words, a slightly imprecise, yet very fast cache is ok by them. (of course, this assumption is dead wrong - don't do that - read on for why!)

So they setup a HashMap in some global manner, and allow any number of nefarious threads bang away on it. Let them put and get to their hearts content.

Now if you happen to know how HashMap works, if the size of the map exceeds a given threshold, it will act to resize the map. It does that by creating a new bucket array of twice the previous size, and then putting every old element into that new bucket array.

Here's the core of the loop that does that resize:

1:  // Transfer method in java.util.HashMap -
2:  // called to resize the hashmap
3:
4:  for (int j = 0; j < src.length; j++) {
5:    Entry e = src[j];
6:    if (e != null) {
7:      src[j] = null;
8:      do {
9:      Entry next = e.next;
10:     int i = indexFor(e.hash, newCapacity);
11:     e.next = newTable[i];
12:     newTable[i] = e;
13:     e = next;
14:   } while (e != null);
15:   }
16: }

Simply, after line 9, variable e points to a node that is about to be put into the new (double-wide) bucket array. Variable
next holds a reference to the next node in the existing table (because in line 11, we'll destroy that relation).

The goal is that nodes in the new table get scattered around a bit. There's no care to keep any ordering within a bucket (nor should there be). HashMap's don't care about ordering, they care about constant time access.

Graphically, let's say we start with the HashMap below. This one only has 2 buckets (the default of java.util.HashMap is 16) which will suffice for explanatory purposes (and save room).

As our loop starts, we assign e and next to A and B, respectively. The A node is about to be moved, the B node is next.

We have created a double-sized bucket array (in this case size=4) and migrate node A in iteration 1.

Iteration 2 moves node B and Iteration 3 moves node C. Note that next=null is the ending condition of our while loop for migrating any given bucket (read that again, its important to the end of the story).

Also important to the story, note that the migration inverted the order of Node's A and B. This was incidental to the smart idea of inserting new nodes at the top of the list instead of traversing to find the end each time and plunking them there. A normal put operation would still have to check that its inserting (and not replacing) but given a resize can't replace, this saves us a lot of "find the end" traversals.

Finally, after iteration 3, our new HashMap looks like this:

Our resize accomplished precisely the mission it set out to. It took our 3-deep bucket and morphed it into a 2-deep and 1-deep one.

Now, that's all well and good, but this article isn't about HashMap resizing (exactly), its about a race condition.

So, let's assume that in our original happy HashMap (the one above with just 2 buckets) we have two threads. And both of those threads enter the map for some operation. And both of those threads simultaneously realize the map needs a resize. So, simultaneously they both go try to do that.

As an aside, the fact that this HashMap is unsynchronized opens it up to a scary array of unimaginable visibility issues but that's another story. I'm sure that using an unsynchronized HashMap in this fashion can wrack evil in ways unlike man has ever seen, I'm just addressing one possible race in one possible scenario.

Ok.. back to the story.

So two threads, which we'll cleverly name Thread1 and Thread2 are off to do a resize. Let's say Thread1 beats Thread2 by a moment. And let's say Thread1 (by the way, the fun part about analyzing race conditions is that nearly anything can happen - so you can say "Let's say" all darn day long and you'll probably be right!) gets to line 10 and stops. Thats right, after executing line 9, Thread1 gets kicked out of the (proverbial) CPU.

1:  // Transfer method in java.util.HashMap -
2:  // called to resize the hashmap
3:
4:  for (int j = 0; j < src.length; j++) {
5:    Entry e = src[j];
6:    if (e != null) {
7:      src[j] = null;
8:      do {
9:      Entry next = e.next;
     // Thread1 STOPS RIGHT HERE
10:     int i = indexFor(e.hash, newCapacity);
11:     e.next = newTable[i];
12:     newTable[i] = e;
13:     e = next;
14:   } while (e != null);
15:   }
16: }

Since it passed line 9, Thread1 did get to set its e and next variables. The situation looks like this (I've renamed e and next to e1 and next1 to keep them straight between the two threads as both threads have their own e and next).

Again, Thread1 didn't actually get to move any nodes (by this time in the code, it did allocate a new bucket array).

What happens next? Thread2, that's what. Luckily, what Thread2 does is simple - let's say it runs through the full resize. All the way. It completes.

We get this:

Note that e1 and next1 still point to the same nodes. But those nodes got shuffled around. And most importantly the next relation got reversed.

That is, when Thread1 started, it had node A with its next as node B. Now, its the opposite, node B has its next as node A.

Sadly (and paramount to the plot of this story) is that Thread1 doesn't know that. If you're thinking that the invertedness of Thread1's e1 and next1 are important, you're right.

Here's Thread1's next few iterations. We start with Thread2's bucket picture because thats really the correct "next" relations for our nodes now.

Everything sort of looking ok.. except for our e and next at this point. The next iteration will plunk A into the front of the bucket 3 list (it is after all, next). And will assign its next to whatever happens to already be there - that is, node B.

Woah. Thar she blows.

So right about now Thread1 goes into, what we like to call in the biz, an "infinite loop". Any subsequent get operations that hit this bucket start searching down the list and, go into, yep - an infinite loop. Any put operation that first needs to scan the nodes to see if its going to do a replace, will, you guessed it, go into an infinite loop. Basically, this map is a pit of infinite loopeness.

If you remember we noted that race conditions cause data corruption. Well, yeah, thats what we have here - just very unlucky data corruption on pointer structures. I'm the first to admit this stuff is tricky - if you found errors in my analysis I'll happily fix this post.

Now I had the happy fortune for a time of sharing an office with Josh Bloch who wrote java.util.HashMap. When I innocently mentioned he had a bug in his code given this behavior, he did indeed (to use Josh's word's) go non-linear on me.

And he was right. This is not a bug. HashMap is built specifically for its purpose and this implementation is not intended as threadsafe. There's a gaggle of ways to make it threadsafe, but in plain, vanilla, (and very fast) form - its not. And needless to say, you shouldn't be using it that way.

Race conditions are nothing to mess with and the worst ones are the ones that don't crash your program but let it wander down some unintended path. Synchronization isn't just for fun you know.

And nefarious uses of HashMap aside, I still attest - this is, indeed, a beautiful race.

Addendum: I've been yelled at a few times for calling any race condition "beautiful". I'll defend myself by our apparently human nature to generally call intricate complexity beautiful (i.e. waves crashing on a shore, nature in general).

Most races end up being about data-corruption. This one is data-corruption that manifests as control-flow-corruption. And it does so fantastically without an error (infinite loops notwithstanding).

As the evolution analogy goes, if you drive a needle into a pocket watch - chances are you'll simply destroy it. But there's a tiny chance you'll actually make it a better watch (clearly, not the case here). And another tiny chance you'll simply make it something "different" - but still, per se, functioning.

Again, my use of "beautiful" might be more appropriate as "a complex mutation with surprising non-destruction" :)

On Java Visibility

2009-06-08T08:00:00.000-07:00

A semi-famous example of broken Java synchronization is this:

class SomeClass {

  private boolean keepGoing = true;

    public boolean get() {
      return keepGoing;
    }

    public synchronized boolean set(boolean x) {
      keepGoing = x;
    }
}

I believe this example is in Josh Bloch's book "Effective Java" (which I now notice I somehow lost my copy in my most recent move - don't tell Josh).

The idea is here that someone (ostensibly) thought that they'd save some performance by not synchronizing a getter and just synchronizing the setter. There definitely is a cost to synchronization and needless to say, getting doesn't change anything - so why bother paying that cost for gets?

As has been pointed out long before this post, without a synchronize on a getter, there is no guarantee of visibility when doing the get. That is, if one thread calls set(false), there's no guarantee that any other thread will know it happened.

Consider code that might use the code above:

class SomeOtherClass {

  SomeClass keepGoing = new SomeClass();

  class Thread1 implements Runnable {
    public void run() {
      while (keepGoing.get()) x++;
      System.out.println("done1");
    }
  }

class Thread2 implements Runnable {
    public void run() {
    keepGoing.set(false);
  }
  }
}

Let's say you start a Thread1 running. And of course, keepGoing.get() is true, so it just keeps looping along. Then lets say an eternity (or maybe 2 seconds) later you start Thread2.

If we had reliable visibility, Thread1 would end the moment after Thread2 sets keepGoing to false.

If you've read Josh's book, its no surprise that it doesn't. Specifically, Thread1 doesn't end. It keeps going. Thread2 ends happily and Thread1 never ends.

The only interesting part to me was that this always worked. Always. Adding to the complexity of visibility concerns is that memory is "leaky". That is, even without guaranteed visibility you often get unreliable visiblity.

So, I dug a little deeper. If you're adventurous enough to grab a debug-version of the JDK and run it with the -XX:+PrintOptoAssembly option. You get to see the optimizations the JIT are making. That is, you see the assembly code version of your Java code - post-optimization. Check out Koshuke Kawaguchi's Blog for some instructions.

So here's Thread1's loop code after JIT optimization at runtime:

02c movq R10, precise klass manybrain/test/Main: 0x0000000040a50968:Constant:exact * # ptr
036 movsbl R8, [R10 + #596 (32-bit)] # byte ! Field manybrain/test/Main.keepGoing
03e testl R8, R8
041 je,s B4 P=0.000000 C=147944.000000
041
043 B2: # B3 <- B1 Freq: 1
043 movl R8, [R10 + #592 (32-bit)] # int ! Field manybrain/test/Main.x
043
04a B3: # B3 <- B2 B3 top-of-loop Freq: 1e-35
04a incl R8 # int
04d movl [R10 + #592 (32-bit)], R8 # int ! Field manybrain/test/Main.x
054 testl rax, [rip + #offset_to_poll_page] # Safepoint: poll for GC # manybrain.test.Main$Thread1::run @ bci:14 L[0]=_
# OopMap{r10=Oop off=84}
05a jmp,s B3
05a
05c B4: # N53 <- B1 Freq: 4.76837e-07
05c movl RSI, #27 # int
061 nop # 2 bytes pad for loops and calls

If you're old school, welcome home.

If you're not, then this might look like a lot of goop. So let's parse out just the interesting parts.

Line 41 is an conditional jump. Basically, if keepGoing (per the comment in line 36) is false, we jump to line 5C (label B4) and end the code segment. You and I know that keepGoing started true, so basically that jump isn't followed.

Lines 43-5a are the loop that does x++.

And checkout line 5a. That is an unconditional jump back to the top of the loop. So what does all this mean? That the JIT did some very aggressive optimization. In fact, remember our original loop from Thread1?

while (keepGoing.get()) x++

The JIT has optimized this to:

if (keepGoing.get()) {
  while (true) x++;
}

No wonder the loop never ends. You've got bigger problems now than a little leaky visibility issue. I'm positive I'm oversimplifying, but effectively the JIT saw your get method wasn't synchronized and made the assumption that it could optimize as such. If you didn't guarantee visibility, it didn't need to either. Obviously, add the synchronization modifier to the get method and all this badness won't happen.

Moral of the story is much like the inevitable topics discussed at a lunch with Jeremy Manson - you can't optimize away correct synchronization; as all you'll probably do is optimize the "correctness" part away.

How long before an email is removed from the system?

2009-01-26T10:38:00.000-08:00

I get this question a fair bit. How long, exactly, is an email available to be read after it enters the system?

Its hard to answer because its literally dependent upon the incoming rate at which email arrives. In effect, new email "pushes out" old email. That's a bit simplified as some email is thrown away based upon spam rules (if you get 100,000 emails with the same subject, you can probably say they aren't all going to be read).

So, whats the average? This weekend, email was lasting around 10 hours before getting "pushed out" of memory. I've seen that jump down to an hour - but mostly its around 5-7 hours.

I'll note that the primary Mailinator use case is that people tend to read email soon after it arrives. In other words, if email only lasted 15 minutes, we'd actually fill the needs of many users.

(I'd be *very* interested in hearing ways you use Mailinator that need mail to stick around longer.)

No, Mailinator didn't spam you

2009-01-08T22:04:00.000-08:00

I must say, it's rare these days but I still now and then get emails from folks that think Mailinator spammed them. As it says on our contact page, this is pretty unlikely. I say that because Mailinator is custom software and that software contains no specific way to actually have a user "send" email. There's no chance of Mailinator being an open-relay as it stands. And of course, there is no place at all on the site to accept an email for sending.

I suppose its possible, but it would involve a hacker breaking into the server, installing some other email server (which would likely conflict with mailinator itself), configuring it, and then start pummeling it for their nefarious purposes. Given that any self-respecting spammer has a billion zombies at their disposal and that this would definitely be discovered very quickly (my colo vendor loves to watch my bandwidth), it doesn't seem like an efficient way to spam.

In any case, I still get accused of spamming at times. And all that accusation takes is for a spammer to forge the return address as a mailinator address. Let me tell you, forging a return address is stunningly easy. Here's 3 million or so guides how to do it if you're wondering.

Below is an actual email header someone sent me.

The interesting parts are really the first two lines. As you see the forged return path is ronb@mailinator.com. Now if you know mailinator, you know ANYONE can check that box. It belongs to no one and everyone (as outlined in the FAQ - Mailinator guarantees NO PRIVACY. All emails are viewable by ANYONE).

The 2nd line (i.e. Received:) shows the IP (and dns) of the server that actually sent the email. Something at abac.net. That looks like a hosting company somewhere. One thing I can tell you though is that that server has zero to do with mailinator. The spam email never ever touched the mailinator server. So even if I devoted my life to stopping this email, there's nothing I could do.

Return-Path: ronb@mailinator.com
Received: from 216.55.169.94 (216-55-169-94.dedicated.abac.net
216.55.169.94)
by smtpin4.mail.de.uu.net (8.14.1/8.14.1) with SMTP id n083RPV6001157;
Thu, 8 Jan 2009 03:27:26 GMT
Message-Id: 200901080327.n083RPV6001157@smtpin4.mail.de.uu.net
From: "RON" ronb@mailinator.com
Reply-To: "RON" ronb@mailinator.com
To: xxxxxxxxxxxxxx --> edited

This is sort of similar to a phishing attack. Someone gets an email from their bank, then goes to the phish site, then loses all their money. In truth their bank had nothing to do whatsoever with any of that but the bank still gets blamed.

The saddest part for me is that even after I respond to people showing them the real culprit, its not uncommon for them to stay mad at me. I suppose its because they then don't know who they're going to yell at now and I'm still available for the job.

Mailinator is about letting you protect your real email address. It might be to prevent spam but at times it might even be to receive spammy email they really want (just not at their primary address).

Regardless what you use it for, it won't email you. It just doesn't do that.

Plenty of people threaten to blacklist Mailinator from ever sending them email again. Yes, please do! As I've said in the past, feel free to put mailinator.com on the tippy-tippy-top of all your spam blacklists. Mailinator doesn't send any email at all - so you can be sure any email that looks like it came from a mailinator address is forged. And I'll sleep just fine if such email gets blacklisted.

Dear World, email addresses are not identity

2008-12-08T15:34:00.000-08:00

Its no secret that part of putting up any website or service is the consideration of security measures to stop people from abusing the system. And as you can imagine, those particular issues are probably doubled or tripled with a site like Mailinator (and of course Talkinator). Needless to say, I've gotten pretty good at unorthodox security.

Anonymity does indeed breed bravery.

The normal Mailinator use case is that you have a need for a quick email address to sign-up for some web service. Some people however, use Mailinator as an actual primary email repository. With the RSS feed, it acts as a convenient dropbox for newsletters or other semi-private email needs.

Unfortunately, there are some wacky funsters out there that think it might be fun to sign-up for some website over and over and over and over (and over). Often this is for some site that has "vote for your favorite band" or "sign up for a free gift" or some such thing.

The primary flaws of these systems is not Mailinator - its that these websites equate the idea of identity with email addresses. Seriously, long before Mailinator existed I had 3 or 4 email accounts that I actively used and another 10 I had probably abandoned. I'm not sure what those site designers were thinking - Mailinator or not, email addresses are FREE.

If you give someone something that has any value at all in exchange for free email addresses, they're going to ask for lots. I'm probably in a unique position to view this, but I see this idea as incredibly broken.

One option I had was to simply ignore this idea. Let crafty script-writers create systems that sign up for Wink accounts 1000 times an hour. (As I write this, I'm watching some automated system using several hundred IPs trying to sign-up for wink.com using Mailinator - and watching the Mailinator system dutifully send each request to the abuse page).

The problem with this is that someday, wink.com will catch on. And they'll ban Mailinator. This is sadly, a wonderfully broken solution to a still existing broken site design.

The problem for me is that I likely have legitimate users that want to sign up for Wink - and I want them able to do so (and I imagine Wink might want more users too, so by extension they'll lose some or all of the ones I lost). What's insanely broken for sites banning Mailinator is that there are tens of Mailinator-copy-cat disposable web services out there. Or even worse, someone with access to a server and a domain, who can install sendmail and create a few thousand accounts. Simply put, banning mailinator is like catching a single mouse and thinking you've solved the mouse problem.

You stop the bad guys, but for about a day until they implement a new system.

I had an interesting discussion with an acquaintance recently. During the conversation I described Mailinator to him. His mouth gaped open and told me he would look into it and probably ban it from his site. I asked what he would be banning it "from". He said he had a trial piece of software that people could sign-up for and download. And he wanted their real information to email them later (i.e. I did my best not to say that he was sending "spam") to see if they wanted to buy.

I noted that sometimes when I download software to try, I do want to enter my real email. I'm interested enough to want to be registered. But other times, I'm just in browsing mode. If given the chance I'd download and check it out, but if you give me too much impedance I'll probably just go check out his competitor.

In those cases when I'm just browsing, I'll use mailinator.

In other words, there are 3 types of potential customers. Those that don't care about his software. Those who really love the idea of trying his software and will do anything to do so. And those who are on the fence.

For obvious reasons, Mailinator is my "on-the-fence" tool of choice. If he banned it, he'd be refusing some subset of those potential customers. So it basically comes down to the question - whats better?

1) Definitely get user information you can spam later - or
2) get your product in front of as many eyeballs as possible.

Also noting the fact that NO email insures any relation to an actual person whatsoever (including yahoo, gmail, hotmail, etc.) - whats the point?

We continued our discussion and agreed that from a marketing perspective, you actually don't want to remove the email sign-up altogether. It actually brings value to some customers. If you remove it or make it optional, most everyone will skip it just to get to the goodies. But by leaving it and knowing that some people, using Mailinator or Yahoo or whatever, will give you temporary email addresses, you're maximizing your potential customer base.

It didn't hurt my argument to mention a few other disposable email services that he'd have to ban too. I sure don't know them all - they seem to come and go a lot. And that surely doesn't count ones that run semi-privately. Basically, it would be a fulltime job to keep up.

Oh. So, back to our script kiddies above. Mailinator includes a system to stop scripts from signing up for websites over and over. I love fun algorithms/data-structures so your homework can be to design something like Mailinator's abuse trigger system - a key-value datastruct that ages with time and is refreshed by lookups that come in at some notable (and tweaked) rate (in the same ballpark as a LRU cache, but definitely more dynamic).

Its unlikely a human will set-off the triggers but its possible. The sad part (for script writers) is that the algorithm doesn't trigger until their script gets going, so its probably a bit heart-breaking to spend a few hours perfecting a script to scrape Mailinator and then have Mailinator detect it only once it gets going and shut it down hard.

The first level is the Abuse Page. If you push it, Mailinator will ban IP addresses - but only under certain conditions. That's rather an imprecise way of stopping abuse. In addition, it looks for patterns of mailbox usage regardless of IP. An obvious one is that if one subject "Welcome to Wink!" shows up a lot in the read emails. Sadly, its difficult to distinguish valid users trying to sign-up for wink amongst the botnet hitting right now - so they'll probably get the abuse page too for the time being.

Potential site abusers taught me a lot and hardened the site considerably. Abuse attempts are still a common occurrence but far less normal than a few years ago. I assume many scripters went to less caring disposable services.

I often get asked if I care if sites ban Mailinator. I don't really. In some cases its prudent if you really do need to email people that use your service. In most cases however, its simply a knee-jerk reaction attempting to patch an otherwise flawed system. Not only is it a sure way to eliminate some potential customers, the flaw will show up again soon when the abusers shift to another method - and probably another method without Mailinator's facilities to stop scripts.

In the end, there is no real identity on the Internet. At least none past an IP address and a subpoena. At best, email is optional identity. And prudently, it should probably be treated that way.

Reserved Names in Talkinator

2008-12-04T10:04:00.000-08:00

Talkinator now includes a beta feature (ok, all features are beta) to allow reserved names. Each room can have only one and the person owning the site (and cut and pasting the code) designates the reserved name and the password to access it.

If you already have a Talkinator in your site, just click the </> icon (i.e. get the code!) in the upper right of any talkinator chatbox and create your Talkinator with your reserved name!

Please let us know how it works and especially if you find any bugs! (email to support@manybrain.com)

So, Mailinator's core-duo server now gets 3 Terabyte a month

2008-11-11T15:55:00.000-08:00

Its been awhile since I've updated my article The Architecture of Mailinator. That was a long time ago and really and tons has changed. To save you reading that article if you're in a hurry, the crux was that Mailinator was originally based on the unix email program sendmail, which pretty much died at around 800,000 emails a day. And then it discusses the design of the fully custom SMTP server designed for mailinator.

In that article, Mailinator was running on a single AMD cpu and processing 3 million emails a day or so. Well, somewhere in there I got a deal with my hosting company and was running a dual dual-core opteron machine. Mailinator ran on that for about a year as our email load doubled a time or two.

Eventually I needed a server for another project and realized that giving Mailinator that dual-opteron was pretty overkill. So I "downgraded" the machine to a core-duo. Which is where it runs now. In the aforementioned article we were getting 3 million emails a day - now I sometimes see 2 million an hour.

For the geeks out there, this is still of course a highly-multithreaded java implementation often running 4000 simultaneous threads. Pure Java I/O (not that stinky and slow NIO.

Today I got an email from my hosting server that I "surpassed my allotted bandwidth". I'm pretty used to getting subpoena's and such, but not so many of these.

Apparently, Mailinator received 3.1 Terabytes of email in the last 4 weeks.

Thats (ballpark) 480 million emails or so in that 4 weeks. As I've said before, that doesn't compare to any bigger email service, but then again, I bet they have more than one server :) I'll also emphasize that numbers I usually throw up here were measured or calculated by me - that 3T number comes from my provider so I hope its accurate as they want money for it !

Thats a LOT of spam (sure there's web traffic in there which might be hefty in other circumstances, but it doesn't hold a candle against the mail traffic).

I've priced things out and although my server still has plenty of computing capacity left, I'm clearly hitting an artificial (i.e. pay for more) bandwidth limit. It makes far more economical sense to simply get another server (per all pricing schemes I find) than to buy more bandwidth on this one.

So, someday I'll get around to writing the next "Architecture of Mailinator" article and although it almost sounds funny to me after all this time, I'll likely be talking about "both" Mailinator servers. Not just our trust little core duo.