Mitchell Hashimoto

Comparing Filesystem Performance in Virtual Machines

2014-01-10T08:51:52-08:00

For years, the primary bottleneck for virtual machine based development environments with Vagrant has been filesystem performance. CPU differences are minimal and barely noticeable, and RAM only becomes an issue when many virtual machines are active[1]. I spent the better part of yesterday benchmarking and analyzing common filesystem mechanisms, and now share those results here with you.

I’ll begin with an analysis of the results, because that is what is most interesting to most people. The exact method of testing, the software used, and the raw data from my results can be found below this analysis.

In every chart shown below, we test reading or writing a file in some way. The total size of the file being written is fixed for each graph. The Y axis is the throughput in KB/s. The X axis is “record size” or the size of the chunks of data that are being read/written at one time, in KB.

The different test environments are native, VirtualBox native, VMware native, VirtualBox shared folders (vboxsf), VMware shared folders (vmhgfs), and NFS. The “native” environments refer to using the filesystem on its own in that environment. “native” is on the host machine, “VirtualBox native” is on the root device within the VirtualBox virtual machine, etc. NFS was only tested on VirtualBox because the performance characteristics should be similar in both VirtualBox and VMware.

For every chart, higher throughput (Y axis) is better.

Small File Sequential Read #

First up is a sequential read of a 64 KB file, tested over various read record sizes. A real world application of a small sequential read would be loading the source files of an application to run, compile, or test.

The first thing you can’t help but notice is that the read performance of NFS is incredible for small record sizes. NFS is likely doing some heavy read-ahead operations and caching to get this kind of performance. I don’t have a good theory on how NFS outperforms the native virtual filesystems.

VMware shared folders just crush VirtualBox shared folders here. VirtualBox shared folder read performance is just abysmal. If you look at the raw data, you’ll see the throughput actually never goes above 100 MB/s, while VMware never goes below 500 MB/s, and peaks at over 900 MB/s.

It is interesting that sometimes the native filesystem within the virtual machine outperforms the native filesystem on the host machine. This test uses raw read system calls with zero user-space buffering. It is very likely that the hypervisors do buffering for reads from their virtual machines, so they’re seeing better performance from not context switching to the native kernel as much. This theory is further supported by looking at the raw result data for fread benchmarks. In those tests, the native filesystem beats the virtual filesystems every time.

Large File Random Read #

This tested the throughput of randomly reading various parts of a 64 MB file, again tested with various read record sizes. This file is 1000x larger than in our previous test. This sort of behavior might be seen when dealing with database reads that hit the filesystem.

The gap between VMware shared folders and VirtualBox shared folders is widened considerably compared to small sequential reads. VirtualBox performs so badly you can barely see it. Again, VirtualBox throughput never peaks above 100 MB/s. VMware, on the other hand, is peaking at 7 GB/s. Because the deviation of the VirtualBox throughput is so small across various test cases, I theorize that there is a single hot path of code in the VirtualBox shared folder system limiting this. They’re clearly doing something wrong.

NFS is less dominating, most likely because the read-ahead benefits are not seen in this test case. Still, NFS performs very well versus other options.

And, just as with the small sequential read, we’re still seeing better performance within the virtual machine versus outside. Again, this can be attributed to the hypervisor doing clever buffering, whereas the raw syscalls to the host machine don’t allow this to happen.

Small File Sequential Write #

Let’s look at the performance of sequential writes of small files. This most accurately describes storing session state, temporary files, or writing new source files.

The first noticeable thing is that NFS performance is terrible for these kinds of writes. There is no real caching that NFS can do here, so you must pay the full penalty for network overhead, then writing to disk on the host side, and finally waiting for an ack again the VM that the write succeeded. Ouch.

The various “native” filesystems perform very well. Again, the virtual machines outperform the host. And again, this can be attributed to buffering in the hypervisors.

The shared filesystems are close, but VirtualBox clearly outperforms VMware in this test case.

Large File Random Write #

The final chart we’ll look at are the results of testing random writes to a large (64 MB) file. Just like our large random read test, this is a good test for how a database would perform.

There really isn’t much of a difference here versus small sequential writes. Because it is a large file, the gaps between the different test environments is larger, but other than that, the results are mostly the same.

NFS continues to be terrible at writes. VirtualBox shared folders continue to outperform VMware on writes, and the hypervisors outperform the host machine.

Hypervisors outperforming the host machine is the most interesting to me. The results of this test clearly show that the hypervisors must be lying about synced writes for performance. This corroborates what I’ve seen with Packer as well, where if the virtual machine is not cleanly shut down, committed writes are lost. fsync() in a virtual machine does not mean that the data was written on the host, only that is is committed within the hypervisor.

Overall Analysis #

With regards to shared filesystems, VMware has the behavior you want. Loading web pages, running test suites, and compiling software are all very read heavy. VMware shared folder read performance demolishes VirtualBox, while the write performance of VirtualBox shared folders is only marginally better than VMware.

If you have the option to use NFS, use it. Again, the read performance is far more valuable than the write performance.

Hypervisor read/write performance is fantastic (because they cheat). Thanks to this data, I’m definitely going to be focusing more on new synced folder implementations in Vagrant that use only the native filesystems (such as rsync, or using the host machine as an NFS client instead of a server).

More immediately applicable, however: if you use virtual machines for development, move database files out of the shared filesystems, if possible. You’ll likely see huge performance benefits.

Finally, I don’t think there are any huge surprises in these results. Vagrant has supported NFS synced folders since 2010 because we realized early on that performance in the shared folders was bad. But having some hard data to show different behaviors is nice to have, and provides some interesting insight into what each system might be doing.

Test Software, Configuration, and Raw Data #

The host machine is a 2012 Retina MacBook Pro with a 256 GB SSD running Mac OS X 10.9.1. Tests here were labelled native.

VirtualBox is version 4.3.4 running a virtual machine with Ubuntu 12.04 with the VirtualBox guest additions installed and enabled. The root device under test in VirtualBox is formatted with an ext3 filesystem. Tests on the root device were labelled VirtualBox native while tests on the shared folder system (vboxsf) were labelled VirtualBox shared folders (vboxsf).

VMware Fusion is version 6.0.2 running a virtual machine with Ubuntu 12.04 with VMware Tools 5.0 installed and enabled. The root device under test in VMware is formatted with an ext3 filesystem. Tests on the root device were labelled VMware native while tests on the shared folder system (vmhgfs) were labelled VMware shared folders (vmhgfs).

The NFS server used is the one built and shipped with OS X 10.9.1. The NFS client is what comes with the nfs-common package in Ubuntu 12.04. NFS protocol 3 was used over UDP. Tests using NFS were labelled NFS in VirtualBox.

The benchmarking software used was the Iozone Filesystem Benchmark compiled from source for 64-bit Linux. The same binary was used in every instance except for the native test, which used a 32-bit OS X binary of Iozone compiled from source. The flags passed to Iozone were -Racb.

Unless otherwise noted, the remaining settings not mentioned here were defaults or not touched.

The raw data for the tests can be found in this Excel workbook. There are also more graphs for this data in this imgur album.

Footnotes:

[1]: This is in the case of a standard web development environment. They are typically neither CPU bound nor RAM-intensive.

Packer

2013-06-28T09:08:33-07:00

For the past two months, I’ve been heads down working on something new. There is still a lot of work left, but I believe it’s time to show everyone what I’ve been up to. Today, I’m announcing Packer: an open source tool for creating machine images for multiple platforms.

Packer is an open source tool for creating machine images, such as AMIs, VirtualBox images, Vagrant boxes, etc. Packer uses a single portable input format – a template – to generate the images for multiple platforms in parallel, so you can create identical AMIs, VMware machines, etc. all at once. This unlocks a lot of interesting use cases.

Machine images are a powerful concept, but the existing tooling around them hasn’t caught up to the agile workflow that modern cloud infrastructures enable. Packer makes machine image creation automated, fast, and integrates with all the industry standard DevOps tools. Here are some examples of what Packer can be used for:

Automatically create AMIs (or any other machine image) for your production usage, perhaps as part of a continuous delivery pipeline.
Concurrently create AMIs for production and Vagrant boxes for development which are nearly identical.
Build software appliances in an automated, repeatable way, perhaps for multiple platforms. Getting software demos or ready-to-go images in the hands of anyone on any platform has never been easier.

And so much more! Packer is a general purpose image building tool. The opportunities to build things around it are endless. I’m excited to see what people will use Packer for.

With the announcement today, I’m inviting anyone and everyone to play with Packer and to get involved with the open source project. There are so many ways to get involved. The most fun might be adding more builders, the components that build images for specific platforms. There are already a handful of committers on the project and I’m looking to add more.

As a reminder: I released version 0.1 of Packer today. It is the first release and a lot of improvements will be coming quickly in the future, so please keep that in mind when offering feedback or judging the project. Remember, when I released version 0.1 of Vagrant, it only worked on Mac, only worked with VirtualBox, only ran Ubuntu machines, and only worked with serverless Chef. However, as the Vagrant community grew so did its capabilities, and now all those limitations have been removed. If there is a feature you’d love to see in Packer feel free to hack on it or ask the mailing list.

The Tao of Vagrant

2013-06-18T11:06:42-07:00

Before even installing Vagrant or seeing how it works, it is important to understand the high-level workflow of Vagrant in an actual working environment. These principles are collectively known as the “Tao of Vagrant.” The following is an excerpt from Vagrant: Up and Running.

In a world with Vagrant, developers can check out any repository from version control, run vagrant up, and have a fully running development environment without any human interaction. Developers continue to work on their own machine, in the comfort of their own editors, browsers, and other tools. The existence of Vagrant is transparent and unimportant in the mind of the developer. Vagrant is the workhorse that creates consistent and stable development environments.

System operations engineers work on systems automation scripts, again in their own editors and tools on their machine. When they’re ready to test these scripts, they vagrant up, and have a complete sandbox matching production, ready to test real world scenarios and systems automation.

The automation system operations engineers develop is not only used in production, but also in development. With every vagrant up, developers are getting fully provisioned development environments using the same scripts that are used to setup production. This way, developers work in environments that mimic production as closely as possible.

If something goes wrong, or they just want to start over from a clean slate, developers and operations engineers can run +vagrant destroy+, which removes all traces of that development environment from their machine. Then a vagrant up again will re-create an identical, fully functioning development environment in a snap.

At the end of the day, Vagrant can suspend, halt, or destroy the development environment, keeping the overall system clean. Never again can developers forget to shut down a stray server process and waste precious compute resources. When ready, vagrant up will bring back a ready-to-go development environment in just a few minutes.

The best part of all this is that this knowledge transfers to every project. Whether working on project A, project B, or even at company A or company B, as long as they follow the Tao of Vagrant, the workflow is exactly the same. As a result, productivity abounds and “works on my machine” bugs disappear.

Of course, it isn’t necessary to follow each and every principle of the Tao of Vagrant to use Vagrant. Vagrant is a general purpose tool and can be molded into your environment as you see fit. But it is important to see and understand the larger vision behind Vagrant.

Automation Obsessed

2013-06-06T09:39:49-07:00

Someone asked me recently to share some previous work I had done. As I looked back at my personal history, and reflected on what I’ve done successfully, it was clear to me that I’m definitely working on things I’m meant to be working on. Automation has been my passion since the very beginning.

I started programming because I wanted to cheat Neopets. I played Neopets daily, and I was frustrated that I would do the same things daily for hours for the same measly amount of Neopoints (the virtual currency used by Neopets). I learned Visual Basic with the goal of writing a program to do these tasks for me. I succeeded at this, and was soon making my measly amount of Neopoints without any human interaction.

I found more things to automate, realizing that computers can do tasks much more efficiently than humans, thus unlocking opportunities that didn’t exist before. I was soon able to generate millions of Neopoints every hour from arbitraging Neopian markets. On average, someone making a few thousand an hour would be impressive, so I was doing quite well, virtually. At this point, I wasn’t particularly interested in Neopets anymore. The automation had taken hold within me, and I was just seeing how far I could push it.

After becoming bored of Neopets, my interest turned to seeing if I could automate things to make real money. I began writing Windows applications that faked ad clicks, played online games that paid real money, etc. This proved to be very lucrative, though the legal notices my parents received quickly ended this phase of my life. Despite the negative outcome, I loved making computers do things for me that were previously manual tasks.

As I began college, I noticed that the poor technical design of the registration system made it incredibly difficult to get the set of classes I wanted. I developed automated registration software that would detect open slots in the full courses, and notify me via text message. While my friends were spending hours every day refreshing course schedules hoping to get into a full class, I was just waiting for a text message. And I always got into the classes, because a human refreshing a browser can’t beat my software that was checking thousands of times per hour. Automation wins again.

Then, in my third year of college, I built Vagrant. The idea that I could automate the actual creation and control of virtual machines struck me in a way I can’t describe. Once the idea hit, I was obsessed. I had to see what I could do, what boundaries I could push, what opportunities existed. The success of Vagrant is proof that automation in this space vastly improved the working lives of many people.

After college, I spent two years working at Kiip. At Kiip, I automated their entire infrastructure. At one point, I accidentally locked myself out of our entire production cluster (whoops, never made that mistake again). I rebuilt the entire production cluster from scratch in less than 30 minutes, and switched to the new cluster with no perceivable downtime. The production cluster was over 50 machines. My obsession with automation reached a point where if I had to do anything twice, I would write a program to do it for me.

Recently, I started HashiCorp. And while it isn’t publicly clear what I’m up to yet, and I’m not quite ready to talk about everything I’m up to, you can be sure of one thing: I’m busy automating. More specifically, I’m developing software to automate everything, and giving it to you.

This should come as no surprise. Automation defines who I am, and always has.

Abandoning RubyGems

2013-03-21T09:23:33-07:00

Vagrant 1.1+ no longer supports RubyGems as an installation method. Instead, you must install Vagrant 1.1+ using pre-made packages or installers. For folks used to the gem-based installation, this has caused a mixture of confusion and disdain. In this post, I enumerate my reasons for abandoning RubyGems, and why it is better for the Vagrant community long-term.

I initially released Vagrant as a RubyGem over 3 years ago. Vagrant is written in Ruby, and at the time I was already comfortable with making RubyGems, so I decided it was a natural distribution mechanism.

I don’t regret this decision for a moment. The Ruby community is very open to cutting edge technologies, and I think that distributing as a RubyGem improved my initial adoption of the project.

Since then, however, I’ve grown and learned that RubyGems is no longer the best choice, for many reasons.

In this post, I’m going to explain why installers make life better for Vagrant users. In a future post, I’ll explain how installers affect plugin developers and that ecosystem.

Non-Rubyists #

Since the initial release of Vagrant in 2010, Vagrant has grown to be used by hundreds of companies and thousands of people. While I don’t have hard numbers to back this up, a majority of the Vagrant users I meet are not Ruby developers.

Prior to making installers, the top complaint I would receive about Vagrant was about RubyGems. As a Ruby person myself, I always considered RubyGems to be simple. But for anyone outside the Ruby community, I learned that the burden of having to learn RubyGems, install Ruby, and so on was a high enough barrier to entry that many people didn’t even try to install Vagrant.

On the other hand, packages that bundle all of Vagrant’s dependencies (including Ruby), make installing Vagrant extremely simple.

Bugs #

Vagrant has a lot of external dependencies: Ruby, RubyGems, OpenSSL, zlib, JSON, and more. With the RubyGems approach, users were left to manually install these dependencies, either from source or their package manager of choice.

Given this flexibility, I was unable to test Vagrant in every conceivable environment. I knew Vagrant worked on my system with my installed versions of dependencies, but didn’t know if it would work in every system.

Because of this, it was common to receive bugs where Vagrant was used with dependencies that were too old, had bugs, or were simply broken due to permission errors and so on.

I don’t care whether this is perceived as my fault or the user’s fault. I don’t blame users for not installing proper versions, especially if they just installed what their package repository had.

However, it is a huge problem when Vagrant crashes due to a broken environment, and then people perceive this as an instability in Vagrant itself, when 9 times out of 10, it was due to a misconfigured environment.

With installers, I am able to ship Vagrant with all the dependencies it needs, and promise the user that Vagrant will work with those dependencies. This is especially helpful in Windows environments.

Flexibility #

Because I strictly control the environment, I’m able to improve and fine tune the dependencies Vagrant relies on.

In Vagrant 1.1, I replaced the old, slow pure-Ruby tar library i was using with libarchive, a high performance, stable, and flexible application and library written in C. libarchive is now shipped with every Vagrant installer, so users don’t have to install another dependency, and don’t really need to care Vagrant is using it.

In Vagrant 1.2, I replaced the downloader from using pure-Ruby HTTP to using cURL. This is much faster and supports many more network protocols, such as FTP. Every installer will ship with cURL pre-installed within it. Again, users don’t need to care.

In the future, I’ll be replacing the pure-Ruby SSH library with a more stable, well maintained SSH library written in C. Once again, users don’t need to care, but they’ll notice that SSH becomes more stable.

With all of these changes, I’m able to make significant dependency changes without the user noticing. In fact, all the user notices is that the Vagrant experience just got a lot better. Things got faster, things don’t crash, and Vagrant supports more features.

Overall, Vagrant got a lot better because I was able to make changes I couldn’t safely make by distributing via RubyGems.

APPLE: My Key to Success

2013-03-04T09:53:39-08:00

No, not the company or the fruit. APPLE is an acronym ingrained into every Apple store employee before they ever even step on the retail floor. And it has continued to guide me ever since.

APPLE:

Approach customers with a personalized warm welcome
Probe politely to understand all the customer’s needs
Present a solution for the customer to take home today
Listen for and resolve any issues or concerns
End with a fond farewell and an invitation to return

To date, I’ve had three successful side projects. I attribute a large portion of each of their successes to APPLE.

My first successful side project turned into a healthy passive income business which I later sold to self-fund HashiCorp. The second side project was Vagrant, a popular tool used by thousands of companies and also now my full time job. The third project, IRCRelay, was a weekend project I worked on with Jack Pearkes over the course of two months. It launched in November 2012, and was profitable in less than 6 weeks.

While much more than just APPLE is required to be successful, I can confidently say that without APPLE, each of the aforementioned projects would have failed.

When doing anything, there is nothing more important than being respectful to your customer or user. Take the time to be welcoming, polite, and attentive. Present a real solution to their problem. Invite them back if they need anything more. Maybe they’ll help you by continuing to support your work, but maybe they won’t. It doesn’t matter. All that matters is being a good, caring person.

As a single concrete example, let’s look at a real email conversation from IRCRelay support, anonymized to protect the customer’s identity. An IRCRelay user, Brian, sends this email only 20 minutes after signing up:

Brian writes:

My username is “brian” please cancel my account.

At this point I could’ve quickly responded to Brian, pointing out we have a “cancel my account” link directly on the page shown after logging in. Similarly, when someone comes into an Apple store wanting a MacBook, the specialist could easily just point to the MacBooks and say “there they are.” They don’t do this. Likewise, instead of pointing to the “cancel my account” link, I responded:

Mitchell writes:

Brian,

Of course! I went ahead and deleted your account. Just so you’re comfortable, I want to remind you that you won’t be billed.

Also, thanks for giving IRCRelay a try, we really appreciate it. Do you mind sharing why you were unhappy with the service?

Approach and Probe. Brian responds:

Brian writes:

Thanks! I appreciate that being so easy. I had trouble with my IRC nickname. Whenever I connected it would be “brian_” instead of “brian.” Initially, I was already connected to IRC from my old client, so I disconnected. But the nick wouldn’t change itself back to “brian.” Contacting help is a bit too much trouble, so I just gave up.

I respond, Presenting a solution:

Mitchell writes:

Brian,

We just deployed a change to our system so that it automatically tries to regain your desired nickname if it wasn’t able to upon connection. This will avoid this issue for all future users. Thanks so much for pointing it out.

If you ever feel like giving IRCRelay another try, I’d be happy to give you an extra month free on top of the 30 day trial, as thanks for pointing out this issue. And please don’t hesitate to contact me directly should you need any more help.

It is important to note here that when working at an Apple store, a solution isn’t just something that solves the customer’s problem, it is the solution that solves the customer’s problem best.

I could’ve easily responded to Brian by educating him on how he could’ve used the /nick command to change his nick. Or I could’ve just offered to fix this for his specific IRCRelay account. Instead, the best solution was clearly to implement and deploy a system to automatically avoid the issue altogether.

The solution took extra time and effort, and maybe Brian won’t even sign up again. But it doesn’t matter. It was the right thing to do, and I genuinely do care about making the experience the best it can be for every customer.

At the end of the email, I End with a fond farewell, and invite Brian to contact me again if he needs anything. Brian has not yet signed up again, but I’m confident he will.

A little bit of APPLE goes a long way.

The New Normal

2013-01-21T14:32:00-08:00

The “new normal” is what I’ve named a phenomenon I’ve seen multiple times in my life. It’s an event where your perception of normality is changed by the people surrounding you. Consequently, your ambitions and chance of achieving those ambitions change as well. Actively seeking a new normal can steer you in a direction you never thought you’d take.

Looking back, four critical events of acquiring a “new normal” define my life today. They can be summed up as: serendipity, opportunity, choice, and confidence.

Surprisingly, I’ve found that many of my friends appear to be following a similar pattern, even in non-technical fields. Contemplating these moments of achieving a “new normal” is a fun thought experiment to see how you’ve grown and steer where you’re going next.

Event One: Serendipity #

My first critical new normal happened when I was 12 years old. I participated in an online BBS dedicated to writing cheat programs for a popular web-based game. I would spend two to four hours per day after school on this BBS, socializing with a group of people that spent their free time finding ways to exploit web games and writing simple Windows programs to automate these exploits.

By the age of 14, I started a business with a friend from this BBS – that I had never met in person – creating web game cheats and selling them to people for $25/month. I made over $400 per month as a freshman in high school, felt fabulously wealthy, and continued this until I received a cease and desist order from a few companies and was forced to shut down (mostly at the insistence of my confused parents).

None of my real life friends understood what I was doing, and my parents were concerned that I was acting abnormally. I had unintentionally changed my perception of normality to spending every waking moment (that my parents let me or didn’t know about) honing my programming skills in order to cheat video games. After all, that’s what all my internet friends were doing. It was normal!

Event Two: Opportunity #

By the time I was 18, my normal was set: I was programming every single day, spending a majority of my time with open source, and blogging about anything and everything. I even submitted a detailed proposal to PHP’s Zend Framework at night while on vacation in Tokyo. I knew I had found a passion.

My activity in the PHP community and popularity of my now defunct PHP blog (which at its peak had over 10,000 subscribers) helped lead to my first opportunity: a short two sentence email asking if I wanted a job as a Ruby on Rails programmer.

After a few weeks of talking back and forth, I got the job. This company specialized in building MVPs for new tech startups. This was my introduction to the world of startups. I worked as a developer for this consultancy at at least four different tech startups a year, helping build their MVPs.

I started building more and more web applications targeted at consumers rather than simply building cool technical projects or working purely on open source. My normal shifted again: Build useful web applications for people, ship fast, and charge for it. I fell in love with startups.

Event Three: Choice #

Closing in now on graduating college, I knew I had to move to San Francisco and work for a startup. This was a fundamental shift in how I previously reached a new normal. Instead of waiting for things to happen, I chose to take the initiative and move to a new city where I had no friends in order to surround myself with what I wanted to become.

Coincidentally, through the same friend I started the web game cheating business with, I met Brian Wong in 2010 and became the first hired engineer for Kiip. I moved to San Francisco in 2011.

At Kiip, I had courtside seats watching Brian raise funding. I was able to meet venture capitalists and founders of dozens of companies through social events. I met young successes, up-and-comers, and industry veterans. Each showed me a piece of what defines San Francisco tech.

Normality had changed again, this time by choosing who I surrounded myself with. It was normal to relentlessly follow your passion. It was normal to believe that you could chase big dreams at unsurmountable odds and succeed. It was normal to start a company, at any age.

Event Four: Confidence #

In mid-2012, after working at Kiip for just over two years, I made the decision to leave and start my own company: HashiCorp.

This was never part of any plan. I thought I’d be at Kiip for at least four years and I thought Vagrant would forever remain an obscure side project. Instead, the rapid increase in popularity of Vagrant, the pressure from companies for certain features and support, and my desire to work on it more hours of the day pushed me to make a change.

I was told many times before that I should start my own business working on Vagrant. But it wasn’t until my perception of normality changed from my choice to move to San Francisco that I was able to gain the confidence to really take the leap.

And this is where I sit today. I believe I am on the cusp of another change in normality. I do not know when that will be or how my behavior will change, but I’m excited to look back and see how my decisions and ambitions are steered because of it.