Spinn3r is hiring for an experienced Senior Operations and Infrastructure Engineer with solid Linux and MySQL skills and a passion for building scalable and high performance infrastructure.

This role is about 80% engineering in future infrastructure work (so that we can scale into the future) and 20% routine operational tasks.

It’s a great opportunity for the right candidate. The fact that most of this will be engineering and building new systems should make it a lot more challenging than a traditional operations role.

My goal is to spend the right amount of time doing long term infrastructure work so that fires and machine failures are fully routine and require no human interaction.

Right now they require little human interaction but it’s still required in some of the more serious circumstances.

This IOPS distribution is very interesting.

I’m playing with a RAID array of 5x Intel X-25E drives.

It turns out they need a lot of tuning. I’ll blog about this later.

What is more interesting is this distribution of IOPS across threads.

This is using sysbench and the seqrd file IO test.

My hypothesis is that it’s wear leveling the ext3 block group.

I also tried using ext3 striding and that did yield a performance boost. I’m going to try to use XFS but I’m on a CentOS box or testing.

Anyone have another theory as to what could be causing this?

I can’t wait until RAID is dead.

Update: This problem actually happens on a single 1x Intel X-25E so it seems like a hardware issue.

Update 2: It turns out that this is NOT a bug with the Intel SSD. I’m pretty sure it’s a bug (or misconfiguration on my part) of sysbench. Performing the same tests with ‘dd’ shows that IO scales linearly up until at least 10 parallel sequential reads. So something else is broken here.

It looks like Google and Bing are now paying for a Twitter firehose feed.

What’s interesting here isn’t the deal – what’s interesting is that Google is PAYING for content.

In the past they have refused to pay for content acquisition. Your pay was the traffic that Google sent your way.

Now the NYTimes can come to Google and they have a bit more leverage this time. You’re paying for Twitter content! Hand us over some change for the NYTimes or we’ll cut you off…

My take on the subject is that content providers like Facebook, MySpace, etc should create open networks for their users.

There’s a huge value in creating an open eco-system around content. Creating artificially walled gardens is so CompuServe/AOL. This is Web 2.0. You’re either open or you’re (eventually) dead.

What do you get when you take 7 Intel SSDs and throw them in a desktop? 1M IOPS:

“So as we look at optimizing some of those things,” he said, “like interrupts, driver speculation, improving the physical interface between SSDs, and the system, we expect great gains in power, performance, and cost.”

Gains in performance, for example, that add up to 1M IOPS (input/output operations per second) in Coulson’s lab, where he hooked up a dual Xeon 5500 desktop tower to seven Intel SSD prototypes – four in the tower and three in a PCIe expansion box – and ran a 4K read/write benchmark on it.

“This many I/Os per second is about four gigabytes per second of storage bandwidth,” he claimed. “As a storage guy, that’s a huge number, a very huge number.”

He also pointed out that during the test, the CPU utilization of the tower was about 50 per cent. “That’s really nice,” he smiled.

Mark Callaghan made some progress with 4k pages in and InnoDB. I ran some number on SSD with 8k pages but my 4k build would dump core.

The numbers for 8k show some potential.

The way I see this working is that one would buffer the first and second level btree pages in memory and the rest of the InnoDB database would be served from SSD.

We’re going to be trying to run InnoDB on 4x 64GB Intel SLC SSDs.

The key is that have high sustained reads with a much larger storage capacity.

With 4k pages we’re going to be doing far less SSD reads. However, the SSDs are so fast that reading the extra data might not impact performance significantly.

As an aside. InnoDB should be able to have a runtime reconfiguration of page size. ALTER TABLE FOO page_size=4096 would be nice.

Spinn3r will be hosting an Open MySQL meetup at Oracle Open World (which is right down the street).

This would be on Wed 10/14 2009 at 7pm … at 580 Howard Suite 301 (Spinn3r HQ)

Oracle owns MySQL, InnoDB, etc so I suspect a lot of Oracle people and MySQL hackers will be interested in attending more of an Open Source and community centered meetup.

We’ll just be hanging out at our offices … we’ll have beer and food.

Feel free to bring your laptops as we have Wifi :)

This is contingent on at least 10 RSVPs as I want to make sure there is interest from the community.

Please RSVP here

It’s a bit late notice so if you could help spread the world by blogging about this that would be GREAT!

Has anyone done any more work on recompiling InnoDB with 4k pages and benchmarking under SSD?

We’re building out a new DB that uses very small records (around 32-64 bytes) so reading a whole 16k for this record should have a performance difference.

I haven’t seen any benchmarks on 16k random read IOPS on the Intel SSD but my hunch is that there will be a 20-30% penalty here.

Though even if it was a 4x penalty that would still be about 9k transactions per second which is pretty good.

On a personal note I just bought a new Mac Book Pro which will be upgraded to the Intel X-25M MLC SSD.

Needless to say I’m very excited!

We’re looking to hire a Senior MySQL DBA over at Spinn3r.

You should obviously have MySQL experience. Love SQL, hate data corruption and slow queries, and preferably live in San Francisco.

Linux experience would be nice as well but not required.

Extra points if you are excited about SSD, *huge* amounts of data, have hacked on Drizzle or XtraDB

Spinn3r is a GREAT place to work. We’re growing fast and have cool new offices in SOMA.

Spinn3r is growing fast. We’ve had an exceptional month (an exceptional year actually). Closing new deals. Releasing new features for our customers. Working on new backend architecture changes, and generally having a lot of fun in the process.

We’ve been posting to Craigslist like mad in the last few weeks but I wanted to take the time to post to our blog.

We’re hiring five new Engineers to join the team with us here in San Francisco.

This is in addition to the two new Engineers we’ve hired in the last couple months.

We’re hiring two Crawl Engineers, Operations Engineer, Support and QA Engineer, and Java Engineer.

Spinn3r is a great place to work. Smart people. Huge amounts of data. Great customers. New offices in SOMA (we’re in an awesome 103 year old building) and plenty of interesting problems to work on…

Here’s the problem I currently have.

We’re looking at deploying the Intel X-25M MLC SSD in production.

The problem being that this drive has a lower number of erase cycles but is much cheaper. Than the Intel X-25E SLC drive.

However, in our situation we’re write once, read many. I’m 99% certain that we will not burn out these drives. We write data to disk once and it is never written again.

The problem is that I can’t be 100% sure that this is the case. There is btree flushing, and binary log issues that I’m worried about…

What would be really nice is an API (SMART?) that I can enumerate the erase blocks on the drive, determine the max erase cycles, and read the current number of erase cycles.

This way, I can put an SSD into production, then determine the ETA to failure.

I can also add this to Nagios and Ganglia and trend the failure date and alert if the derivative is too high and the drive will soon fail.

Further, I can figure out if a database design is flawed. If I deploy a new database into production and the failure ETA is too high after 24 hours I know that something is wrong. Either a misconfiguration or a problem with the design.

I think this would solve a LOT of the problems with deploying SSD in enterprise environments. (MySQL, Oracle, etc)