SoldFire RSS

Extending The Storage Disruption Cycle

Thu, 26 Jan 2012 00:00:00 GMT

"There comes a time when a storage company needs to define itself by what it does for customers and not by the machinery it uses to do so."

Chris Mellor, "How to tell if your biz will do a Kodak", The Register

The Register's Chris Mellor penned a great article the other day reflecting on the continuous cycles of innovation and disruption that have come to characterize the storage media industry. He uses Kodak to paint the picture of an incumbent getting capsized by a media transition. He goes on to cite other examples across tape and optical media where incumbents failed to manage the transition to the next generation media.

As the storage industry has transitioned through different media types there have always been opportunistic stopgap innovations that have bridged the gap from one generation to the next. Virtual Tape Library (VTL) technology is a great example of an innovation serving as a transitional bridge between the tape and disk eras. Once applications were written with the capability to natively interface with disk, deduplication and compression drove down solution costs quickly making it an effective bulk storage medium. Once financially viable, the flood gates were opened and tape was relegated as a deep archive. Similarly, today we are seeing flash-based caching and tiering technologies forming a similar transitional bridge while the $/GB economics of flash fully converge with, and eventually eclipse, disk.

So with history as a guide for how this plays out, why will the disk to flash media transition be any different than the ones before it? Well, I suspect this cloud thing might have something to do with it.

In the enterprise IT sector, systems always seem to consume features over time. At its core, the cloud is a massive infrastructure system that when used properly is an extension of existing IT. However, cloud infrastructures will increasingly chip away at the incumbent IT footprint by rapidly incorporating new innovations into its architecture. These enabling innovations allow cloud providers to continually expand their portfolio of cloud services. Over time the IT use cases applicable to this medium naturally expand as applications and interfaces catch up, performance improves and the economic value proposition can no longer be ignored.

So what does this mean? From our perspective, the cloud adds a third leg to the innovation sequence we have witnessed in the past. New component level technologies will continue to enable new architectures. But where it gets interesting is when these new architectures drive the performance and economics to enable new cloud services.

In storage, the media innovations that Mellor refers to, and their related price/performance value proposition, are a powerful enabling force behind new storage architectures. Applied to traditional IT cost centers these architectures are interesting, when applied to profit-driven cloud services they are game changing. Amazon's recently announced DynamoDB service is an early instantiation of this extended innovation sequence where component level technologies (SSD), enable new architectures that drive new services. Fortunately for the end-customers, the economics of flash are only getting better from here. Now is it up to the storage industry to innovate on top of this medium, delivering next generation systems that can extend the reach of cloud hosted services to an even wider range of application workloads.

-Dave Cahill, Director of Strategic Alliances

Inefficiency & Unpredictability...A Service Providers Worst Enemy

Tue, 24 Jan 2012 00:00:00 GMT

In our first two posts on storage tiering we talked through the difference between capacity-centric vs. performance-centric approaches and also exposed some of the hidden costs of an automated tiering implementation. Closing out this mini-series I wanted to touch on a few other deficiencies inherent to an automated tiering solution.

Within a storage infrastructure it is IOPS, not capacity, that are the most expensive and limited resource. In a tiered architecture, SSDs are inserted into the equation to try and improve the balance between IOPS and capacity. However, while an SSD tier may reduce performance issues for well-placed data, the usage of this expensive tier remains inefficient. This inefficiency stems from a lack of granularity in the data movement of a tiered system. If a sub-LUN tiering system needs to move hot data chunks anywhere from 32MB to 1GB, it will likely promote a lot of cold data in the process. This overhead forces sub-optimal utilization of the premium SSD capacity.

Another potential problem area from tiering, specifically in a multi-tenant environment, is dealing with IO density - that is, how IO is distributed across a range of disk space. Applications whose IOs are concentrated within close proximity to each other (IO dense) will gain greater benefit from sub-LUN tiering than those whose IOs are spread more evenly over the entire logical block address space (IO sparse). Because tiering mechanisms measure data usage at the chunk level, an application who has more hits within a small number of chunks is more likely to be promoted than an application who spreads the same number of IOPS across more chunks. From an array performance perspective this approach is reasonable, as you get more performance within the same resource footprint. However, in a multi-tenant setting with data distributed across many distinct application this leads to serious problems with fairness and performance consistency across workloads.

We originally discussed the performance implications of tiering in July of last year. In a multi-tenant setting this performance variability exposure is magnified. Customers are continually exposed to the risk that the promotion of another customer's hot data will result in the demotion of their own. The order of magnitude difference in latencies and IOPS between the different tiers makes it practically impossible for a service provider to guarantee performance to an individual application (or tenant) under these conditions.

In recognition of the deficiencies of a tiered architecture, SolidFire sought a better way. Our Performance Virtualization technology decouples the tight binding between the storage performance and capacity, resulting in a far more precise allocation of IOPS and capacity on a volume by volume basis regardless of issues such as IO density. Instead of best guess efforts as to the size and tiers of media required to meet customer performance requirements, a service provider can now dial-in IOPS and capacity individually at the volume-level from cluster-wide independent pools of capacity and performance. These allocations can also be dynamically adjusted over time as application requirements change. All things considered, Performance Virtualization is a far more efficient way to address IOPS scarcity, without exposing customers to the inefficiency and unpredictable performance inherent in an automated tiering architecture.

-Dave Wright, Founder & CEO

Amazon launches DynamoDB...We like what we see!

Wed, 18 Jan 2012 00:00:00 GMT

Amazon launched a new service today: DynamoDB. It's a scaleable NoSQL database service that will run in the AWS cloud. It is akin to a hosted version of Cassandra or MongoDB with unlimited scalability. The most notable section of Werner Vogel's blog announcing the new service is worth repeating:

Cloud-based systems have invented solutions to ensure fairness and present their customers with uniform performance, so that no burst load from any customer should adversely impact others. This is a great approach and makes for many happy customers, but often does not give a single customer the ability to ask for higher throughput if they need it.

As satisfied as engineers can be with the simplicity of cloud-based solutions, they would love to specify the request throughput they need and let the system reconfigure itself to meet their requirements. Without this ability, engineers often have to carefully manage caching systems to ensure they can achieve low-latency and predictable performance as their workloads scale. This introduces complexity that takes away some of the simplicity of using cloud-based solutions.

The number of applications that need this type of performance predictability is increasing: online gaming, social graphs applications, online advertising, and real-time analytics to name a few. AWS customers are building increasingly sophisticated applications that could benefit from a database that can give them fast, predictable performance that exactly matches their needs.

Looking under the covers a bit further here there are two really interesting enabling components of the DynamoDB service that deserve highlighting:

All-SSD- the service is deployed using 100% SSDs to provide consistent high performance at a very large scale. This is notable in that it is AWS' first use of SSDs in their cloud architecture.
Guaranteed Throughput - The DynamoDB service includes a concept called "Provisioned Throughout". This is essentially a guaranteed QoS model, where a customer can purchase reserved capacity (measured in queries per second), rather than paying for the actual queries run. Applied to a storage service, this would be akin to paying based on guaranteed IOPS. Currently Amazon EBS's current pricing model is based on actual IO operations with no guaranteed throughput or latency.

Amazon DynamoDB is a strong endorsement of several of SolidFire's key principals. The first being that the cloud needs Solid-State Drives (SSD) to adequately support the evolving performance demands of multi-tenant storage. The second is the idea that as more of these performance-sensitive applications make their way to the cloud there is a clear requirement for guaranteed QoS controls that can dynamically support performance requirements at a much more granular level. Finally, and building off the first two, is the validation that when armed with the enabling architecture to confidently and economically deliver performance-based services, service providers can stand-up cloud service offerings based on committed performance.

Amazon is a great indicator on the pulse and direction of the industry. The broader implications here for running performance sensitive applications in a cloud environment are intriguing to think about. Here at SolidFire, the continued innovations around the enabling architectures required to make this a reality are what get us really excited.

-Dave Wright, Founder & CEO

The Diseconomies of Tiering

Tue, 17 Jan 2012 00:00:00 GMT

In the initial post of our series on tiering we covered the merits of a proactive performance-driven approach to tiering relative to the more traditional capacity-centric discussions. Today we take a closer look at some of the less obvious cost implications of "automated" tiering. On the surface, the promise of tiering looks like an clear win - SSD performance with spinning disk capacity and cost. However, the true economics of this type of solution are not nearly as compelling as some vendors would lead you to believe. Considered in the context of the unique burdens faced by cloud service providers and the proposed value proposition is even less appealing.

To start with, the "SSD performance" promise part of the catchy tagline above must be caveatted by the fact that this only proves to be the case if the data is actually residing in the SSD tier. Easier said than done. The ability to guarantee SSD performance in a tiered architecture requires a substantial SSD tier and/or extremely accurate data placement algorithms. Rightsizing the former skews the proposed economics of a tiered solution substantially, while the latter has been long on promise but short on delivery for at least three generations of marketing executives. Before the industry marketed this functionality as Automated Tiering it was known as Information Lifecycle Management (ILM) and a few years before that it was Hierarchical Storage Management (HSM). Regardless of what you call it, tiering has always been impaired by the inability to accurately predict and automate the movement of data between tiers. In the context of cloud environments the significant scale requirements and extremely low application-level visibility make solving this challenge even more difficult.

It's also important to consider the flash media requirements of a tiered solution. The write patterns in the flash layer of a tiered architecture require a higher grade flash solution to withstand the impact of write amplification and churn. Vendors are forced to use the most expensive SLC flash to ensure adequate media endurance. The cost impact even modest amounts of SLC flash destroy the economic advantage of a tiered architecture relative to an all-MLC design. In many examples we've seen that the "combined" $/GB of a storage solution that incorporates SLC-flash, 15k SAS and SATA is actually higher than an all-flash MLC solution with similar raw capacity. Importantly, this price advantage for MLC over tiered storage is achieved before factoring in the favorable impact of compression and deduplication for the all-flash solution, making the flash design even more compelling.

Tiering also hurts capacity utilization and controller performance. In order to ensure data is in the right place at the right time it is constantly being promoted and demoted between the flash and disk tiers. There needs to be a certain capacity buffer to accommodate this movement. There is also a controller processing cost to keep up with all this activity. Most legacy systems have limited CPU and controller memory relative to their overall capacity, making the overhead of tiered storage processing one more burden for them to manage. Even complex tiering requires only a fraction of the processing power and memory needed for in-line data reduction features like compression and dedupliction, which is why those features are seldom found on legacy primary storage controllers. A recent article from TechWorld references a Forrester Research report by Andrew Reichman (@ReichmanIT) that expands on the data management burden of a tiered storage topology.

The issues outlined above are just a few examples of the hidden costs embedded in an "automated" tiering solution. In some cases these deficiencies may be acceptable in smaller IT environments. However, in a large scale multi-tenant cloud infrastructure the capital and management costs of these shortcomings are magnified. The hyper-competitive nature of service provider business model necessitates a more efficient approach.

-Dave Wright, Founder & CEO

Capacity vs. Performance Tiering

Tue, 10 Jan 2012 00:00:00 GMT

In our end of year blog we reviewed a number of the unique storage challenges that infrastructure service providers face in building and operating a large-scale, and profitable, cloud offering. A clear understanding of these issues provides a more constructive lens through which to the viability of a storage solution within a high-performance cloud-scale setting. This approach is particularly useful for understanding the basis of SolidFire's thoughts on the merits of "automated" storage tiering in a large scale cloud.

As promised, we kick off our first of three blogs on this topic below. If you happened to miss our initial thoughts on this subject you can go back and read them here and here. We look forward to your feedback as we go.

Within the enterprise, storage tiering has become a popular vendor solution to improve performance for a subset of applications. With tiering the performance gain is achieved by retrofitting a disk-based array with an SSD tier and some intelligent fetching/data placement algorithms. Tiered storage systems are most effective when an IT manager has direct visibility into the usage profiles of the applications that reside on the system. This allows the IT manager to size each tier appropriately, continually ensuring there is enough room in "fast disk" to accommodate demand. When data is not in demand it is then moved to slower speed disk. Overall, this is both a reactive and human-centric model that requires constant monitoring and adjustments to ensure each storage tier is rightsized to accommodate the access patterns of different volumes across the data set. The continuous promotion and demotion of data to the different tiers also comes at the cost of endurance due to excess wear on the flash media.

When operating a large scale public cloud environment customer applications and their associated usage patters are largely unknown to the service provider. How do you most effectively allocate tiers of storage without ongoing visibility into the access requirements of a particular application? How big should the SSD tier be? How much SATA capacity should be used? When should data be promoted or demoted between tiers? Might a better question be; how many IOPS need to be available within the storage system? Unfortunately, for cloud service providers with unpredictable demand patterns across a large number of tenants, trying to spec out a system in this manner is impossible.

From SolidFire's perspective, the best way to manage performance in a multi-tenant cloud environment is to approach this problem from the demand side of the equation (i.e. application performance) as opposed to the supply side (i.e. storage capacity). Proactive performance management based on IOPS demanded by the application offers a far more efficient approach to allocating storage resources, rather than trying to guess the right quantity and capacity of each tier within the system. Armed with fine-grain performance controls, storage performance management should no longer be a complex, reactive and resource intensive experience. By leveraging a system that can assign and guarantee IOPS on a volume by volume basis, all of the guesswork around right sizing for application performance is eliminated.

For a quick graphical depiction of how SolidFire brings this concept to life, check out our 90 second video on Performance Virtualization.

-Dave Wright, Founder & CEO

Looking Back Before We Charge Forward

Tue, 13 Dec 2011 00:00:00 GMT

2011 was a foundational year for us here at SolidFire. Emerging from stealth mode at Structure in June, to a great VMworld panel and TechFieldDay appearance in September, and more recently announcing our new financing round in late-October, we have been hard at work. The best part is we are just getting started. Beyond enhancing our product, building our team and spreading the word, we have spent countless hours with cloud service providers (CSPs) listening to the challenges they encounter when attempting to deploy profitable high-performance cloud-storage infrastructures. Today we are solving these problems with a select group of early access customers, and we look forward to making the SolidFire system broadly available in 2012. We don't think that cloud computing will ever be the same.

Conversations with CSPs throughout the past year has continued to reinforce our belief that this customer segment is unique in its scale, business model and the solutions that it requires. The driving force behind this conclusion are four important qualifiers that clearly differentiate their IT environment and resulting storage system requirements from that of the traditional enterprise. These factors are:

Ability to Provide Predictable Performance
Massive Scale
Multi-Tenancy
Lack of Application-level visibility

Individually, each of these factors impose unique pressures on the IT environment. Taken together, they demand an entirely new approach. Deeply understanding the architectural implications of these collective burdens provides a more constructive lens from which to assess the viability of one solution versus another in a cloud-scale setting.

A frequently debated topic that highlights the importance of evaluating customer requirements from a more holistic viewpoint is that of automated storage tiering. Originally blogging on the topic back in July, we have continued to evolve our thinking on the topic and I would like to introduce a blog series in which we cover our view on tiering at length.

Talk to any IT manager about how they are keeping up with performance demands and you will increasingly hear talk about resorting to unpredictable and resource intensive band-aids like tiering. In a controlled single system environment the dynamic tiering of data between SSD and SATA drives would seem to make sense. Unfortunately, the economics of these more tactical approaches, while viable in smaller topologies, start to break down under the burdens imposed in a cloud environment.

Once you introduce the elements of multi-tenancy, multiple applications, and the need to scale across multiple systems, a tiered approach exposes CSP customers to "all or nothing" performance disparity. Working around the unpredictable nature of this setup requires human intervention eroding the proposed cost benefits of the automated tiering value proposition. When evaluated against criteria above, the shortcomings of an automated tiering approach start to become very clear. Cloud service providers are forced to seek out alternative solutions that are better aligned with both the performance controls required in a multi-tenant cloud environment, and the efficiency mandated by the hyper-competitive nature of the cloud services market.

There is little debate that Quality of Service (QoS) is a key competitive differentiator for CSPs. Consequently, they cannot afford to gamble with the performance variability inherent to a tiering or cache-based solution. The manual intervention required to tune and optimize these architectures on an ongoing basis is the antithesis of a profitable cloud-scale management model. Coming out of the holiday break we will further explore storage tiering in even greater detail. We will look at the differences between capacity and performance tiering, dive into the true economics of tiered solutions, and hash out the merits of local versus global deduplication. As always, please provide your feedback here on our blog. We look forward to the conversation.

Happy and safe holidays to everyone and we look forward to seeing you in 2012.

-Dave Wright, Founder & CEO

SolidFire adds fuel to all-SSD storage solution with $25M in funding

Mon, 31 Oct 2011 00:00:00 GMT

Over the last few months we have been writing about a number of topics surrounding SolidFire's all-SSD storage technology. It is important for us that we strike a balance between being informational about SolidFire, but also educational about how some of the most successful cloud service providers in the world are thinking about SSD technology and how guaranteed QoS is impacting their business. Here are SoftLayer and Virtustream discussing their thoughts on the use of solid-state technology in their cloud.

Currently there are a number of world-class cloud service providers evaluating over 500TB of SolidFire's all-SSD storage technology. They are evaluating the solution technically, but also evaluating it from a business perspective as well. For every customer we work with, SolidFire technology is radically changing their business. These IaaS providers are now able to invite new mission critical and performance sensitive applications into their cloud, and build new revenue streams and customer value around guaranteed performance. There is a very good reason that 3Par, EMC, and NetApp customers have all joined our Early Access program.

No other storage technology in the world has SolidFire's capability to combine revolutionary performance management, in-line storage efficiency, and full system automation.

There are many service providers out there simply making due with what has been available in the market. If you are reading this blog, you are probably one of them. You, and each of our early access customers all feel the same way - current technology can't get me to where I want to go.

SolidFire can. SolidFire can bring your cloud to the next level and add to your bottom line in a way that no 3Par or NetApp system ever could. Think deduplicating a single volume is interesting? How about deduplicating your entire data store across thousands of customers. SolidFire offers not just incremental change, but rather a massive leap forward in how storage systems really SHOULD be built. Why wait for your current vendor to drag themselves up to date?

To help get SolidFire technology in front of every cloud service provider, today we announced the closing of our $25 million Series B funding round, bringing our total funding to $37 million. We will be investing in our sales and marketing teams to broaden our reach, and will be accelerating our technical development as well.

SolidFire is on a fantastic roll and we want to give you the chance to learn about our technology. We have a webinar coming up on November 17th and want to urge you to carve out an hour to spend with us. We will be talking about: How Performance Virtualization Enables New Storage Services in the Cloud.

If your cloud is held back by complex, expensive storage systems and would like to know more about our solution attend our webinar or Talk with Us!!

-Jay Prassl, VP of Marketing

The Challenges of Cloud Service Providers-Part Three - Recap

Tue, 18 Oct 2011 00:00:00 GMT

To wrap up our VMworld video series hosted by Silicon Angle TV, I sat down with Virtustream's Matt Theurer and Softlayer's Duke Skarda to discuss as a group, some of the challenges faced by cloud service providers. This conversation focuses largely around the barriers that these two companies face with traditional storage systems in the cloud, and the opportunities that flash storage presents.

For both companies, the use of all-SSD based technologies is changing the way they think about storage, and how they approach resolving the gap between server and storage performance. Matt discusses how SSD technology has inverted the capacity / performance imbalance that has existed for many years and how capacity will soon be the limiting factor within cloud storage architectures; a much easier metric to manage. Duke explaines how block storage is a fundamental building block of cloud infrastructure, and traditionally the most problematic part to deal with.

I also got a bit of airtime to talk about the history of SolidFire and how my experience at Rackspace, and evaluating how traditional storage is used within the cloud, both helped me shape the technology of SolidFire and the market focus of the company. It is important to keep in mind that SSDs do not constitute a different approach to storage. SSDs are just part of the system. How that system is architected, the functionality designed around the SSDs, and deep knowledge of your customer and their key feature-set, are all required when delivering a next generation storage solution.

Many thanks to Matt and Duke for sharing their views on performance storage in the cloud, and to Silicon Angle TV for hosting us!

-Dave Wright, Founder & CEO

The Challenges of Cloud Service Providers-Part Two - Virtustream

Wed, 12 Oct 2011 00:00:00 GMT

At VMworld earlier this fall Matt Theurer, SVP of Solutions Architecture and Rodney Rodgers, Chairman and CEO of Virtustream took some time to sit down with the folks SiliconAngle TV to discuss some of the challenges that cloud service providers are facing. During their discussion they talked about the specifics of their business and their focus on enabling high performance applications like SAP within their shared infrastructure. Key to their success in this space has been their ability to carve up compute, networking and IOPS and bundle them into what they call an "infrastructure unit". Customers can combine as many IUs as needed to meet their requirements, and this enables Virtustream to provide some of the most comprehensive SLAs in the industry.

Matt takes the conversation a bit deeper discussing some of the more granular performance challenges posed by traditional spinning media. He discusses how the ability to guarantee storage performance would allow them to be even more exacting in their SLAs and raise their overall efficiency. At SolidFire, one of our primary goals is to enable cloud service providers to allocate storage performance as easily as they allocate storage capacity; and to do so for thousands of volumes within a shared infrastructure. This capability allows companies like Virtustream to wrap SLAs around exact performance metrics and maintain customer performance expectations regardless of the activity within the system.

-Jay Prassl, VP of Sales & Marketing

The Challenges of Cloud Service Providers - Part One - Softlayer

Tue, 04 Oct 2011 00:00:00 GMT

Nathan Day and Duke Skarda of SoftLayer were kind enough to talk with the guys from Silicon Angle/Wikibon.org on the Cube at this years VMworld. During their discussion they touched on a major challenge that many cloud service providers are dealing with today… storage performance. One of the points that was brought up was fine grain control on Quality of Service. They referred to "per volume" or "per account" control as storage nirvana. At SolidFire, our architecture was designed from the ground up with this in mind. The ability to guarantee consistent QoS to thousands of applications and thousands of customers is how SolidFire is making storage nirvana a reality.

-Adam Carter, Director of Product Management

Cloud: The Triumph of Automation Over Administration

Thu, 29 Sep 2011 00:00:00 GMT

What makes cloud computing a better way to manage an IT infrastructure? Is it the flexibility? The cost savings? The improvements in utilization and efficiency? More importantly, what is the fundamental difference between cloud computing and traditional IT management that enables these benefits? In a word, the answer is automation.

Automation is what allows cloud infrastructures to expand rapidly without a corresponding increase in administrator headcount. Automation is what allows business teams and developers to deploy and scale applications in the cloud in minutes instead of weeks. Without automation, it would be impossible to keep web-scale applications such as Twitter, Facebook, and Ebay running. Also, without automation, the cost of managing these web-scale environments would quickly spiral out of control.

Web-scale companies such as Google and cloud service providers such as Amazon and Rackspace have long realized that automation is the key to running their data-centers and their business. But when it comes to storage, automation has remained elusive. Storage vendors selling products to IT administrators have created systems that are administration-centric, with fantastically complex tools that require months of training and certification to master. In a cloud setting, these tools don't scale because there simply aren't enough humans to manage them. For years, storage vendors have treated automation as an afterthought, something outside the norm, which has forced cloud providers to perform unnatural acts like scripting vendor-specific CLIs.

At SolidFire, we think automation is the only way to scale the cloud and is our first and most important focus when it comes to management. We have built into the core of our system a comprehensive REST-based API that allows service providers to automate any aspect of the system - from deployment to provisioning, to security, reporting, and billing. Failure mitigation, adding capacity, performance management, backup and restore capabilities are all included as well. Our web-based and CLI tools are simply wrappers around the API, providing a good way to get started on your way to a fully automated storage environment.

While our API makes it easy to integrate SolidFire into home-grown cloud management stacks, we are taking things a step further by creating pre-built integrations with cloud stacks such as VMware vCloud Director and OpenStack. Our product management and development teams will be at the OpenStack Conference next week as part of that effort. If you are attending the conference ping us on twitter (@Solidfire) or swing by our booth.

One final point: these days, everyone claims to have APIs, and automation is fast becoming a buzzword across the industry. If a system enables you to automate just one or two aspects of storage management, this is not cloud-scale automation.

The next time a storage vendor talks about automation, try this simple test: ask them if you can you take their system out of the box, put it in a rack with power and a network connection, and have APIs handle the rest. That means you walk away, and you don't have anyone log into this box as long as it is on your network. That is the true essence of cloud-scale automation. If you'd like to see how a storage API can accomplish that, see us at the OpenStack conference or feel free to contact us.

-Dave Wright, Founder & CEO

Get Rid of the Guesswork

Wed, 14 Sep 2011 00:00:00 GMT

Building a cloud infrastructure requires careful planning and technologies that give you confidence. Balancing system performance, efficiency, and cost are important to be competitive in the cloud market, and are required for building a profitable Block Storage as a Service offering. If you are using educated guesses, or assumptions around performance and efficiency as you build out your service you are opening yourself up to risk.

So why talk about this now? Well, because SolidFire has invested in a tool that eliminates the guesswork that is often used to plan for thin provisioning, compression, and deduplication. We have talked earlier about efficiency and how these storage technologies in concert with SSDs, can be huge game changers in the cloud. Historically it has been difficult to design these advantages into an infrastructure without debilitating performance impact. Trying to understand how the storage system itself affects efficiency complicates this even further. Each cloud service provider is different and efficiency results can vary based on the data stored and the type of thin provisioning, compression, and deduplication a vendor offers.

It's hard to have any rule of thumb that would expose how much thin provisioning, compression, or deduplication can save. You can imagine that thousands of virtual desktops would compress and dedupe like crazy, but exactly how much? Are you sure they will as much as you hope? You also don't always know all the details about how a particular vendors feature works, or how it might work when combined with other features. Does dedupe span multiple volumes? How do I account for dedupe in a multi-tenant environment? What segment size does it scan?

SolidFire's answer to these questions is to stop guessing.

SolidFire developed a command line utility with the ability to look at a specific set of data and say exactly how much SolidFire capacity would be required to store that exact data. eScanner evaluates block devices, files, file trees, or vmdk files and tells you exactly how much of that data is real, how much it would compress, and how much it would deduplicate on a SolidFire storage system. The same utility is also capable of aggregating multiple data sets so you can see how much more effective deduplication gets as you put more data on a SolidFire system. It's refreshing to be able to deliver clear and direct answers about efficiency, and to set expectations about data reduction rates based on real data.

I'm excited about our recent release of the eScanner utility and really interested to see what feedback we get about the different data sets users scan. I encourage you to download eScanner, run it as widely as you can, and see how much more efficient your data could be stored within a SolidFire system.

-Adam Carter, Director of Product Management

Designed for Solid State

Mon, 22 Aug 2011 00:00:00 GMT

For the past 20 years, network storage systems have been designed around spinning disk, with the form factor, performance characteristics, and reliability profile of the HDD dominating every architectural and design decision that was made. Many of these systems with 10-20 year old architectures are now bolting on solid state disk, which comes with a number of tradeoffs that Adam previously discussed. However, today, rather than focus on the problems with traditional architectures and SSDs, I want to focus on the advantages of a system designed from the ground up for solid state.

We did just that here at SolidFire. We built one of the first general-purpose storage systems that has been designed exclusively for solid state storage. Our use of solid state did not simply influence small decisions about data layout and I/O scheduling, but rather drove the entire architecture of the system. We completely rethought how a storage system could function if you were to remove disk from the picture, and ended up with a storage architecture that has very little resemblance to a traditional SAN.

From the outside, the system may look similar to other scale-out storage systems, with nodes and drives and iSCSI networking, but underneath the covers is something so different, it could never be built with spinning disk.

This fresh approach gives our customers tremendous benefits; such as increased performance, "hard" fine-grained quality of service guarantees, in-line deduplication, and reduced SSD wear. These technology advantages are enabling cloud service providers to invite mission-critical, performance-sensitive applications into a cloud infrastructure with greater confidence.

-Dave Wright, Founder & CEO

Not All QoS Is Created Equal

Tue, 09 Aug 2011 00:00:00 GMT

Quality-of-Service (QoS) features exist in everything from network devices, to hypervisors, to storage. When multiple tenants share a limited resource, QoS helps provide some level of control over how that resource is shared and prevents the noisiest neighbor from disrupting everyone.

In networking, QoS is an important part of allowing real-time protocols such as VoIP to share links with other less latency sensitive traffic. Hypervisors provide both hard and soft QoS by controlling access to many resources including CPU, memory, and network. QoS in storage is less common, but is now available on many high-end arrays. However, most approaches to storage QoS are "soft" - that is, based on simple prioritization of volumes, rather than hard guarantees around performance. Soft QoS features are effective only as long as the scope of the problem is small enough. In enterprise environments where an administrator has visibility across a global portfolio of applications, and performance fluctuations are not penalized as heavily, it is conceivable that prioritization can be managed with this more simplistic approach.

However, in a large scale cloud environment, these soft QoS implementations come up short. When multiple tenants share storage, the concept of priority is ineffective. From the perspective of the CSP, unlike the enterprise storage admin they aren't afforded the luxury of application level visibility. Consequently, it does little good to assign a priority level to a set of applications that they have no control over. From the customer perspective, priority is a relative ranking lacking any real clarity on absolute performance. If a customer has a priority of 10 and everyone else is at 5, they may have twice the priority, but it will come at the expense of all the other tenants on that system. Moreover, even if performance is good, there is no guarantee it will stay that way. While the priority level may be controlled, the performance delivered to a particular level is still "best effort" in the context of all the other workloads on the system. This creates an unpredictable environment for both cloud service providers, and more importantly, their customers.

At SolidFire, one of our founding premises was that solving the performance challenges for cloud service providers required a completely different approach to Quality of Service. SolidFire has architected hard QoS controls into the system that are defined in terms that actually mean something to a customer, IOPS and MB/s. Each volume is configured with minimum, maximum, and burst IOPS and bandwidth. The minimum IOPS provides a guarantee for performance, independent of what other applications or tenants on the system are doing. The maximum and burst controls the allocation of performance and delivers consistent performance to tenants. For the cloud provider, SolidFire QoS enables SLAs around exact performance metrics and complete control over the customer's experience. For cloud consumers, clear expectations around storage performance provide confidence and stability. With guaranteed performance, IT administrators can finally deploy their tier 1 applications with confidence in the cloud.

-Adam Carter, Director of Product Management

Server-Side Caching: A complex stopgap making up for deficiencies in current array designs

Wed, 27 Jul 2011 00:00:00 GMT

Chris Mellor from The Register recently wrote an informative article ("Why should storage arrays manage server flash?") that outlines the merits of server side caching. You can read the entire article here. At SolidFire, we take a slightly different viewpoint on the subject and wanted to take the opportunity expand on it some more here.

There are certainly advantages to server-side SSD caching. Most notably, it reduces load on storage arrays that are being taxed far beyond what they were originally designed for. However, in the long run I think we'll see server-side SSD caching as nothing but a complex stopgap making up for deficiencies in current array designs.

If you look at "why" it's claimed server-side cache is necessary, it basically boils down to:

-The array can't handle all the IO load from the servers, particularly when flash is used with advanced features like dedupe

-The reduction in latency from a local flash cache

The first is a clear indication that current array designs aren't going to scale to cloud-workloads and all (or mostly all) solid state storage levels of performance. Scale-out architectures are going to be required to deliver the controller performance needed to really benefit from flash.

The second is based on the assumption that the network or network stack itself is responsible for the 5-10ms of latency that he's reporting. The reality is that a 10G or FC storage network and network stack will introduce well under 1ms of latency - the bulk of the latency is coming from the controller and the media. Fix the controller issues and put in all-SSD media, and suddenly network storage doesn't seem so "slow". Architectures designed for SSD like TMS, Violin, and SolidFire have proven this. Local flash, particularly PCI-attached, will still be lower latency, but that micro-second performance is really only needed for a small number of applications.

EMC and Netapp have huge investments in their current architectures, and are going to try every trick they can to keep them relevant as flash becomes more and more dominant in primary storage, but eventually architectures designed for flash from the start will win out.

-Dave Wright, Founder & CEO

The best automated tiering is no tiering at all

Wed, 13 Jul 2011 00:00:00 GMT

Recently Hitachi and EMC have gotten in a blog fight about whose automated tiering technology is better. But are they asking the wrong question? Is tiering even the right solution to storage performance problems to begin with?

To be sure, on the surface the concept of automated tiering sounds great - your hot data goes in fast SSD storage, while seldom accessed data is on cheap spinning disk. The problem with this type of optimization is that it is executed from a global perspective across the entire storage system. The storage array optimizes the IO load across all of the data it controls. If you only have a single application, or perhaps a small number of applications running on your storage array, this global optimization probably works pretty well.

That's a good fit for a traditional enterprise SAN deployment, but consider a cloud environment where you have hundreds or thousands of virtual machines and applications, with new applications coming and going all the time. From the perspective of an individual application, storage performance can be radically unpredictable. One day, the array may decide the application's data is "hot" and serve it out of SSD with < 1ms response time, the next day another hot application may come online and suddenly response times jump to 10ms as the data gets pushed out to a SAS or SATA tier. This type of unpredictability is especially problematic in a multi-tenant service provider environment, where customers aren't aware of each other and any radical change in performance is likely to trigger a support call. It's not the storage array's fault - it's still trying to globally optimize, but try telling that to a customer whose website or database is suddenly slow.

So if tiering isn't the answer, what is? Simply putting all the data on SSDs helps, but doesn't take advantage of the fact that some applications and data are actually more active than others, and there is a hesitance to "waste" SSD performance on less active data. At SolidFire, we think the answer to these issues is performance virtualization. SolidFire's unique performance virtualization technology de-couples capacity from performance and allows service providers and their customers to dial in the exact performance required for each application. Have a lot of data that doesn't need much performance? No problem, those IOPS aren't wasted - they're simply available for other more demanding applications on the storage cluster. Either way, you get the performance you expect day after day, without any surprises, and if you need more or less, you can change it instantly. Skip the data tiering stopgap and get an all solid-state solution that can optimize performance across thousands of applications.

-Dave Wright, Founder & CEO

SolidFire Launch at Gigaom Structure

Tue, 05 Jul 2011 00:00:00 GMT

If you have been following us here at SolidFire over the last few months, you know that we officially came out of stealth-mode at the Gigaom Structure conference in San Francisco in June. Structure provided us a great launch-pad to begin talking publicly about our company, our technology, and the availability of our early access program.

You can watch a playback our Founder and CEO, Dave Wright, participating on a panel titled Cloud Storage: Moving Beyond Backup.

We have had fantastic press and analyst coverage over the past couple of weeks; numerous editors, bloggers, and analysts talked through their articles, blog posts, or tweets about the SolidFire solution, and you can find it all in the News section of our website. We also refreshed our website to enable customers to begin to dive deeply into our technology and understand the impact that a SolidFire solution can have on their primary storage portfolio. Make sure you take a few minutes to check our videos to learn about the benefits of SolidFire performance, efficiency and automated management each in 90 seconds or less.

As a company, we are growing rapidly here in Boulder, Colorado, and it is great to be part of Boulder's thriving storage and start-up community. We have new people joining the team every week, and new customers joining our early access program almost daily. It is an exciting time for us at SolidFire, as well as for our early access customers who are now able to offer scalable primary block storage to thousands of servers.

For Dave, Adam, and I, Structure was a great opportunity to finally let loose our enthusiasm for the work that we are doing at SolidFire. When you are passionate about what you do, stealth-mode can be a drag, and all of us are glad to be getting on with the business of helping cloud providers enable high-performance and high-efficiency primary block storage within the cloud.

To stay in formed on what is new with SolidFire and to keep up with our growing community, sign up on our mailing list or follow us on Twitter @solidfire.

-Jay Prassl, VP of Sales & Marketing

Time for Solutions

Mon, 20 Jun 2011 00:00:00 GMT

Over the past few weeks, we've talked about the challenges of providing high-performance primary storage at cloud scale, challenges like performance, efficiency, and management. Today, I'm excited to stop talking about problems, and start talking about SolidFire's solutions.

This week SolidFire is taking the wraps off our all-SSD storage technology and announcing our early access program. On our new website you'll find our product described in detail for the first time, including the SolidFire Element™ operating system and the SF3010™ storage node.

After reading more about SolidFire, you may still wonder:

Is it really possible to build an all-SSD storage system that scales to petabytes of capacity and millions of IOPS?
Can that performance and scalability be achieved at a price per gigabyte similar to disk?
Can you really guarantee IOPS to thousands of individual volumes and put an SLA around storage performance?
Can efficiency technologies like deduplication and compression actually be built to run in real-time without affecting performance?

We certainly think so, having built the first storage system that can do all this and more. If you'd like to find out how, come talk with us and see what SolidFire can do for you.

-Dave Wright, Founder & CEO

The challenges of primary storage at cloud scale

Tue, 14 Jun 2011 00:00:00 GMT

Founder and CEO Dave Wright explains why traditional storage just doesn't work at cloud scale.

Challenges of Block Storage as a Service at Cloud Scale Part 3 - Management

Mon, 13 Jun 2011 00:00:00 GMT

So far in our series of blog posts on the challenges of deploying Block Storage as a Service, we have discussed why it's difficult to get both good performance and high efficiency with primary storage in the cloud. Because of these issues, what service providers are often left with is a sprawling, underutilized storage infrastructure that is extremely difficult to manage at scale.

Where does this management challenge come from? It's primarily caused by a disconnect between how traditional enterprise storage systems have been managed and how cloud providers want to build and manage their infrastructure. In the enterprise, expensive and complex storage equipment is looked after by experienced storage administrators. Given the cost of the equipment and the value of the data being stored, having a well-trained human configuring and managing the storage on a daily basis makes a lot of sense. Traditional storage companies have built their management systems around the demands of storage administrators, and, as a result, have created complex and feature-filled administration tools.

The problem is that this model doesn't scale. For a large-scale cloud, where you are growing quickly and deploying new storage on a weekly or even daily basis, and adding customers 24 hours a day, hiring an army of storage administrators to setup, configure, provision, manage, and troubleshoot that storage is not a viable option. The efficiencies of the cloud are not based on armies of administrators; they are based on efficient management through automation. Service providers don't want to administer their storage; they want to automate it.

An illustration I like to use is to compare deployment of compute capacity to storage. Most cloud providers are extremely efficient at deploying new compute capacity. Automated server configuration, deployment, and management tools allow new racks of servers to be plugged into the network and immediately added to the pool of available compute capacity. All of these activities are accomplished without an administrator ever logging in or configuring a single thing. How can you do that with storage today? Setting up new storage arrays is a complex and time-consuming process. Provisioning new storage or adding capacity is something that has to be done carefully to avoid disruption and ensure security and data isolation are preserved. Automated alerting and reporting are primarily done through proprietary vendor tools or complex integrations. Any automation capabilities or APIs tend to be afterthoughts and cover only a small portion of the system's functionality.

What service providers are really looking for is a storage system that was designed with automation in mind from the start, with APIs that are both comprehensive yet easy to integrate, and with management capabilities that can be consumed by a machine just as easily as they can by a human. Only then is storage really ready for cloud scale.

Performance, efficiency, and management are just three of the challenges facing cloud providers who want to deploy primary block storage at scale. SolidFire was built from the ground up to address these challenges and many others. Soon, we will be telling you just how we do that. I can't wait!

-Dave Wright, Founder & CEO

Challenges of Block Storage as a Service at Cloud Scale Part 2 - Efficiency

Fri, 10 Jun 2011 00:00:00 GMT

In Part 1 of our series on the challenges service providers face delivering block storage as a service, we discussed the challenge of performance. Today, we're going to talk about a second challenge service providers face - storage efficiency. The ratio between how much storage capacity a service provider buys, and how much they are able to sell is a critical driver of bottom line profitability - yet obtaining high utilization rates - is a constant struggle.

Part of the reason for this inefficiency goes back to our discussion of the imbalance between storage capacity and performance. In an effort to provide as consistent performance as possible, it is common place that service providers are forced to deploy far more capacity (spindles) than they are able to use in order to provide the right number of IOPS. All that wasted capacity continues to consume space, power, and cooling, and drags down the profitability of the capacity that is sold.

Another challenge to high utilization rates is how service providers plan for growth and deploy new capacity. While most storage systems are designed to allow for capacity expansion through disk shelves, in practice, many service providers deploy storage "fully configured" from day one. Reasons for deploying full storage configurations include; better pricing from the vendor, reducing the risk and complexity of adding new capacity while operating, and the fact that much of the cost of the storage system is in the controller and software which need to be purchased up front. Whatever the reason, the result is the same - low utilization rates during early deployment, which brings down overall utilization and reduces profitability.

Over the past few years, efficiency technologies that allow you to store more data in less space, like compression and deduplication, have started to appear in primary storage systems. While on the surface these features should be a huge boon to service providers looking to increase their efficiency, in reality they are seldom used. Again, it comes down to the balance between performance and capacity. These efficiency features often incur a significant performance penalty while providing space that can't actually be used. In fact, many service providers don't even use thin provisioning, a storage feature that has been standard for years. Why? Because it makes capacity planning more difficult, and they get better performance by fat provisioning volumes up front.

What service providers really want is storage that is designed and balanced to run at consistently high utilization rates, and can be grown incrementally over time, so that it can be profitable from day one.

-Dave Wright, Founder & CEO

Challenges of Block Storage as a Service at Cloud Scale Part 1 - Performance

Mon, 06 Jun 2011 00:00:00 GMT

For service providers who want to offer Block Storage as a Service as part of their cloud compute offering, a number of challenges exist. At SolidFire, we're focused on solving the biggest problems that the service providers encounter when trying to build scalable, reliable, and profitable network-based primary storage. In this first of a three part blog series discussing these problems we will address the challenge of performance.

Over the past 20 years, a huge performance imbalance has been created between processing power, which has doubled every 18-24 months under Moore's law, and storage performance, which has barely improved at all due to the physical limitations of spinning disk. Meanwhile, storage capacity has exploded by a factor of 10,000 over that time. The result is that while capacity is plentiful and cheap, storage performance (measured in IOPS) is expensive.

For a service provider looking to sell block-based primary storage as a service, that imbalance makes it difficult to sell storage on a per-gigabyte model, which is how it is most commonly sold today. Customers who may buy only 50 or 60 GB of space for their application still expect reasonable performance - but when that customer is put on the same set of disks with dozens of others, their "fair share" of IOPS doesn't amount to very much. Even worse, performance will vary considerably based on how many other apps are on the same disk and how active those apps are at any given time. The result is poor, unpredictable performance, and unhappy customers. Today, service providers offering Block Storage via enterprise storage arrays typically deal with this challenge by using lots of fast, expensive FC and SAS disk, and utilizing only a fraction of the available capacity (a technique known as under-provisioning or short-stroking). Even with this approach, it's difficult or impossible for providers to guarantee customers any particular level of performance, short of putting them on their own dedicated disks and eliminating much of the benefit of an efficient multi-tenant cloud.

So what about flash? Doesn't it solve the performance problem? Today that's true only in part. As we previously discussed, most enterprise storage today makes only limited use of flash as a cache or tier of storage for hot data, and overall array performance is often limited by the controller. While cache and tiering technology does a good job of "globally optimizing" array performance by putting the hottest data in low latency flash, it can actually end up causing more headaches for service providers by making storage performance even more unpredictable for customers. From the perspective of an individual customer, their data may be blazing fast one minute as it is served from flash, and the next minute slow as a dog as it got bumped to disk, because another customer was "hotter." At this point, expect a support call. Inconsistent, unexplainable performance is one of the biggest complaints about block storage in the cloud today, and automated tiering and cache just makes it worse. All service providers want is an endless amount of storage performance that can be carved up and sold in predictable, profitable chunks. Is that too much to ask?

-Dave Wright, Founder & CEO

Just How Expensive is Flash?

Thu, 26 May 2011 00:00:00 GMT

For most people there are two common associations with SSDs: expense and performance. The performance side is hard to argue - SSDs can be 100X faster than spinning disk on many workloads. But what about cost?

By historical standards, solid state storage is now amazingly inexpensive. The myth that flash is expensive is propagated by enterprise storage vendors selling flash modules for $25-$50/GB or more. The performance benefits of flash allows customers to justify the ridiculous price, but also limits their use of flash to only the most critical, most performance sensitive applications.

The reality of the situation is that flash chip prices have been on a dramatic decline over the last 5 years, dropping in price by 50% or more a year as demand increases and process sizes shrink. Spot prices for MLC flash are now around $1-$1.25/GB, and high-capacity MLC SSDs can now be had for under $2/GB. Of course, most enterprise storage vendors aren't using MLC. The limitations of their architecture often don't allow it. However an architecture designed from the ground up around SSDs, balancing use of SLC and MLC technologies, is a different story.

In comparison to Enterprise SATA drives which sell for $0.15/GB, $1/GB for flash may still seem expensive. For applications where capacity is the only concern, that's certainly true, and will continue to be the case for many years. However for primary storage applications, those that require even a modest number of IOPS, a comparison to SATA drives does not make much sense. A better comparison with flash would be 10K & 15K SAS drives, or even complex tiered solutions that use SSD, SAS, and SATA. With 15K SAS drives at $1/GB or more, flash is not far from closing the gap.

For SolidFire however, the really exciting part is what happens when you combine the falling price of solid state storage with efficiency technology that dramatically increases effective capacity and requires a fraction of the power, cooling, and space of spinning disk. Suddenly spinning rust doesn't look so cheap after all, does it?

-Dave Wright, Founder & CEO

SANs don't do justice to SSD

Tue, 10 May 2011 00:00:00 GMT

So what would happen if you took the HDDs in your SAN and replaced them with latest SSD drives? Don't faster disks = faster storage technology? Unfortunately it is not that straight forward. Traditional SAN architectures dramatically complicate the use of SSD because both the hardware and software were designed around spinning storage media - not SSDs.

Today there is an ever-widening gap between compute and storage IO. Large multi-core servers packed with memory are capable of delivering a high number of IOPS to extremely fast networks, and traditional storage systems have languished with high latency and poor IO. Compute technology has been outpacing SAN and disk performance for years. At this point traditional SANs are engulfing more than their fair share of IT budget trying to keep up. So why doesn't the use of SSDs as HDD replacements fix this problem alone? The answer lies within storage controllers and the storage operating system. Within traditional storage architectures these aged components do more harm than good to SSDs, and are unable to take advantage of their benefits.

Controller IO Bottleneck
Traditional storage controllers were designed to manage thousands to tens of thousands of IOPS, not the hundreds of thousands to millions of IOPS that SSDs are capable of delivering. Current controllers simply can't keep up.

Traditional SAN architectures are not designed to maintain the integrity of SSDs
Data layout architectures that optimize for deficiencies in spindle physics are ineffective with SSDs. Write patterns and redundancy mechanisms such as RAID cause write amplifications that put unnecessary loads on SSDs. These algorithms accelerate the wear of SSD media and have lent to the myth that SSD are inferior to HDDs and wear quickly. So for the record, it is legacy storage architectures and how they manage SSDs that limits SSD use and life cycle. Today's SSD duty cycles can be on par with HDD and getting better.

Limited Deployment of SSD
Predominant use of SSDs within traditional SANs are as either cache or a small storage tier. SSDs used in these modes receive tremendous write traffic and churn which places tremendous wear on the drives. To compensate most manufacturers require the exclusive use of the most expensive and wear resistant SSDs which drives up solution cost. Think of the cost and wear implications if you deployed SSD across an entire legacy SAN architecture... not a pretty picture.

The solution to leveraging SSDs in an intelligent and and cost effective manner is a new storage architecture. An architecture built from the ground up around SSD technology that sizes cache, bandwidth, and processing power to match the IOPS that SSDs provide while extending their endurance. It requires an architecture designed to take advantage of SSDs unique properties in a way that makes a scalable ALL SSD storage solution cost effective - today.

-Adam Carter, Director of Product Management

Welcome to SolidFire

Mon, 09 May 2011 00:00:00 GMT

Primary storage for cloud computing requires a new solution.
That simple truth is one of the biggest lessons I learned from my time at Rackspace.

Cloud computing amplifies many of the shortcomings of today's enterprise storage systems. Demands around scalability, performance, efficiency, availability, and automation increase dramatically when you move from an enterprise environment supporting a few dozen applications to a cloud that is the backbone of thousands.

Since it's inception, SolidFire has been focused on solving one critical problem: How to provide high-performance primary storage to thousands of applications in a cloud computing environment. Today we are starting to take the covers off the amazing technology we've built to address this challenge. Technology that has the potential to not just revolutionize the cloud computing world, but the entire landscape of primary storage.

Behind the technology is an equally amazing team composed of storage industry veterans from LeftHand Networks, IBM, and HP, distributed computing wizards from Cornell and Georgia Tech, and industry experience in virtualization and cloud computing from VMWare and Rackspace.
We're fortunate to be backed by a team of investors from Valhalla, Novak Biddle, and NEA with both the long term vision and deep pockets necessary to help us build a world class company.
Over the coming weeks and months, I look forward to sharing more of our vision, our team, our product, and the groundbreaking technology that makes it all possible.

-Dave Wright, Founder & CEO