Joerg's IT Blog

Ae we living in a cloud computing bubble?

2016-08-25T17:32:00.000-07:00

Talking with our customers, I keep hearing the same thing "IT management has decided on a Cloud First strategy".

Are we working in a cloud computing bubble? That is, have business leaders (and perhaps some IT leaders) been lulled into assuming that moving or deploying to cloud is as easy as 1-2-3, without any heavy lifting required? So, have at it, spend the money, get cloud for cloud’s sake?

Listening to the cloud vendor messages, one can be forgiven for thinking that cloud deployments are a snap, and will quickly put a business on the path to digital nirvana. However, the history of enterprise software tells a different story. There have been many cases in recent decades of companies slapping expensive technology solutions on top of calcified processes and even more calcified business models, and expecting overnight success — but getting none.

Technology is essential to keep up, and the faster an organization can move to digital, the better. But like the tires on a race car, technology only makes the ride smoother, but is not the reason for success. A scan of the Forbes Global 2000 List of the World’s Largest Public Companies between 2006 and 2016 shows other forces at work that determine business success. The top companies in 2006 were Citigroup, GE, Bank of America, AIG, and HSBC — four US-based, the fifth in the UK. The top companies this year are ICBC, China Construction Bank, Agricultural Bank of China, Berkshire Hathaway, and JPMorgan Chase — the top three based in China, the next two in the United States. The point here is that no amount of advanced technology would have necessarily helped the 2006 leaders hold their leads, as they were knocked off their perches by players that emerged from other parts of the global economy with different models and markets.

Successful organizations understand that underlying corporate culture and business models, led by forward-thinking managers or leaders who nurture a spirit of innovation among all levels of employees, are what matter in the long run. They also understand that technology supports this in a big way, but cannot replace any deficiencies.

This is a point to keep in mind when considering the fact that enterprises may be investing hundreds of thousands or millions of dollars, euros, pounds or rupees in cloud computing solutions every year, yet are still uncertain about how and where this technology is best applied. IDC recently released estimates that up to $96.5 billion will be spent on cloud services worldwide this year alone, a number that will reach $195 billion by 2020. Gartner adds a few billion to the equation, suggesting that the worldwide public cloud services market is forecast to reach $204 billion this year alone.

Is this money being well spent? A recent survey of more than 400 enterprise IT executives, conducted by Wakefield Research and sponsored by LogicWorks, confirms there is general uncertainty of IT and business leaders on how to best leverage the cloud to drive growth and efficiency across their organizations, as well as the need for more thoughtful planning for both cloud migration and ongoing cloud maintenance. Eight in 10 executives believe that their company’s leadership team underestimates the time and cost required to maintain resources in the cloud.

There are issues with staffing for the new cloud and digital realities as well. The need for technically proficient people does not go away when things are shifted to the cloud. Even as the demand for enterprise-level, cloud-based services expands, the LogicWorks survey finds 43 percent believe their organization’s IT workforce is not completely prepared to address the challenges of managing their cloud resources over the next five years. It is a problem compounded by the high demand and relatively low supply for workers skilled in cloud, security, DevOps engineering and other IT positions.

Organizational preparation needs to be the most essential piece of cloud and digital engagements, and according to the LogicWorks survey, not enough is being done. In a compelling read at TechTarget, Mark Tonsetic, IT practice leader at CEB, outlines four ways to tell the “cloud story,” to ensure that the money and time spent on technology is met by appropriate transformation of the business:

Cloud and digital transformation are one in the same. Cloud may offering a compelling cost savings, but this is only one small piece of its value proposition, Tonsetic says. Information technology should be seen for what it is becoming: “an ingredient in corporate growth.” He urges cloud proponents to “stress speed and flexibility gains that can enable enterprise digital strategy,” and elevate this discussion to the board and investor level. Enterprise digitization and growth is today’s corporate holy grail.

Cloud and digital elevates the roles of IT leaders. IT leaders need to shift their focus from governance and policy to advice and education on the technology options available to their businesses. IT leaders need to serve as partners to their business users, making the “The cloud story that IT leaders need to tell their business partners should be about how IT will build new partnerships with the business to explore and exploit cloud opportunities. Relay the case for cloud computing in the language of new business opportunities, new models for business engagement and collaboration, and new opportunities for technology careers.”

Cloud and digital transform IT departments into competitive service brokers. Corporate IT no longer needs to operate as the owners and operators of email, CRM or even ERP systems, Tonsetic relates. These departments are now in a different kind of business — consulting and working with corporate leaders to define and execute transformational digital strategies.

Cloud and digital means new types of career opportunities. As mentioned in the LogicWorks survey, more than four in 10 executives say they don’t have the available skills to move forward with cloud. At the same time,there are highly capable IT staff members still involved in legacy or on-premises systems development, maintenance or operations. This pool of talent shouldn’t go to waste. “Forward-thinking IT organizations are careful to send teams the message that cloud and other technology developments present an opportunity for experimentation, innovation, and growth across IT — a refreshing change of pace for teams that have labored mostly to the tune of efficiency, Tonsetic relates.

Modernizing backups, or as I like to call it, data protection

2016-08-09T17:51:00.000-07:00

Nearly every enterprise IT organization out there is experiencing a significant increase in customer expectations around application performance and business continuity. Response time once measured in seconds are now measured in milliseconds, and downtime measured in hours or days is now expected to be minutes, if that.

To keep pace with all of that change in expectations, many organizations are implementing application modernization programs that allow them to take advantage of new technologies such as cloud enabled applications, etc. These changes are also driving changes in the infrastructure that those new/modern application run on. Modern network technologies, IaaS, even more virtualization, and containers are all being driven deep within the modern datacenter by these needs.

What hasn't been keeping up, however, are backups, or as I prefer to call it, data protection. The old full's and incremental backups to some medium such as tape, or even disk, just isn't getting it done anymore. The entire concept of backups/data protection is being replaced with Business Continuance. The focus has shifted from a siloed attempt to protect the businesses data, to ensuring that the business can continue to run.

With organizations considering availability, it’s no longer about simply needing to restore a single file or folder. It’s about complex processes like recovering a multi-tiered application that spans multiple servers and bringing each server and the data it requires into a consistent state with the others. It’s far more involved than the restore jobs of yesteryear.

In some ways, basic backup no longer has a place in the modern data center. As new technologies have come into use—the foremost being virtualization—you can now easily move workloads from one location to another. You can even performing maintenance during the day. The options around availability are much greater and more flexible than what backups alone provide.

Even so, the idea of a backup—that is, having copies of your data—is still viable. Now data centers have moved to advanced concepts like replicating data at the disk level or entire virtual machines, both from one store to another or even one site to another. This provides both increased protection and faster recoverability.

Organizations today aren’t just looking at availability on a per-application or per-server basis. The goal is to make everything available in the event of an outage. So it’s not “we have our order processing back online, but e-mail is still down.” Now it’s essential to have the entire business back up and running, not just a few services.

Should you have an availability event, can you benefit from backup alone? Backups certainly still have a place. For example, if you’re replicating changes to a VM and the source VM is somehow corrupted, that corruption will simply get replicated. So having a backup of the critical data on that server can play a role in ensuring recoverability. However, backup as the only method is no longer an option for businesses focused on being available.

As newer technologies have emerged, the frequency of backups has also shortened. In previous years, your backup window simply couldn’t be anything less than nightly. These days, backups occur

much more frequently—even during production hours. And with technology like instant VM recovery in place, the concept of restoring a backup job is somewhat obsolete.

If your organization is like most, you’ve already begun or have made the investment in a modern data center. Despite your desire to simplify what you manage, it’s a complex mix of virtual machines, servers, storage and networking. Because you’ve made the investment to meet the demand to maintain operations, traditional backups alone just won’t scale to meet the availability needs of the organization in such an advanced data center environment.

Your organization must have standards for what is and is not acceptable downtime. Comparing the businesses’ required levels of availability against what’s currently attainable can help you create a service baseline from which to work towards availability. Begin with the business requirements around application and environment availability, instead of what your backups can do today. This will help IT look for ways to cost-effectively take advantage of current technologies or invest in new ones to make meeting availability requirements a reality.

Being a good listener is one of the key's to success, in any business

2015-12-15T11:40:00.000-08:00

I'm in the sales business, and in my business learning to listen well is a major key to success. Actually hearing what people are saying, and not just waiting for your turn to speak is critical if you want to learn what it is your customer is looking for in a solution. Knowing what your customer needs, allows you provide a solution that the customer will embrace, and increase your chances of a sale.

Unfortunately, all to often I've been in meetings where there is at least one person who is more interested in formulating what they are going to say next than they are in hearing what the person who's currently speaking has to say. In my business (IT) I find this to be endemic. I suspect that has a lot to do with the passion with which many IT professionals bring to their jobs. That passion is a good thing, but it can also be a two edged sword. Not hearing and absorbing what the current speaker has to say means that they often come across as "preachy" or "professorial" and miss important points of view and information.

So, I would encourage everyone to become a better listener. But being a good listener does not come easy for some of us. It takes time, practice and dedication. What comes to your mind when you think about listening to a friend or co-worker? Do you find yourself thinking about what you want to say in response to what they have said or are you fully engaged with what they are talking about? When it comes to connecting with others, it’s all about consciously listening to them and the information that they are sharing with you.

1. Eye contact
When it comes to being a good listener, it’s important for you to have eye contact with the other person. It shows that you are paying attention and engaged with the conversation. When you don’t have eye contact with the other person, it shows that you don’t care and are not interested in what they have to say. Practice having eye contact with the next person you have a conversation with.

2. Find the “Why” and “What”
For you to be a good listener, you need to find out the “Why” and “What.” Why are they talking to you and what is the message they are trying to share with you? Being a good listener takes practice and when you are able to practice finding out the “Why and “What” of the other person, you will be much more engaged in the conversation.

3. Focus on the other person
It’s easy for us to think about what we want to say after the other person has stopped talking. This will not make you a better listener. If you are constantly thinking about your response, you will always miss out on carefully listening to the other person. Focus on what they have to say. Find out the “Why” and “What” and maintain eye contact. Once the other person stops talking, then think about your response. But while you are listening, you must be consciously listening with your ears. A lot of times, when we listen to people, we are thinking within our brain what we want to say rather than opening our ears and purely listening to their message.

4. Limit distractions
We live in a society that is filled with so many distractions. We are constantly listening to so much noise that it’s a challenge to truly listen to another person. In order for you to be a good listener, you need to limit distractions during your conversation, whether it be the television, telephones or interruptions. It takes a mental decision to limit distractions when you are listening to someone else. How can you possibly be a good listener if you have the television blasting or you phone continues to ring? It would be near to impossible to be a good listener with these distractions. Limit as much interruptions as you can when you are listening to someone else. This not only shows them that you care but you are practicing good social skills.

5. Engage
Engage yourself in the conversation. Being engaged is showing your attention towards the other person. Let the other person know that they have your attention and focus. When you are not engaged in the conversation, the other person will notice and will most likely not want to talk to you again. Show the other person that you care about them and are interested in what they have to say. One way you can show this is by responding with a short comment, such as “Yes” or “I understand.” This expresses to the other person that you are truly listening. Make sure that you allow the other person to primarily do the talking while you are still engaged.

I believe that if you become a better listener, you'll be more successful both in business and in your personal relationships.

2015-12-15T10:52:00.000-08:00

Folks,

Please note that I've changed the name of my blog. The name change reflects the change that I, the company I work for, and the industry is making. Converged/hyperconverged infrastructure, the cloud, Openstack, DEVOPS, etc. are all changing the IT landscape and if you aren't changing with it, then you're going to get left behind.

So, I'm changing the name and focus of this blog since, hey, none of want to be called a dinosaur, do we?

--Joerg

Container Wars!

2015-05-12T17:25:00.000-07:00

The container wars have started!

Containers have a huge amount of hype and momentum, and there are many spoils for whoever becomes dominant in the container ecosystem. The two major startups innovating in this space–CoreOS and Docker–have declared war on each other as part of gaining that control.

The Current Landscape

Recently, CoreOS announced Tectonic. Tectonic is a full solution for running containers, including CoreOS as the host OS, Docker or rkt as the container format, and Kubernetes for managing the cluster. It also uses a number of other CoreOS tools, such as etcd and fleet.

Despite Docker being an option on Tectonic, CoreOS’s messaging is not focused on Docker, and neither the Tectonic site nor the announcement included a single mention of Docker. It’s clear that CoreOS is moving in a different direction. CoreOS’ CEO Alex Polvi says that “Docker Platform will be a choice for companies that want vSphere for containers”, but that Rocket is the choice for “enterprises that already have an existing environment and want to add containers to it”. The latter is a far larger prize.

Companies will choose Docker Platform as an alternative to things like Cloud Foundry. Companies like Cloud Foundry will use things like Rocket to build Cloud Foundry.

Docker meanwhile, doesn’t mention CoreOS or Kubernetes anywhere in their docs, and on their site only mentions them in passing. Docker CEO Solomon Hykes reacted fairly negatively to the announcement of rkt back in December. He has also started to use the phrase “docker-native” to differentiate tools that Docker Inc. builds from other tools in the ecosystem, indicating other tools are second class.

Right now, both companies provide different pieces with their respective stacks and platforms.

To run containers successfully on bare metal server infrastructure, you need:

A Linux host OS (Windows support for containers is coming with the next release of Windows).
The container runtime system to start, stop, and monitor the containers running on a host.
Some sort of orchestration to manage all those containers.

Tectonic provides all of these, with CoreOS, Docker or rkt, and Kubernetes. However, it appears to omit a pre-defined way to build a container image, the equivalent of the Dockerfile. This is by design, and there are many ways to build images (Chef, Puppet, Bash, etc) that can be leveraged.

On Docker, things are less clear. Docker isn’t opinionated on the host OS, but also doesn’t provide much help there. Docker Machine abstracts over it for some infrastructure services, where you use whatever host OS exists already, but doesn’t provide much help when you want to run the whole thing. Docker Swarm and Docker Machine provide some parts of orchestration. There is also Docker compose, which Docker has been recommending as part of this puzzle, but simultaneously saying it’s not intended to be used as part of production. Of course they have a Dockerfile to build the containers, though some indicate that this is immature for large teams.

The impression you get from Docker is that they want to own the entire stack. If you visit the Docker site, you could be forgiven for thinking that the only tools you use with Docker are Docker Inc’s tooling. However, Docker doesn’t really have good solutions for the host OS and orchestration components at present.

Similarly, Docker are pushing their own tools instead of alternatives, even when those tools aren’t really valid alternatives. Docker Compose, for example, is being pushed as an orchestration framework though this feature is still in the roadmap.

The container landscape is fairly new, but Docker has a pretty clear lead in terms of mindshare. Both companies are trying to control the conversation: Docker talking about Docker-native and generally focussing it’s marketing around the term Docker, while others in the space – CoreOS and Google for example – are focussing the conversation on “containers” rather than “Docker”.

This is made a little bit difficult by the head start that Docker has – they basically created the excitement around containers, and most people in the ecosystem talk about Docker rather than the container space. Docker has also done an incredibly good job of making docker easy to use and to try out.

By contrast, CoreOS and Kubernetes are not tools for beginners. You don’t really need them until you get code in production and suffer from the problems they solve, while Docker is something you can play around with locally. Docker’s ease of use, everything from the command line to the well thought out docs to boot2docker, are also well ahead of rkt and and CoreOS’s offering – which are much harder to get started with.

How does this play out?

If you’re a consumer in this space, looking to deploy production containers soon, this isn’t a particularly helpful war. The ecosystem is very young, people shipping containers in production are few and far between, and a little bit of maturing of the components would have been useful before the war emerged. We are going to end up with a multitude of different ways to do things, and it’s clear we’re far from having one true way.

From a business perspective, it’s difficult for any of the players to capitulate on their directions. Docker is certainly focusing on building the Docker ecosystem, to the exclusion of everyone else. Unfortunately, they don’t have all the pieces yet.

Other companies who want to play in the ecosystem are unlikely to be pleased by Docker’s positioning. CoreOS certainly isn’t alone in their desire for a more open ecosystem.

Ironically, Docker came about due to a closed ecosystem with a single dominant player. Heroku dominates the PaaS ecosystem to the extent that there really isn’t a PaaS ecosystem, just Heroku. Dotcloud failed to make inroads, and so opened its platform up to disrupt Heroku’s position and move things in a different direction such that Heroku’s dominance didn’t matter. In Docker, they certainly appear to have succeeded with that. Now that Docker is the dominant player is the disruptive ecosystem, CoreOS and other players will want to unseat them and fight on a level playing field before things settle too much.

The risk for Docker is that on this trajectory, if they lose the war they risk losing everything. If nobody else can play in this space, all of the companies that are left outside will build their own ecosystem that leaves Docker on the outside. Given that Docker lacks considerable parts of the ecosystem (mature orchestration being an obvious one), their attempts at owning the ecosystem are unlikely to succeed in the near term.

Meanwhile, CoreOS will need to replicate the approachability of the Docker toolset to compete effectively, and will need to do so before Docker solves the missing parts of its puzzle.

All of the other companies are sitting neutral right now. Google, MS, VMware are all avoiding committing one way or the other. Their motivations are typically clear, and it doesn’t benefit any of them to pick one or the other. The exception here is that the open ACI standard is likely to interest VMware at least, but I wouldn’t be surprised to see Google doing something in this space, too.

There is massive risk for all of the players in the ecosystem, depending on how this plays out. Existing players like Amazon, Google and Microsoft are providing differentiated services and tools around containers. The risk of not doing so, of owning no piece of the puzzle, is being sidelined and eventual commoditization. The one API that abstracts over the other tools is the one which wins.

Long story short – this is the start of a war that will probably be quite bloody, and that none of us is going to enjoy.

The Data Center of the Future, what does it look like?

2015-04-27T17:20:00.005-07:00

Folks,

I've been spending a lot of time talking with customers about storage, flash, HDDs, Hyper-converged, cloud, etc. lately. What's become clear to me recently, yes, I'm a little slow, is that all of these technology changes are driving us toward sea changes in the enterprise data center. In this blog posting, I want to talk a little about how things are changing in regards to storage. I'm going to talk a bit about Flash vs. HDD technology and where I see each of them going in the next few years, and I'll finish up with a discussion on how that will effect the enterprise data center going forward as well the the data center infrastructure industry in general.

I believe that the competition between flash and hard disk-based storage systems will continue to drive developments in both. Flash has the upper hand in performance and benefits from Moore's Law improvements in cost per bit, but has increasing limitations in lifecycle and reliability. Finding well-engineered solutions to these issues will define its progress. Hard disk storage, on the other hand, has cost and capacity on its side. Maintaining those advantages is the primary driver in its roadmap but I see limits to where that will take them.

Hard disk Drives (HDDs)
So, let's start with a discussion of HDDs. Hard disk developments continue to wring a mixture of increased capacity and either stable or increased performance at lower cost. For example, Seagate introduced a 6TB disk in early 2014 which finessed existing techniques, but subsequently announced an 8TB disk at the end of 2014 based on Shingled Magnetic Recording (SMR). This works by allowing tracks on the disk to overlap each other, eliminating the fallow area previously used to separate them. The greater density this allows is offset by the need to rewrite multiple tracks at once. This slows down some write operations, but for a 25 percent increase in capacity -- and with little need for expensive revamps in manufacturing techniques.

If SMR is commercially successful, then it will speed the adoption of another technique, Two-Dimensional Magnetic Recording (TDMR) signal processing. This becomes necessary when tracks are so thin and/or close together that the read head picks up noise and signals from adjacent tracks when trying to retrieve the wanted data. A number of techniques can solve this, including multiple heads that read portions of multiple tracks simultaneously to let the drive mathematically subtract inter-track interference signals.

A third major improvement in hard disk density is Heat-Assisted Magnetic Recording (HAMR). This uses drives with lasers strapped to their heads, heating up the track just before the data is recorded. This produces smaller, better-defined magnetized areas with less mutual interference. Seagate had promised HAMR drives this year, but now says that 2017 is more likely.

Meanwhile, Hitachi has improved capacity in its top-end drives by filling them with helium. The gas has a much lower viscosity than air, so platters can be packed closer together. This allows for greater density at the drive level.

All these techniques are becoming adopted as previous innovations -- perpendicular rather than longitudinal recording, for example, where bits are stacked up like biscuits in a packet instead of on a plate -- are running out of steam. By combining all of the above ideas, the hard disk industry expects to be able to produce around three or four years of continuous capacity growth while maintaining price differential with flash. However, it should be noted that all of the innovation in HDDs is around capacity. I believe that HDDs will continue to dominate the large capacity, archive type of workloads for the next 2 or 3 years. After that ... well, read the next section on flash.

Some argue that the cloud will be taking over this space. However, even if this is true, cloud providers will continue to need very cheap high capacity HDDs until flash is able to take over this high capacity space as well based on $$/GB.

Flash
Flash memory is changing rapidly, with many innovations moving from small-scale deployment into the mainstream. Companies such as Intel and Samsung are predicting major advances in 3D NAND, where the basic one-transistor-per-cell architecture of flash memory is stacked into three dimensional arrays within a chip.

Intel, in conjunction with its partner Micron, is predicting 48GB per die this year by combining 32-deep 3D NAND with multi-level cells (MLC) that double the storage per transistor. The company says this will create 1TB SSDs that will fit in mobile form factors and be much more competitive with consumer hard disk drives -- still around five times cheaper at that size -- and 10TB enterprise-class SSDs, by 2018. Moore's Law will continue to drive down the cost per TB of flash at the same time as these capacity increases occur this making flash a viable replacement for high capacity HDDs in the next 3 to 5 years. Note that this assumes that SSD's will leverage technology such as deduplication in order to help reduce the footprint of data and drive down cost.

The following is a chart from a Wikibon article on the future of flash:

As you can see from the graph above, by 2017 the 4 year cost per TB of flash will be well below that of HDDs, and that this trend will continue until 2020 when the 4 year cost per TB of flash hits $9 per TB vs $74 per TB for HDDs. You can read the entire article here.

Conclusions
So, what does all this mean? Among other things, it means that you can expect a shift to what the Wikibon article calls the "Electronic Data Center". The Electronic Data Center is simply a data center where the mechanical disk drive has been replaced by something like flash, thus eliminating the last of the mechanical devices (they assume tape and tape robots are already gone in their scenario). This will reduce the electricity and cooling needs, as well as the size/footprint of the data center of the future.

Let's assume for a moment that Wikibon is correct. What does this mean to the data center infrastructure industry?

Companies that build traditional storage arrays will need to shift their technology to "all flash", and they need to do it quickly. You can see this already happening in companies such as EMC with the acquisition of XtremIO in order to obtain all flash technology. Companies like NetApp, on the other hand, are developing their all flash solutions in house. In both those cases, however, all flash solutions are facing internal battles against engineering organizations that are vested in the status quo. That will mean they could be slow to market with potentially inferior products. However, their sheer size in the market may protect them from complete failure.
What about the raft of new startups producing all flash arrays? Might the above provide an opening for one or more of those startups to "go big" in the market? What about the rest? My take on this is that indeed, one or more might have the opportunity to "go big" due to the gap that might be created by the "big boys" moving too slowly or trying to shoe-horn old existing technologies into the data center. Most of them, however, will either die off, or be acquired buy a larger competitor.

However, I think that there is an even larger risk to the "storage only" companies both new and old. I believe that a couple of other market forces will put significant pressure on these "storage only" companies, including the new all flash startups.

Specifically, the trends towards cloud computing, Hyper-converged, and more and more emphasis on automation that is being driven by other IT trends such as DevOps will make standalone storage arrays less and less desirable to IT organizations. This will force those companies to move beyond their roots into hyper-converged infrastructure, for example where they currently have little or no engineering expertise or management experience.

The companies who are able to embrace these kinds of moves will likely have a bright future in the data center of the future. However, issues around "not invented here", and lack of engineering talent in the new areas of technology are going to make it a challenge for those very large storage companies going forward. Again, how they address these issues is going to be a determining factor in their future success.

To wrap it it up, I firmly believe that not everything is "moving to the public cloud" in the enterprise space. What I do believe is that:

Some workloads currently running in the enterprise data center will move to the public cloud, and be managed by IT.
Some workloads will remain in "private" clouds owned and operated by IT. However, those private clouds must offer at all of the same ease of use the internal customers that the public could offers. Most likely, they will leverage web-scale architectures (hyper-converged) in order to make management and management automation easier.
Hybrid cloud management software will be used to allow both management, and automation to span between the enterprises private cloud and it's public cloud(s).
DevOps and similar initiatives will drive significant automation into the hybrid clouds I describe above, as well as significant change to IT organizations.
these changes will all be highly disruptive, and those IT organizations that embrace change will have an easier time over the next few years than those that don't. Very large IT organizations will have the hardest time making the changes. Yes, it is hard to turn the aircraft carrier. However, internal customers are demanding it of IT, and will go outside the IT organization to get what the want/need if necessary.

In the end the Data Center of the Future will look very different than the current enterprise data center. It will be a hybrid cloud that spans on-premise, and public clouds. It will be an all electronic data center that uses significantly less footprint, and electricity than current data centers. And finally, this infrastructure will leverage significant automation and be managed by an IT organization that looks very different than the current IT organization.

Structured or Unstructured PaaS??

2015-04-22T15:23:00.001-07:00

Words, labels, tags, etc. in our industry mean something – at least for a while – and then marketing organizations tend to get involved and use words and labels and tags to best align to their specific agenda. For example, things like “webscale” or “cloud native apps” were strongly associated with the largest web companies (Google, Amazon, Twitter, Facebook, etc.). But over time, those terms got usurped by other groups in an effort to link their technologies to hot trends in the broader markets.

Another one that seems to be shifting is PaaS, or Platform as a Service. It’s sort of a funny acronym to say out loud, and people are starting to wonder about it’s future. But we’re not an industry that likes to stand still, so let’s move things around a little bit. Maybe PaaS is the wrong term, and it really should be “Platform”, since everything in IT should eventually be consumed as a service. I'm already hearing about XaaS (X as a Service) which pretty much means anything as a server, or perhaps everything as a service.

But not everyone believes that a Platform (or PaaS) should be an entirely structured model. There is lots of VC money being pumped into less structured models for delivering a platform, such as Mesosphere, CoreOS, Docker, Hashicorp, Kismatic, Cloud66, Apache Brooklyn (project) and Engine Yard acquiring OpDemand.

I’m not sure if “Structured PaaS” and “Unstructured PaaS” are really the right terms to use for this divergence of thinking about how to deliver a Platform, but they work for me. The Unstructured approach seems to appeal more to the DIY-focused start-ups, while Structured PaaS (eg. Cloud Foundry, OpenShift) seem to appeal more towards Enterprise markets that expect a lot more “structure” in terms of built-in governance, monitoring/logging, and infrastructure services (eg. load-balancing, higher-availability, etc.). The unstructured approach can be built in a variety of configurations, aka “batteries included but removable“, whereas the structured model will incorporate more out-of-the-box elements in a more closely configured model.

Given the inherent application portability that comes with either a container-centric model, or PaaS-centric, both of these are areas that IT professionals and developers should be taking a close look at, especially if they believe in a Hybrid Cloud model – whether that’s Private/Public or Public/Public. It’s also an area that will drive quite a bit of change around the associated operational tools, which are beginning to overlap with the native platform tools for deployment or config management (eg. CF BOSH or Dockerfiles or Vagrant).

It’s difficult to tell at this point which approach will likely gain greater market-share. The traditional money would tend to follow a more structured approach which aligns to Enterprise buying centers. But the unstructured IaaS approach by AWS has led it to a significant market-share lead for developers. Will unstructured history be any indication of the Platform market? Or will too many of those companies struggle to find viable financial models after taking all that VC capital and eventually just be a feature within a broader structured platform? I want to hear what you think, all respectful comments are welcome.

Some thoughts on Converged and Hyperconverged Infrastructure

2014-12-26T11:03:00.000-08:00

First we had converged infrastructure and then hyperconverged infrastructure. If the trend continues, next up will be the ultimate hyperconverged infrastructure. As these systems continue to evolve and become more advanced, they are gaining in popularity. Very few people doubt that products from Nutanix or the newly released VMware EVO:RAIL will be huge hits. The real question is whether a hyperconverged infrastructure is right for your data center.

Before converged infrastructure, the IT world had limited choices when deploying x86-based architecture. Rack/tower units and blades were the only choices available. A common feature among these approaches is that they often used external storage to accommodate virtualization -- a common feature and a downside. Blades had an additional advantage in a shared networking, power and cooling infrastructure. Of course, the downside was that you had a higher density of servers in a single enclosure, which could increase your risk in the event of a failure. Rack servers had the advantage of separating the failure points at the additional cost of rack space and power, cooling and networking infrastructure. The middle ground between the two extremes did not exist until the introduction of converged and hyperconverged infrastructure.

Hyperconverged infrastructure in particular is looking to take the best points of the blades and rackmount servers and combine them into a better approach. This is a major step forward from most converged infrastructure solutions that marry traditional blade servers, external storage, and networking into a single consumable "block".

Compute: One of the drawbacks to blades was the high density of blades in the chassis. A single chassis failure could affect many hosts, bringing down hundreds or even thousands of virtual machines. A hyperconverged infrastructure often resembles four blade-style servers in a 2U form factor. This reduction of the possible outage footprint can be more appealing to the customer looking for higher availability. You are still gaining the benefit of data center consolidation while preserving a level of outage protection.

Networking: Connecting everything together presents a challenge in almost all environments. A rack server's ports are normally allocated for production traffic and storage. These ports come with a per-port cost, along with management overhead. In blade environments, virtual switching is often required, which adds an additional pair of switches to your environment but removes the trouble of having to cable the blades to it. A converged infrastructure does not include virtual switching and requires connections from each node to the existing switching infrastructure. This approach resembles the rack environment, just without the storage connections.

Storage: Traditional servers use internal server storage or larger external storage frames. The external storage frame enables the shared storage concept, and virtualization was quick to take advantage of the benefits shared storage enabled, including features like vMotion, load balancing and high availability. Hyperconverged infrastructure turns this on its head by using localized storage that is shared across the four hosts within the single frame or even further, across the entire cluster. This gives the advantage of shared storage without the need for the costly storage frames or the dedicated fiber infrastructure that often accompanies it. The local disk can be an enterprise-class spinning or solid-state drive, giving the converged infrastructure tremendous IOPS potential. As more converged infrastructure product vendors couple their hardware with software-defined storage abilities, the traditional storage frame designs are showing their age.

Compute, networking, storage: When you compare the benefits of the hyperconverged infrastructure over the traditional infrastructure, it's hard to not to see all of the positives (and very few negatives). Hyperconverged infrastructure has found that perfect midpoint between large blade enclosures and the single-server approach. With all of the benefits, where is the downside to going with the hyperconverged approach?

Design: When you look at your requirements and want to come to a decision about what hardware platform (hyperconverged or not converged) to go with, a key factor should come to mind: Do you plan to deploy or design in one or two deployment factors, or are you more likely to deploy in a two- to four-node method?

Nodes in a converged infrastructure are prepackaged, and knowing how you purchase is a big factor in knowing which is right for you. Converged infrastructure, by its nature, is not designed to support a single compute deployment where you would add additional compute nodes to an existing enclosure similar to a blade enclosure. The enclosure exists as a prepackaged unit that works with all of the compute nodes in the cluster.

As businesses trend toward the prepackaged approach, they also need to consider how they will approach working with a converged infrastructure.

Downside: With all the positives of a converged infrastructure, there are some downsides. The prepackaged nature of converged infrastructure is a higher investment in capital costs. This simple math would suggest it costs four times the price of a single server. However, this is a flawed assumption because a converged infrastructure also replaces some storage and networking needs. Traditional external storage frames with fiber cabling and switches are expensive.

The second downside can also be leveraged as an upside since today's IT departments are often silos of professionals responsible for defined infrastructure roles with little cross-training or responsibility. While virtualization has started to break down these IT staff silos, it is still a work in progress and moving very slowly for many organizations. Infrastructure ownership and the division of groups have deep roots in IT. Combining these silos in not simply about reducing the hardware pieces; it can also affect staffing levels. The integration of virtualization has caused some jobs to be eliminated and others expanded, but IT continues to survive and evolve, and the same will occur with a hyperconverged infrastructure. Sometimes the introduction of hyperconverged infrastructure can be the catalyst needed to break down these silo's since it often comes with management software that handles servers, storage, and some networking tasks from the same GUI.

Hyperconverged infrastructure isn't for everyone right now, with its prepackaged requirements and price. However, its ease of use and integration of storage and networking means it's only going to grow. The breakdown of separate IT silos is not a stopping point, as it is something that the business world cannot ignore even if traditional IT would like to. Just like virtualization, it is not a question of whether to use Hyperconverged infrastructure. The real question is whether you choose to adopt it soon or find yourself in a constantly shrinking pool that is having trouble keeping up.

I think that the definition of company culture is wrong

2014-12-20T10:57:00.000-08:00

Although cool, your company’s free organic food and Uber allowance are not “culture.” The fact that everyone, including the CEO, comes to work clad in jeans and a hoodie is not “culture” either.

When we talk about culture, too often we talk about the wildly luxurious perks, or we celebrate employees’ traits or behaviors that have little to do with their actual performance at work, as if those were the reasons why the tech sector has been so successful.

Something like, “Company X is cool because they give everyone a free puppy and they’re known to be rockstar engineers and they all kite surf,” is really only describing a company at a very surface level, if at all.

Unfortunately, celebrating, or at minimum acknowledging, that definition of culture is now the cost of doing business, especially when you’re hiring talented engineers, fresh out of school. I can imagine that when you’re looking for your first job, a doggy-day-care subsidy might seem more like a concrete, positive reason to work at a given company, versus transparency or continuous improvement.

Company culture should be about the business

Here’s how I would define company culture:

It’s the set of values, traits, and systems that are deliberate, obvious and inherent in a company, existing to make the company successful.

When you define (and celebrate!) company culture this way, it becomes obvious why culture should be important to an organization, why culture eats perks for breakfast. Culture is the way the work gets done, the decisions get made, the people get hired. If you’re not focused on culture as a holistic mechanism to build the company’s success, then you’re coming at it from a potentially bifurcated point of view.

Call it culture, beliefs, values, organizing principles, whatever. It can be real and right now, but also aspirational. But no matter what, it has to come back to driving company success.

Let’s take a value like distributed decision making: trusting and empowering the experts (not necessarily the execs) in a company to make decisions in their domain. A company might value distributed decision making because it increases the speed of business; employees can move faster and do more if they don’t have to wait on executive approval, and if the decisions are made by the people with the most information.

Sounds great, right? Is that kind of empowerment part of your company’s culture? Here’s how you can tell:

When you hire someone, do you ask questions to see if they have decision making skills, if they’re able to make decisions without looking upwards?
Do you get rewarded through formal rewards programs for having made good decisions?
Are negotiation and evaluation skills critical to your career development?
When you finish big projects, do you do a post-mortem that includes assessing whether or not you made the right decisions?

Basically, your company’s culture should be baked into the org structure, the people systems (hiring, firing, performance management), and especially into your business plan.

Let’s get to work on the real definition of company culture

And I know, this process of inculcating a true definition of culture sounds like a lot of work. No organization is perfect at living up to its values. But rather than spend hours and hours wordsmithing the perfect vision for your company’s culture, the real work is find and bridge the gap from where your culture is today and where you want it to be. Aligning the company culture with the success of your business takes it from a nice-to-have to an imperative - a way to achieve success, not just market it.

The Task of Democratizing Big Data

2014-05-17T09:59:00.003-07:00

Companies that fail to take advantage of the opportunities presented by "big data" management and analytics technologies can expect to fall behind the competition and possibly go out of business altogether.

The world is just getting started with big data technologies like Hadoop and MapReduce, and several obstacles – such as a dearth of skills and old-fashioned thinking about data -- continue to stand in the way of their adoption.

But, companies that embrace the concept now are the ones who will lead the way in the not-too-distant future when entry barriers are not so high. Companies that exploit big data will gain the ability to make more informed decisions about the future and will ultimately bring in more money than those that do not.

The phrase "big data" is most often used to refer to the massive amounts of both structured and unstructured information being generated by machines, social media sites and mobile devices today. The phrase is also used to refer to the storage, management and analytical technologies used to draw valuable business insights from such information. Some of the more well-known big data management technologies include the Apache Hadoop Distributed File System, MapReduce, Hive, Pig and Mahout.

There is certainly no shortage of hype around big data management technologies, but actual adoption levels remain low for two main reasons. First, Hadoop and other big data technologies are extremely difficult to use and the right skill sets are in short order. Today, organizations often hire PhDs to handle the analytics side of the big data equation, and those well-educated individuals demand high salaries.

The skills used to manage, deploy and monitor Hadoop are not necessarily the same skills that an Oracle DBA might have. For instance, if you want to be a data scientist on the analytics side, you need to know how to write MapReduce jobs, which is not the same as writing SQL queries by any means.

The second major obstacle standing in the way of increased adoption centers on the notion that most companies currently lack the mindset required to get the most out of big data.

Most large companies today are accustomed to gaining business insights through a combination of data warehousing and business intelligence (BI) re¬porting technologies. But, the BI/data warehousing model is about using data to examine the past, whereas big data technologies are about using data to predict the future. To take advantage of big data requires a shift, a very basic shift in some organizations, to actually trusting data and actually going where the data leads you. Big data is about looking forward, making predictions and taking action.

As with all emerging technologies, big data management and analytics will eventually become more accessible to the masses -- or democratized -- over time. But some important things need to happen first.

For starters, new tools and technologies will be needed to reduce the complexity associated with working with big data technologies. Several companies -- like Talend, Hortonworks and Cloudera -- are working to reduce big data difficulties right now. But, more innovation is needed to make it easier for users to deploy, administer and secure Hadoop clusters and create integrations between processes and data sources.

Right now you need some pretty sophisticated skills around MapReduce and other languages, or SAS and others to be a top line data scientist. We need tools that can abstract away some of that expertise so that you don't need to have a PhD to really explore big data.

The task of democratizing big data will also require a great deal of user training and education on topics like big data infrastructure, deploying and managing Hadoop, integration and scheduling MapReduce jobs. We really need to tackle the problem from both ends. One is to make the tools and technologies easier to use. But we also have to invest in training and education resources to help DBAs and business analysts up their game and operate in the big data world.

Modernizing Your Backups

2014-05-12T09:21:00.000-07:00

This week, I'd like to spend a little time talking about backup modernization, or as I prefer to call it, data protection modernization. The process we use for traditional backups hasn't really changed much in 20 or 30 years. We do a full backup once per week, and take some kind of incremental backup of our data every day in between. These backups are aways copied to some other storage mechanism like tape, or now disk and a retention is attached to the backups that defines how long we need to keep that backup. Those retentions are important, since they define things like how much dedicated backup disk we need, or how many tapes we need to have on hand, etc. They also play an important roll later on when/if we decide to change the way we do backups.

But first, let's talk about the fact that traditional backup processes are really beginning to become more and more problematic. Why? There are actually a number of reasons. First, and perhaps the most obvious, are that data sets are becoming lager and larger every day. This means that either the backups are taking longer and longer to complete, or, more and more backup infrastructure needs to be put in place. Dedicated 10GbE connections, backup to disk, more and more and faster and fast tape drives, all need to be put into place to keep up. Yet it's a losing battle. The data sets just keep getting bigger. For example having a NAS array today that holds a petabyte of data isn't terribly unusual like it was not all that long ago. These bigger and bigger data sets are now benign to outstrip the ability of the storage system to send data to the backup system in a timely manner. Things like NDMP are just not able to keep up with these very large data sets. So data set size is certainly one of the more pressing reasons that people are beginning to look into modernizing their backups.

Another reason that people are bringing to look at modernizing their backup processes is that backup windows are getting smaller and smaller, and in some cases, closing completely. Back in the day, we had all night to run backups. Yes, of course we had to dodge in-between the batch jobs, that that was easy enough to do when you had 12 or more hours to do that. Those days are pretty much over. Today you are luck to get any time at all to backup the data, and as I said above, in some cases, you really don't have a window at all.

Finally, Recovery Time Objectives (RTO's) are getting shorter and shorter and the Recovery Point Objectives are getting smaller and smaller. What this means for the backup administrator is that they must take more backups, and must be able to retire form those backups more quickly.

So, what to do? The first step that many of my customers have taken s to start to include snapshots are part of the backup process. This addressees the issue of RTOs and RPOs since you can take those snapshots quickly, and you can recover from them quickly. You can also take multiple snapshots per day, so you have a much more fine grained ability to recover that data to a particular point in time. However, must people continue to do their regular backups as well, based on the premiss that snapshots aren't backups since they don't make a full copy of the data to another storage medium. However, for some customers it's beginning to become so problematic to do those tradition fills and incrementals, they are revisiting this position. Specifically, if they were to have a problem with their storage array, such that they lost data, and couldn't recover from a snapshot, isn't that the definition of a disaster in the data center? if you accept that premise, then you can start to consider a combination of snapshots, and say data replication for disaster recover, as a fixable, complete backup solution and drop traditional backups entirely.

A move to nothing but snapshots and replication as your data protection mechanism solves a number of issues. It address the ever growing backup infrastructure, for example, by leveraging space you already have on your storage array, and a DR plan (replication) you may very well already have in place. Admittedly, for some longer retentions it might mean you need a bit more disk space in your array, but because of the nature of snapshots it's probably the same or less space than you would need for disk based backups. If you are already backing up to an external backup to disk array like a Data Domain, you can repurpose your DD budget and add the space you need to your storage array to hold all of the snapshots you need/want.

Another method now bringing to become popular to modernize your backups is to leverage change block tracking. This is a mechanism in which the backup application, the storage array, or the hypervisor keeps track of the specific blocks that have changed, and the backup application only "backs up" these changed blocks. This can reduce the amount of backup traffic from the storage array to the backup infrastructure significantly, thus addressing the issue of the ever growing backup data sets. If you couple this with CDP (Continuous Data Protection) or near CDP functionality, it will also address the RPO issues, and since recovery from this kind of backup often means sending less data back to the storage array/application it can also address the RTO issues.

However, since you are probably already do some kind of backup, most like a traditional backup, the question becomes, how do I get from my current traditional backups to one of these more modern backup techniques? While on the surface it may seem simple enough, there are a number of issues to consider. First, you need to consider your existing backups. Those backups have a retention, and so you need to keep you existing backup software/mechanism in place, at least until the retentions on those existing backups have expired. One question that often crops up in this regard is what if I have backups with very long retentions, like 7 years? Does this mean I need to keep my existing backup mechanism in place for 7 years?Well, that's certainly one way to handle the problem. One way to mitigate the issue a little if you can, is to PTV your existing backup servers once you've switch all you backups to the new method. You can then shut down those VM's, and only spin them up if you need to get back at that old data for some reason. Another way to address the issue to to recognize that backups with long retentions are often not backups at all, they are actually archives, and they probably shouldn't have been backups in the first place. This is the perfect opportunity to start a dialog with your customers about the difference between backup and archive, and getting an archive mechanism in place to handle that data. The difference between archive and backup is a topic near and dear to my heart, but it's also beyond the scope of this posting. Just keep this in mind when you go to do your backup modernization planning.

The other issue to that you should consider when planning to modernize your backups is management. Much of the utility of today's backup software such as CommVault, NetBackup, and TSM is around managing the backups. Scheduling them, monitoring that they complete successfully, and reporting on them both from a administrative perspective, but also up the management tree and to your customers so that everyone is assured that their data is protected. Many people think that moving to a new more modern backup process means getting rid of these tried and true software programs. However, these may be an advantage to keeping them in place. For example, that reportage mechanism that is so important to your business then also stays in place. Considering that many snapshots, for example, are managed by software provided by the array manufacturer, and often only manage the snapshots on once array at a time, you could end up in a situation where your backups are modernized, but your backup management has taken a step back in time. this is also true if you bring on several deterrent techniques to backup you data. For example, I know of customers who use snapshots and replication for the databases, and then use something like Veeam to backup their virtual infrastructure. This has the potential to create an even bigger management/administrative/reporting headache.

So, if you can leverage your current backup software to manage your snapshots, and/or perform CDP like functions via change block tracking, then I believe that you've hit on the best of both worlds. The good news is, that most of the backup software vendors are recognized this, and are moving aggressively to add these kinds of features into their products. Admittedly, some are further ahead in some areas than others, but it's not like you have to change overnight, so implementing the features as they appear in your backup software isn't necessarily a bad thing.

It takes courage to say "yes"

2014-05-03T09:16:00.000-07:00

Today I want to talk about something a little different. While my posts on here have, in the past, all been technical, some of us are also in leadership roles. So, I think that occasionally I might share some of my near 30 years of experience in that regard as well.

What I want to talk about in this post is, from a leadership pony of view, it really does take courage to say "yes", especially to a new idea. “Definitely not,” is quicker, simpler, and easier than, saying, “Tell me more.” But, a quick “no” devalues and deflates teammates.

Some of the reasons that leaders are constantly saying "no" include:

They think that it makes them look weak when they say "yes" too often.
They prefer the "safety" of the state-quo. This is another way of saying they are afraid of change, or at least it makes them uncomfortable.
They haven’t clearly articulated mission and vision. Off-the-wall suggestions indicate the people in the ranks don’t see the big picture.

There are some dangers to offhanded yeses however. Offhanded yeses can dilute your resources, divide energy, and distract focus. So, what do good leaders do? They explore "yes". I know that takes time, but I believe that the time spent is a good investment.

Here are 8 questions to yes:

What are you trying to accomplish?
How does this align with mission or vision?
Who does this idea impact? How?
How will this impact what we are currently doing?
What resources are required to pull this off?
How does this move us toward simplicity and clarity? But, remember new ideas often feel complex at first.
Is a test-run appropriate?
How will we determine success or failure?

Leaders who say yes end up doing what others want and that’s a good thing. Remember too that courageous leaders are willing to risk being wrong sometimes in order to be right most of the time. They know that decisions move the organization forward. They know that a lack of a decision is in fact a decision; it’s a decision to do nothing and that’s a decision that is almost always wrong and at times catastrophic.

So, are you a leader that says "yes"?

Openstack Icehouse release, a first look

2014-04-23T16:42:00.000-07:00

On April 17, the OpenStack Foundation announced the availability of the ninth release of OpenStack, codenamed Icehouse. The release boasts 350 new features, 2,902 bug fixes and contributions from over 1200 contributors.

Icehouse focuses on maturity and stability as can be seen by its attention to continuous integration (CI) systems, which featured the testing of 53 third party hardware and software systems on OpenStack Icehouse.

The hallmark of the Icehouse release consists of its support for rolling upgrades in OpenStack Compute Nova. With Icehouse's support for rolling upgrades, VMs no longer need to be shut down in order to install upgrades. Icehouse "enables deployers to upgrade controller infrastructure first, and subsequently upgrade individual compute nodes without requiring downtime of the entire cloud to complete." As a result, upgrades can be completed with decreased system downtime, thereby rendering OpenStack significantly more appealing to enterprise customers. There are also some added functions for KVM, Hyper-V, VMware, and XenServer which are too numerous to go into here. See the Openstack Icehouse release notes for more details.

Icehouse also features a "discoverability" enhancement to OpenStack Swift that allows admins to obtain data about which features are supported in a specific cluster by means of an API call. Swift now also supports system-level metadata on accounts and containers. System metadata provides a means to store internal custom metadata with associated Swift resources in a safe and secure fashion without actually having to plumb custom metadata through the core swift servers. The new gatekeeper middleware prevents this system metadata from leaking into the request or being set by a client.

On the networking front, OpenStack now contains new drivers and support for the IBM SDN-VE, Nuage, OneConvergence and OpenDaylight software defined networking protocols. It also supports new load balancing as a service drivers from Embane, NetScaler, and Radware as well as a new VPN driver that supports Cisco CSR.

Meanwhile, OpenStack Keystone identity management allows users to leverage federated authentication for "multiple identity providers" such that customers can now use the same authentication credentials for public and private OpenStack clouds. The assignments backend (the source of authorization data) has now been completely separated from the identity backend (the source of authentication data). This means that you can now back your deployment's identity data to LDAP, and your authorization data to SQL, for example.

The Openstack Dashboard (Horizon) has support for managing a number of new features.

Horizon Nova support now includes:

Live Migration Support
HyperV console support
Disk config option support
Improved support for managing host aggregates and availability zones.
Support for easily setting flavor extra specs

Horizon Cinder support now includes:

Role based access support for Cinder views
v2 API support
Extend volume support

Horizon Neutron support now includes:

Router Rules Support -- displays router rules on routers when returned by neutron

Hoizon Swift support now includes:

Support for creating public containers and providing links to those containers
Support explicit creation of pseudo directories

Horizon Heat support now includes:

Ability to update an existing stack
Template validation
Support for adding an environment files

Horizon Ceilometer support now includes:

Administrators can now view daily usage reports per project across services.

In total, Icehouse constitutes an impressive release that focuses on improving existing functionality as opposed to deploying a slew of Beta-level functionalities. OpenStack's press release claims "the voice of the user" is reflected in Icehouse but the real defining feature of this release is a tighter integration of OpenStack's computing, storage, networking, identity and orchestration functionality.

Simplivity vs. Nutanix

2014-04-12T09:11:00.000-07:00

At a high level both of these products provide the same service(s) for the user. Certainly the two “leap-frog” each other in terms of features, but at this point in time, they are very close. Both of them are “Hyper-converged VMware appliances”, though Nutanix is able to support other hypervisors such as Hyper-V and KVM as well as VMware. Simplivity will allow large customers to utilize their own hardware, however, the customer must buy the Simplivity software as well as the OmniCube Accelerator Card for each server since the card is what does all of the writes in the Simplivity architecture.

From an architectural perspective both systems provide a “hyper-converged” solution made up of X86 servers with internal storage which are networked/clustered together. You grow the overall system by simply adding additional nodes to the cluster. As of this writing, Nutanix offers more different kinds of options for those nodes, giving the user more flexibility on how the clusters is gown. Both systems provide for multiple tiers of storage including SSD and HDDs and will automatically move hot data between tiers. It should be noted that Nutanix offers an interesting feature that Simplivity does not. Nutanix has the concept of “data locality”. With data locality, when you v-motion a VM to a different node in the cluster, Nutanix will move the datastore(s) for that VM to the same node (assuming there is space). This movement is done in the background, over time so as not to impact performance.

As of the latest version both systems provide deduplication of data natively built into the system. There is some discussion about which method of deduplication is “better”, however, in the end I believe that both will provide the user good deduplication results. Both systems also provide compression of data at the lower tiers.

Again, in regards to backups, replication, DR, etc. both system provide very similar features. Both systems allow for replication of deduplicated/compressed data thus providing “WAN optimization”, both systems provide for snapshots, and both systems replicate data within the cluster for data durability. Simpilivity is able to provide one feature that Nutanix is not currently able to support, and that is replication to the “cloud”. Specifically, Simplivity provides their software as a VM image running in AWS which can be federated to an Omnicube running in the users data center.

In regards to management. Both systems provide for a GUI management environment which allows the user to manage the entire footprint from a single pane of glass. Again, how this is implemented is somewhat different. Nutanix provides a somewhat traditional management GUI based on HTML 5 that can be used to manage the Nutanix system. Simplivity takes a different approach. Simplivity utilizes a Vcenter plug-in to manage the Simplivity Omicube. This ties Simplivity to VMware, and will make it more difficult to support other hypervisors.

In conclusion, I believe that the two products would provide effectively the same capabilities for most customers with the single exception of the AWS support that Simplivity provides. This support would provide the ability for customers to create a Hybrid cloud infrastructure that span the customers private cloud and the AWS public cloud.

Is 2014 the Year of Object Based Storage?

2014-04-02T13:37:00.003-07:00

Object based storage has actually been around for a long time. Some implementations started to appear as early as 1996, and there have been different vendors offering the technology ever since. However, it has never experienced the “explosion” in usage that some were predicting that it would.

It least until now.

IDC said the OBS market is still in its infancy but it offers a promising future for organizations trying to balance scale, complexity, and costs. The leaders include Quantum, Amplidata, Cleversafe, Data Direct Networks, EMC, and Scality, with other notables such as Caringo, Cloudian, Hitachi Data Systems, NetApp, Basho, Huawei, NEC, and Tarmin.

Last year OBS solutions were expected to account for nearly 37% of file-and-OBS (FOBS) market revenues, with the overall FOBS market projected to be worth $23 billion, and reach $38 billion in 2017, according to IDC. At a compound annual growth rate (CAGR) of 24.5% from 2012 to 2017, scale-out FOBS – delivered either as software, virtual storage appliances, hardware appliances, or self-built for delivering cloud-based offerings – is taking advantage of the evolution of storage to being software-based.

IDC predicts that scale-up solutions, including unitary file servers and scale-up appliances and gateways, will fall on hard times throughout the forecast period, experiencing sluggish growth through 2016 before beginning to decline in 2017.

IDC said emerging OBS technologies include: Compustorage (hyperconverged), Seagate Open Storage platform, and Intel’s efforts with OpenStack. The revenue of all of OBS vendors combined is relatively small right now (but expected to grow rapidly) with a total addressable market (TAM) expected to be in the billions. Noted Ashish Nadkarni, Research Director, Storage Systems, IDC. “Vendors like EMC and NetApp have not ignored this market – if anything they have laid the groundwork for it.”

One of the challenges that IT continues to confront is the growth of unstructured data. This growth creates challenges around data protection, as well as for users when they go to find their data. Object based storage addresses both of these issues. Use of technologies like Erasure Codes allows OBS to store data in a way that is both highly durable, as well as geographically distributed. This eliminates the need to create multiple full copies of the data in multiple locations, as you would have to do with traditional NAS arrays. So, rather than having to place storage systems that comprise 300% of your actual data size, you can utilize as little as 50%.

In addition, because many object storage systems are software solutions that can be run on nodes using low cost server hardware and high capacity disk drives, they can cost significantly less than proprietary NAS systems. Throw in better data protection and enhanced features that can enhance search performance and efficient data tiering and it’s easy to see why OBS is catching on.

So, what’s the downside? There are a couple. First, it’s performance. OBS typically cannot match the performance of traditional NAS arrays. With object retrieval latency in the 30-50ms range, applications that require high performance are going to have a problem with OBS. This is one of the reasons that AWS recommends that you put data on Elastic Block Storage if you need good performance, as opposed to using S3. The other challenge is that applications today are often not written to access data on OBS. Therefore changes to applications must be made, or the OBS storage must be accessed through a NAS gateway. Introducing a NAS gateway, however, eliminates the flat namespace, as well as the ability to attach meaningful metadata to your files/object. This reduces the utility of OBS significantly. However, the use of NAS gateways as an interim solution may simply be a necessity if OBS is to take over the NAS space.

Backing Up Openstack

2014-03-08T11:21:00.000-08:00

Today, I want to talk a little backup backups. Specifically, how to backup your Openstack environment. But not only how to backup the contents of your Openstack environment, but how to backup Openstack itself.

The thing to keep in mind here is that OpenStack is based around a modular architecture in which a number of different components can be combined together to offer cloud services on standardized hardware. These modules are freely available under the Apache license.

Backing up OpenStack

Backup solutions are typically developed with either operating systems or applications in mind. OpenStack is neither. OpenStack is merely a collection of components that can be combined to provide various types of cloud services. As such, OpenStack administrators must consider what needs to be backed up and how to perform the backup.

OpenStack backups should focus on backing up configuration files and databases for Openstack itself. The configuration files can be backed up at the file level since Openstack is just software running on a Linux machine.

The /etc/nova /var/lib/nova folders should be backed up on both the cloud controller and the compute nodes. However, you must exclude the /var/lib/nova/instances folder on any compute nodes. This folder contains live KVM instances. Restoring a backup that was made of a live KVM instance will typically result in an unbootable image.

One of the most important folders to include in your backup is /etc/swift/. This folder contains the ring files, ring builder files, and swift configuration files. If the contents of this folder are lost, the cluster data will become inaccessible. As such, it is a good idea to copy the contents of this folder to each storage node so that multiple backups exist within your storage cluster.

Some other folders that contain configuration data and should be included in your backups include:

/etc/keystone
/var/log/keystone
/etc/cinder
/var/log/cinder
/etc/glance
/var/log/glance
/var/lib/glance
/var/lib.glance/images

In addition to the folders listed above, there are also several databases that need to be backed up. Typically the databases will reside on the cloud controller, which doubles as a MySQL Server. This server hosts databases related to the Keystone, Cinder, Nova, and Glance components of OpenStack.

You can back up these databases by using the mysqldump command. The command requires you to specify the names of the databases that you want to back up as well as an output file. For example, if you wanted to back up the keystone database to a file named KeystoneBackup, you could do so with the following command:

# mysqldump –opt keystone > KeystoneBackup.sql

As a shortcut, you can substitute the all-databases parameter in place of the database name. For instance, if you wanted to back up all of the databases to a file named MyCloud, you could use the following command:

# mysqldump –opt –all-databases >MyCloud.sql

What's missing?

Backing up configuration files and databases will allow you to protect your OpenStack configuration, but there are some things that are not protected by this type of backup. This method does not protect individual objects within object storage. Similarly, block storage data is also left unprotected. According to the OpenStack documentation, these types of data are left for users to back up on their own.

You can use any compatible backup application. The OpenStack documentation basically says that it is up to the users to back up data residing on the virtual machines that they create. As such, the backup application would have to be compatible with the virtual machines. It should be noted at this point that users of the public cloud have the same problem. Public cloud providers also leave it up to the user to backup their virtual machines and the data that this virtual machines use.

Of course, this raises the question of how you can better protect an OpenStack cloud. One thing to keep in mind is that like any cloud environment, OpenStack makes use of server virtualization. In fact, OpenStack is designed to work with a number of different hypervisors. You can see the full hypervisor support matrix at:https://wiki.openstack.org/wiki/HypervisorSupportMatrix

One way that you can better protect your OpenStack environment is to adopt a backup application that is specifically designed for the hypervisor that you are using. You will still need to protect the OpenStack configuration files and databases, but you can use the backup software to protect the individual virtual machines and their contents

Another thing that you can do is to adopt a backup application that is OpenStack aware. However, this is more easily said than done. As previously mentioned, OpenStack is a collection of modular components that can be used to construct a private cloud. As such, none of the major backup products come preconfigured to back up OpenStack clouds.

Backup vendor Druva recently made headlines when they announced that their inSync software now supports OpenStack based scale-out storage. The software is designed to access OpenStack storage using the SWIFT OpenStack storage access protocol. It will also have the ability to back up file and object storage, as well as mobile endpoints (laptops, smart phones, etc).

Similarly, Zmanda supports the OpenStack framework with its Amanda enterprise backup software. The software is designed to create backups from the remote server layer.

Both Druva and Zmanda back up specific OpenStack resources, as opposed to the entire OpenStack infrastructure. It should be possible to also use traditional backup apps like NetBackup for Linux to back up the required components. However, NetBackup is not OpenStack aware. It would, therefore, be the backup admin's responsibility to manually configure a backup job that includes all of the required config data and databases.

The key to adequately protecting your OpenStack environment is to determine what it is that needs to be protected and then build a backup solution to meet those needs. While there are commercial products that can back up certain OpenStack resources, those products may not offer the level of protection that you require. You may have to combine commercial backup products with script-based backup techniques.

PaaS Outlook for 2014

2014-02-09T12:24:00.000-08:00

Yup, it's that time of year again where everyone makes their 2014 predictions. I guess I'm no exception...

In this blog posting I’d like to spend a little time talking about Cloud, and specifically, about PaaS. But first, a little background material.

First, lets define the different “kinds” of Clouds there are:

Public Cloud – Gartner defines public cloud as a style of computing where scalable and elastic IT-enabled capabilities are provided as a service to external customers using Internet technologies—i.e., public cloud computing uses cloud computing technologies to support customers that are external to the provider’s organization.

Private Cloud – Webopedia defines Private cloud as the phrase used to describe a cloud-computing platform that is implemented within the corporate firewall, under the control of the IT department.

Hybrid Cloud – SearchCloudComputing.com defines a hybrid cloud as a cloud-computing environment in which an organization provides and manages some resources in-house and has others provided externally.

There are some others, but they are all basically variations of the above.

Once you have a Cloud solution, the question is, what kind of Cloud is it? Here are the definitions that NIST provides.

Infrastructure as a Service (Iaas) - The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., host firewalls).

Platform as a Service (PaaS) - The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment.

Software as a Service (SaaS) - The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure2. The applications are accessible from various client devices through either a thin client interface, such as a web browser (e.g., web-based email), or a program interface. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

I recently did some research on PaaS, but I found the research tough to do because it’s almost impossible to put all of the PaaS players in the same bucket, and common patterns are hard to find.

Unlike the IaaS players that provide IT resources as a service, PaaS providers are really solution development platforms. Therefore, they are built around the types of problems they solve, not some industry-accepted approach.

At the heart of the problem is the fact that PaaS is today’s most ill-defined area of cloud computing. The approaches, features, and definitions vary widely, with many PaaS providers offering a specific focus. This may include support for specific programming languages such as Salesforce.com’s Heroku, support for Ruby, Node.js, Python, and Java, or perhaps tight integration with major databases, such as Oracle’s Cloud Platform. Or, perhaps it’s the delivery model, with private PaaS offerings from Active State, App Fog, or Apprenda, for those of you who can’t yet trust the public PaaS offerings from Google or AWS. Then there is an entirely new set of PaaS providers such as Elastic Box that bring a completely different approach to the problem.

Overall it’s largely a function of the providers all of which are trying to be relevant in this emerging marketplace. PaaS is the last frontier of cloud computing, and thus the least defined. So, it’s still possible for vendors to manipulate the market by positioning their products to better define what PaaS is and its value, or, more likely, their campaigns will just confuse people.

In 2013, the PaaS market took on some new dimensions. Private PaaS players saw strong growth as some enterprises looked to keep applications and data in-house. Also, there is greater support for the emerging use of DevOps, better database integration, and better support for emerging multi-cloud deployments. This builds upon, not replaces, the traditional uses of PaaS to automate application development, testing, and deployment processes.

Moreover, the PaaS market saw increased meshing with the IaaS space in 2013. This includes strong showings from AWS Elastic Beanstalk, and other IaaS-focused players. We also saw the arrival of some new PaaS players, including Oracle, and we got a clearer picture of how Saleforce.com’s and Pivotal’s PaaS offerings will likely exist in the emerging market.

Given all of these developments, there is a need to reevaluate the PaaS market and the PaaS players, in terms of how PaaS truly fits within an enterprise application development strategy. Questions are emerging such as: When will PaaS work for enterprise IT? When will PaaS not work for enterprise IT? What is the changing value of PaaS technology, now, and into 2014?

Some confusion in the attempts to answer these questions, and complexity has emerged. This led to some pushback when it comes to PaaS within enterprise IT. Many consider PaaS too complicated and too limiting for most development efforts, and for most developers.

For instance, most PaaS offerings place the developer into a sandbox, with only the features and functions that the PaaS provider furnishes to build and deploy applications. While this makes development an easy and controlled process, many developers need to gain access to the resources and tools required to support specific features, such as remote and native APIs, as well as middleware and database services. While the PaaS providers consider this abstraction from the underlying “metal” a path to productivity, many developers don’t agree.

PaaS does provide the ability to automate much of the development and deployment activities, as well as provide the developers with the ability to offer self- and auto-provisioning capabilities. This means that application developers can focus on the applications, and not have to deal with the purchase of hardware, software, and development tools to support increasing demands on the applications or the need to scale.

Moreover, PaaS supports new and more innovative approaches to delivery, including DevOps and the move to “continuous delivery.” Approaches such as continuous integration, automated testing, and continuous deployment allow software to be developed to a high standard and easily packaged and deployed. This results in the ability to rapidly, reliably, and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead.

If there is a core pattern that is a part of most PaaS, it’s that it’s solution-oriented. PaaS providers are focused on being the factory for cloud applications, and they understand there are many paths to get to that goal. As such, the offerings are very different from provider to provider, and thus the market is fragmented, complex, and confusing to those in enterprise IT.

I suspect this situation won’t improve much as we enter 2014. However, PaaS continues to be a consideration for those moving to the cloud. How and if it’s leveraged will be defined by the particular enterprise.

Forecast, Cloudy with a lot of public and a touch of private...

2013-11-25T14:09:00.000-08:00

If you believe that the pundits tell you, then private cloud is all the rage for enterprise customers. Certainly, if you look at what we've been doing here at EVT, there seems to be some evidence to suggest that's actually true. Our Enterprise customers seem to all be either interested in, currently deploying, or running some kind of private cloud.

Forrester Research says that 31% of Enterprise customers already have a private cloud in place and 17% plan to build one over the next year. However, when you dig down a little further, what you'll find is that only 13% actually have something that fits the "true" definition of private cloud. Most have some kind of virtualization in place with some added software to help manage that virtual infrastructure. But, more often than not, those so called private clouds are missing some key elements of a "true" private cloud.

Part of the problem could be that IT has a very loosey-goosey definition of what private cloud really is - and therefore what it brings to the table for IT and for IT's customers. The National Institute of Standards and Technology (NIST) says that for Infrastructure as a Service (IaaS) to be considered as a cloud it must have 5 attributes:

On-demand self-service
Broad network access
Resource pooling
Rapid elasticity
Measured service

IT's definition of a cloud is often very different, and can vary from "We have a data center" to "we look just like Amazon Web Services". But without the 5 essential characteristics above, IT will not be able to achieve the goals of going to "The Cloud" that most people are trying to achieve. The scalability, elasticity, and cost savings that the public cloud promises to the business customers of IT are the real goals that IT should be looking to match with the private cloud.

So why is public cloud growing so rapidly? AWS was a $2 billion business last year, and they are predicted to double that this year. Yet, as you can see above, private cloud seems to be struggling to gain traction in the data center. Especially when you consider the number of data centers that have a private cloud in name only (PCINO). I suspect that there are a number of reasons.

First, moving to a true private cloud is a very difficult cultural and organizational hurdle for most IT departments. It really means a shake-up of IT at the most fundamental level all the way from the top to the bottom. It means that IT will truly have to morph into that service organization that they have been trying to morph into for a long time, and many have yet to reach. That's the cultural change. They also need to change from an organizational perspective. They need to move away from vertically siloed departments within IT such as Server, Storage, network, etc. to horizontally organized departments is key to IT achieving the results that they desire and to be able to compete successfully with public cloud providers.

It should be noted here that IT often attempts to "cheat" the organizational change by "matrixing" people from existing IT departments into new "cloud" organizations. This often leads to failure since those "matrixed" people often bring with them old ideas about how things should be run as well as old processes and procedures.

This change also must go beyond just IT. For example, the purchasing department must understand the new model for purchasing converged infrastructure. For example, they can't be allowed to "break up" the converged infrastructure and purchase the individual components through old, existing vendor relationships. This continues on into IT as well, converged infrastructure means just that, converged. This often means that equipment that was traditional purchased directly from the manufacturer may now be part of the converged infrastructure stack and thus will be purchased as part of that solution. These old relationships with vendors and manufacturers often get in to way to achieving "true" cloud.

So, IT's inability to make the cultural and organizational changes to successfully compete with the public cloud is one reason I believe that private cloud adoption is where it is today. A related reason is that in some cases IT recognizes the issues, and actually starts to utilize the public cloud to deliver services to their end users. This is often an attempt to reel back into the fold "shadow IT" that has already deployed solutions in the public cloud. This if often followed quickly by a discussion on IT's part of hybrid cloud. In many cases that's because IT feels it just can't compete against the public cloud for all applications, and thus comes to the reluctant conclusion that rather than lose the entire pie, it's willing to give some part of the pie to the public cloud and build a private cloud for the rest. There's also an unspoken idea on the IT of IT that once they get their hybrid cloud up and running, they they will eventually prove to the business that they are better than public cloud and thus a majority of the application will move into that private cloud over time leaving only a small hand full of applications in the public cloud.

In the end, I think that unless IT can address the barriers to private cloud discussed above, that their dream of making the public cloud a temporary home is actually just a pipe dream. But in either case, IT's future is one in which they are a service provider and service director that helps the business find the best, most cost effective home for their applications.

Is OpenStack ready for prime time yet?

2013-09-08T19:15:00.000-07:00

For those who've been reading this blog for a while, or who know me, you know that while I've been in the data center business for a long time, that lately I've been focused on storage and backup. However, over the last couple of years I've been watching the infrastructure business change. What I find interesting is that what's old is new again!

When I first started out in "Open Systems", network, server, and storage was all managed as a single entity. So, here we are again. A "pod" or stack is just network, server, and storage all managed together, as a single entity. The new wrinkle here is that we also size them as a single entity which provides a number of advantages. But that's for another blog. As a matter of fact, I plan to write a couple of blogs on IaaS/PaaS/SaaS, how to move successfully to "the cloud", and data protection a cloud environment.

In this blog, I want to talk about one of the "stacks" called "OpenStack". The first questions I get asked when I first begin to talk about OpenStack is, what's the difference between a "stack" and a "pod"? Why is it called OpenStack and not OpenPod? The confusion is quite understandable, since the amount of hype and marketecture around everything having to do with "the cloud", including this topic, is enormous. As a matter of fact, it's so bad, that some of the terms are, in my opinion, starting to become meaningless. So I like to start out any discussion of any of these topics with a couple of definitions so that the audience and I are on the same page. According to Wikipedia:

OpenStack is a cloud computing project to provide an infrastructure as a service (IaaS). It is free open source software released under the terms of the Apache License. The project is managed by the OpenStack Foundation, a non-profit corporate entity established in September 2012 to promote OpenStack software and its community.

This begs a definition of IaaS (Infrastructure as a Service) again form Wikipedia:

In the most basic cloud-service model, providers of IaaS offer computers - physical or (more often) virtual machines - and other resources. (A hypervisor, such as VMware, Hyper-V, Xen or KVM, runs the virtual machines as guests. Pools of hypervisors within the cloud operational support-system can support large numbers of virtual machines and the ability to scale services up and down according to customers' varying requirements.) IaaS clouds often offer additional resources such as a virtual-machine disk image library, raw (block) and file-based storage, firewalls, load balancers, IP addresses, virtual local area networks (VLANs), and software bundles. IaaS-cloud providers supply these resources on-demand from their large pools installed in data centers. For wide-area connectivity, customers can use either the Internet or carrier clouds (dedicated virtual private networks).

Note that IaaS can also be implemented in a private cloud (in your data center), or in both the public and a private cloud called a Hybrid Cloud. This ability to utilize the resources of both a private cloud, and a public cloud, is becoming more and more interesting to large enterprises. Again, more on this in a later blog where I will talk about the economics of "cloud".

OK, so enough of laying the groundwork. Let's talk about OpenStack, and see of we can answer the basic question, is it ready for "prime time"? Can I use it in the enterprise to implement my private cloud IaaS infrastructure? The answer is, maybe. Let's talk about it a bit.

First, clearly the interest in OpenStack is definitely growing, and growing quickly. You can see this by looking at the attendance of The OpeStack Summit which started out life with a $15,000.00 budget, and 75 people were basically coerced to go. The most recent OpenStack Summit had a $2 million budget and over 3,000 attendees. So, clearly, interest is up, but no where near the kind of interest that VMware has managed to get. The most recent VMworld had over 23,000 attendees. So, no doubt, lots of interest. But what's driving the interest? Obviously, cost is a big consideration. Since OpenStack is open source, the cost of implementing it is significantly lower than for any of the commercial software out there. But are there hidden costs that perhaps make it not as good a "buy" as perhaps one might think at first blush? the short answer to that is "yes", just like it is with any open source software. Things like support costs as well as the cost of finding/training staff, etc. all add to the TCO of any open source solution, including that of OpenStack.

But lets talk about OpenStack itself a bit. One of the things that I think was holding back OpenStack was the difficulty of deploying the solution. However, this is rapidly being address by software such as Canonical's Juju. There are also a number of companies that provide IOpenStack based solutions such as Pistson OpenStack. Piston provides a turn-key OpenStack solution that includes:

The other way we can tell if anything is ready for prime time is if we look at existing adoption of the technology. A year of two ago, there were almost no enterprise implementations of openStack outside of some service providers such as Rackspace, as well as NASA. This has changed, companies such as Bloomberg, Comcast, and Best Buy have all implemented OpenStack.

At the most recent OpenStack Summit Bloomberg CTO Pravir Chandra, one of several company executives who detailed their real-world experience with the platform at the summit, said his team set a high bar for OpenStack. Bloomberg’s goals included capabilities such as high availability, no cascading failures, and smooth scale down and scale up. As described in GigaOM:

"They were able to get there by deploying OpenStack along with considerable custom work of their own, both above and below that layer. They ended up setting up the high-availability databases and figuring out how to aggregate logs from the hypervisor level."

A story about Best Buy in ITWorld describes Bestbuy.com as "the poster child for organizations that can benefit from the cloud." The online retailer built an internal cloud on OpenStack that the company says speeds up the ecommerce site, allows faster development cycles, and scales.

For example, at the beginning of the Christmas shopping season last year, Bestbuy.com saw a spike of eight times its normal traffic, Joel Crabb, chief architect, told ITWorld. "If that doesn’t scream out for elastic scaling, I don’t know what does."

OpenStack also dramatically cut costs for Best Buy, company executives told summit attendees. Director of eBusiness Architecture Steve Eastham said past releases of the website cost about $20,000 to provision a single managed VM. With OpenStack, he said, the company is spending around $91,000 per rack.

So I think that it’s still an open question about how OpenStack will ultimately stack up against Amazon Web Services in the public cloud infrastructure sector and VMware in the (mostly) private cloud market, where legacy applications are in play. But OpenStack evangelists like Rackspace CTO John Engages are gearing up to bring their solutions to enterprise customers. In an interview, he told Ryan Cox:

The enterprise community is thirsty for the cloud and that ball will soon drop. The opportunity to innovate in open source with OpenStack is one that the legacy solutions in enterprise will soon be eaten. Mobile devices, Big Data, your and my Internet of Things … access to all of these through infrastructure that can scale quickly at low cost is a common theme we’re hearing at the OpenStack Summit 2013.

So, back to our original question, is OpenStack "ready for prime time"? I think that the answer is, maybe. If you're looking to build a private cloud infrastructure, I think it's a ready option. If you're looking for a hybrid solution, it's a bit less clear, but it's certainly possible.

Let me know what you think in the comments. I'm particularly curious if our involved in a OpenStack deployment.

Upgrading Your Storage Microcode

2013-06-03T10:44:00.000-07:00

Folks,

I was just reading a posting by Chris Evans on this topic at http://architecting.it/2013/06/03/managing-microcode-upgrades/ and he makes a lot of great points. I agree with everything that Chris posted, only I would go even further and say that based on my experience that having a regular process for upgrading your storage microcode is critical to managing any storage environment.

There seem to be three competing philosophies that cause problems on this topic "in the wild":

"If it Ain't Broke, Don't Fix It!" - This is the idea that you should only patch or upgrade your storage infrastructure if you run into a problem. I run into this approach more often than you would think, and invariably what this means is that you will run into every problem that exists in the microcode and have to deal with it on an "emergency" basis. It also means that you will often go for long periods of time without patching or updating, and then when you hit a problem, you have a huge jump, which almost always means that you also have a lot of servers that need HBA firmware and/or driver updates. This usually ends up being aHUGE and painful project, that, in some people's minds simply confirms why they are avoiding doing the storage microcode upgrades in the first place. What they don't realize is that the main reason it's so painful is that they are so far behind. If they actually kept up, then the pain would be less and spread over time.
"Pick a standard, and keep it as long as possible" - This approach is one I see fairly often as well. Here the storage team picks a "standard" version of the OS, ans sticks to it only patching it when there is a problem, or until they are forced to change because new hardware doesn't support than version of the OS any longer. Then they adopt the new version of the OS as their standard, and bring everything up to that level. It's actually similar to #1, and suffers from the same sorts of issues.
"Apply every patch and/or upgrade the vendor releases as soon as it becomes GA" - I see this much less frequently mainly because people are afraid, often rightfully so, that patching/upgrading this frequently will cause more problems than it solves.

The process that Chris outlines in his blog post, is, in my opinion, the right way to go. Apply your patches either quarterly, or twice per year in predefined upgrade windows. This doesn't mean that you can't apply patches to resolve specific issues as they arise.

But I would go a bit further in my definition of the process. Specifically, I would have a process that works something like this:

Between upgrades (i.e. during the quarter or 6 month period between upgrades) I would pull down every patch and upgrade that the storage manufacturer releases into the storage team lab, and apply it to a lab box. I would then apply a set of regression tests to validate that the patch/upgrade worked in my environment, with my servers, HBA's, etc.
About a week prior to my upgrade window I would pull together an "upgrade" package where I decide what patches/upgrades, etc I was going to apply to the storage, as well as any that were required for the HBA's, host OS's, etc. Note it's critically that the host HBA's be upgraded to the latest version of their drivers, etc. that are supported by the patches/upgrades that you are going to roll out to avoid issues. Upgrades to the servers are often avoided even more than the storage OS upgrades since they are usually the source of outages (reboot required) and due to the fact that it's not the storage team doing those upgrades in many cases.
I would actually have two windows, once for arrays that support dev/test, and one for arrays that support production if it's possible. I would then roll out the patches to dev/test, and let them bake there for a week or two, and then roll them out to production. This isn't 100% necessary, especially if you've done good testing in your lab, but it would be nice.
Go to step #1 and start the process all over again.

When I've described this process to people I often get push back like "hey, that means that we will constantly either testing, or performing upgrades"! This is especially the case if you decide to go on the quarterly schedule. My response if "yup, because that's part of what a storage team does, and why you have a storage team". Frankly, the team's time is better spent on this, than on, say, doing a lot of LUN allocations which you can automate, and even delegate, once it's automated.

The bottom line is, it's a "pay me now, or pay me later" situation and I would rather do as much of my patching/upgrading in a proactive manner, than in a reactie maner where's an a big emergency, and a big project with a lot of downtime at once.

Keeping the Lights on Syndrome

2012-09-23T15:12:00.002-07:00

Does your IT organization suffer from "Keeping the Lights on Syndrome"? For those of you who are asking, what the heck is "Keeping on the Lights Syndrome" here's a quick definition of the problem. "Keeping the Lights on Syndrome" is a situation that more and more IT organizations are finding themselves in where they are spending 70% of their IT budget on "keeping the lights on" and only 30% on innovating with the business and modernizing their technology.

So, what's the right number? Should it be 60% - 40%? 50% - 50%? Well for a lot of years the number has been closer to 50-50%, and that's probably a good number to strive toward for most IT organizations. The next question I'm often asked is how can I address this problem? What can I do to get my Infrastructure and Operations costs down? I'm already virtualizing my server infrastructure, and I'm looking at more virtualization including virtualizing my storage and my networks, what more can I do to get my I&O expenses down even further?

The answer is that virtualization has been a great help to keep the number down to just 70-30%. Without virtualization a lot of organization might be staring at 80-20%, or even 90-10%. Ok, so what's the next step you ask? Please don't say "cloud", I've heard that enough already in the last year! As a matter of fact, every manufacturer of infrastructure, and infrastructure software has been telling me that all I have to do is buy their solution and I have a "cloud solution" in place.

I tend to agree, the term "cloud" is over used. So, let's not use it here, lets instead look at some practical things that the I&O organization can do to address the "Keeping the Lights on Syndrome". Longer term, yes something like IT as a Service whether it's implemented using a private (internal) cloud, a public cloud, or a combination of the two called Hybrid Cloud doesn't matter. But that's a longer term solution. So what can the I&O organization do in the shorter term to address the problem, and maybe lay the groundwork for the longer term cloud solution as well?

What can I&O do beyond completing the current drive toward virtualizing almost the entire infrastructure? They can start "comoditizing" their infrastructure. What is "commodity" infrastructure? Its the idea that you buy your infrastructure including network, server, and storage as a single unit. Some people call this converged infrastructure, but what ever you call it the idea is to buy your infrastructure as a single SKU which defines a single unit of capacity for your infrastructure.

How does this help with the "Keeping the Lights on Syndrome"? It removes a major cost from your I&O organization. That cost being the cost of developing the "right" solution for each and every application, and then building a customer infrastructure to support that "right" solution. instead you buy your infrastructure capacity in "chunks", and then carve those "chunks"into standard sized pieces. That doesn't mean that those standard peices must all be identical, but rather there should be a limited number od standard sized "chunks". For example, Small, Medium, Large, and X-Large.

How does this help your I&O organization address the "Keeping the Lights on Syndrome"? It does so by making your purchasing more efficient. It also reduces the amount of engineering you have to do by eliminating most of the custom engineering and custom building that is still happening in the I&O organization in spite of the fact that you have virtualized much of your infrastructure.

So, how would this work, you ask me? My application teams need to have their requirements met! My response is that it would work just like buying a car. If you have a family of, say, 5 people, and you like to go on family driing trips, do yo ugo to the Ford dealership and tell them what you want in a car, and then have them build you a custom car that exactly meets your needs? No, of course not, you go to the dealership and choose among several different offering that they have, and then buy the one that is closest to meeting your needs. Sure, you can "customize" that car buy picking the color, the size of the engine, maybe pick some custom wheels, etc. But all of this is based on a limited set of standard platforms that Ford builds, it's not custom from the ground up.

Right now, I would argue that, even with virtualization, most I&O organizations are still building custom cars from the ground up. What I'm suggesting is that instead the I&O organization should be buying a "standard" platform, provide some standard sized "environments" that the application teams can pick from, and then only "customize" the application environments based on the standard platforms/environments. So, lets say for example that you have a converged infrastructure where a single unit of converged infrastructure can handle any combination of 2,000 small VM's, 1,000 medium VM's, 500 large VM's, and 250 X-large VM's. When the applications teams need new VM's they simply request one of the standard sizes. No custom engineering required. If, however, none of the standard sizes fits the needs of the application teams, then a custom engineered VM is built for them. The key here is to keep the number of custom VM's down to a minimum. Mostly this can be done through the charge back process by making any custom VM cost significantly more than an X-large VM.

What does this buy the I&O organization? First it addresses the "Kepping the Lights on Syndrome" by reducing the cost of deploying new infrastructure. It also makes the I&O organization more agile since it saves all of the time that is needed to engineer custom solutions. Finally, this approach also lays the groundwork for automation the deployment of the infrastructure, otherwise known as Infrastructure as a Service and IaaS is one of the first layers on the way to ITaaS and "cloud".

So, what are the barriers to implementing a converged infrastructure solution for your I&O organization? There are a number of them, actually, and they are all organizational in nature. First, you need to pick a partner that can provide you with the right converged infrastructure for your I&O organization. This can be an issue because typically your purcasing organization already has agreements in place with your storage, server, and network vendors, so if you want to continue to use that technology you are going to have to get your purchasing people on board and they are going to have to talk with your storage, server, and network suppliers about working with a partner that can pull together all three and provide them as a single SKU. Once you have that worked out, you need to get buy in from your engineering and architecture organization. The architecture organization is going to be threatened by this move to converged infrastructure since they will perceive this a a move to reduce their control over the infrastructure in the organization. However, the engineering organization is the one that will likely feel the most threatened by the move to converged infrastructure. They will very like view it as a direct attack against them and will though up every argument for why "this won't work" you can imagine. Finally, your storage, server, and network administration organizations will need to be revamped. Managing a converged infrastructure with 3 separate teams. Unless you reorganize to support/manage your converged infrastructure by a single organization much of the advantage of pulling storage/server/network together physically can be lost.

Finally, let me say that several of our customers who are at various stages of implementation of converged infrastructure, IaaS, and Cloud infrastructure, and how successful these initiatives are is directly related to the organization's ability to change and embrace the new technology. It also is directly related to the partner's ability to deliver on the organizations needs. Without a good partner who understand the needs and goals of the organization, they are doomed to failure.

ILM/HSM part 2, Return of ILM/HSM

2012-08-17T13:51:00.003-07:00

Folks, Sorry it's been so long since my last posting! Time fly's when you're having fun and I've been having a lot of fun over the last year. What have I been doing, you ask? Well a lot and among other things I've been trying to help our customers sort through a changing storage environment, and I've learned a few things in the process. What's all this change I'm referring to? Well, among other things, Flash/SSD has really started to take off, and that has a lot of implications for the storage team. So I have spend a lot of time helping our customers sort through the different options, etc. and discovered some things in the process that I would like to share with you. But first, a quick review of what's up with storage and Flash/SSD. As I indicated above, Flash/SSd is really beginning to make in-roads into the data center. Flash/SSD comes in basically three different flavors. First, Flash/SSD's can be used in something that looks like a traditional storage array. There are a couple of different variations of this type of storage array. Some use SSD drives in place of traditional disk drives, and some use Flash memory directly. Typically, the arrays that use SSD's provide many of the same features as other tradition storage arrays such as snapshots, replication, etc. Arrays based on Flash memory, on the other hand, typically provide better performance than arrays that use SSD drives mainly because the avoid all of the overhead involved with the SCSI protocol, etc. However, these arrays also often don't have all of the features we need in the data center such as snaps and replication, etc. In both cases, from a storage management perspective you would manage it much like any other storage array in your data center. Second there are the traditional storage arrays with Flash/SSD added to them. Again, these arrays come in basically two flavors. In both cases, however, an effort is made to only utilize the Flash/SSD for data which is currently "in use" or "hot" in an effort to keep the costs down. With the first flavor, SSD drives are used to hold "hot" blocks of data, with "cool" blocks of data being stored on traditional disk drives. This requires sophisticated software that monitors how "hot" the data is and moves it appropriately. With the second type of array Flash is added to the controller and used to extend the cache. This has the advantage that the software is a simple extension of the existing controller software, and as I mentioned above, the overhead of the SCSI protocol is avoided. The downside is that this only provides a performance boost for the read half of the equation. Finally, there is the ability to add Flash memory to the servers that run your applications. Once again, there are two flavors here. The first, and simplest flavor is to utilize the Flash memory as an extended disk cache. The advantage to this is that it accelerates I/O to/from any disk arrays you may already own. The down side is that it is often limited in what kinds of OS's it works with. The second flavor makes the Flash memory appear to the OS on the server as a disk drive. This has the advantage of very high performance, but is limited in size. It is also limited in that you can't use features like server clustering, etc. since this data can't be shared among a group of servers. So what's the lesson learned from all of the above? I think that there are a couple. One is that if we are going to utilize some or all of this technology in the data center, we are really looking at bringing back the old ILM/HSM days. For the "Flash/SSD" only arrays, because of their cost, most data centers aren't going to bring them in to replace all of their traditional storage array capacity. So some way to move data from the expensive storage to the less expense storage needs to be found if costs are going to be kept under control. With the second type of array software to move the data is supplied, but there are questions about how effective this software is particularly in keeping up with quickly changing "temperature" data. The third type of Flash/SSD certainly improves performance, but increases the "storage islands"in your data center unless some kind of ILM/HSM software can be applied. Where this leaves us is with many of the same issues that, ultimately, derailed ILM the last time around. The main issue at the time was the classification of the data. Getting the business to classify their data was very difficult, and in the end,we often threw up our hands and just moved data based on "last access date". While this works for file based data, it doesn't work for database data, for example, at all.

NetApp Deduplication An In-depth Look

2011-06-12T12:51:00.000-07:00

There has been a lot of discussion lately about the NetApp deduplication technology, especially on twitter. We had a lot of misinformation and FUD flying around, so I thought that a blog entry that takes a close look at the technology was in order.

But first a bit of disclosure, I currently work for a storage reseller that sells NetApp as well as other storage. The information in this blog posting is derived from NetApp documents, as well as my own personal experience with the technology at our customer sites. This posting is not intended to promote the technology as much as it is to explain it. The intent here is to provide information from an independent perspective. Those reading this blog post are, of course, free to interpret it the way they choose.

How NetApp writes data to disk.

First lets talk about how the technology works. For those who aren't familiar with how a NetApp array stores data on disk, here's the key to understanding how NetApp approaches writes. NetApp stores data on disk using a simple file system called WAFL (Write Anywhere File Layout). The file system stores metadata which contains information about the data blocks, has inodes that point to indirect blocks, and indirect blocks point to the data blocks. One other thing that should be noted about the way that NetApp writes data is that the controller will coalesce writes into full stripes when ever possible. Furthermore, the concept of updating a block is unknown in the NetApp world. Block updates are simply handled as new writes, and the pointers to the updated blocks are moved to point to the new "updated" block.

How deduplication works.

First, it should be noted that NetApp deduplication operates on a volume level. In other words,all of the data within a single NetApp volume is a candidate for deduplication. This includes both file data, and block (LUN) data that is stored within that Netapp volume. NetApp deduplication is a post-process that occurs based on either a watermark for the volume, or on a schedule. For example, if the volume exceeds 80% of it's capacity a deduplication run can be started automatically. Or, a deduplication run can be started at a particular time of day, usually at a time when the user thinks the array will be less utilized.

The maximum sharing for a block is 255. This means that if there are 500 duplicate blocks,there will be 2 blocks actually stored with 1/2 of the pointers pointing to the first block and 1/2 of the pointers pointing to the second block. Note that this 255 maximum is separate from the 255 maximum for snapshots.

When deduplication runs for the first time on a NetApp volume with existing data, it scans the blocks in the volume and creates a fingerprint database, which contains a sorted list of all fingerprints for used blocks in the volume. After the fingerprint file is created, fingerprints are checked for duplicates, and, when found, first a byte- by-byte comparison of the blocks is done to make sure that the blocks are indeed identical. If they are found to be identical, the block‘s pointer is updated to the already existing data block, and the new (duplicate) data block is released. Releasing a duplicate data block entails updating the indirect inode pointing to it, incrementing the block reference count for the already existing data block, and freeing the duplicate data block.

As new data is written to the deduplicated volume, a fingerprint is created for each new block and written to a change log file. When deduplication is run subsequently, the change log is sorted, its sorted fingerprints are merged with those in the fingerprint file, and then the deduplication processing occurs as described above. There are two change log files, so that as deduplication is running and merging the new blocks from one change log file into the fingerprint file, new data that is being written to the flexible volume is causing fingerprints for these new blocks to be written to the second change log file. The roles of the two files are then reversed the next time that deduplication is run. (For those familiar with Data ONTAP usage of NVRAM, this is analogous to when it switches from one half to the other to create a consistency point.) Note that when deduplication is run an an empty volume, the fingerprint file is still created from the log file.

Performance of NetApp deduplication.

There has been a lot of discussion about the performance of Netapp deduplication. In general, deduplication will use CPU and memory in the controller. How much CPU will be ustilied is very had to determine ahead of time, however in general you can expect to use from 0% to 15% of the CPU in most cases, but as much as 50% has been observed in some cases. The impact of deduplication on a host or application can very significantly and depends on a number of different factors including:

    •    The application and the type of dataset being used
    •    The data access pattern (for example, sequential versus random access, the size and pattern of the
    •    I/O)
    •    The amount of duplicate data, the compressibility of the data, the amount of total data, and the
    •    average file size
    •    The nature of the data layout in the volume
    •    The amount of changed data between deduplication runs
    •    The number of concurrent deduplication processes and compression scanners running
    •    The number of volumes that have compression/deduplication enabled on the system
    •    The hardware platform—the amount of CPU/memory in the system
    •    The amount of load on the system
    •    Disk types ATA/FC, and the RPM of the disk
    •    The number of disk spindles in the aggregate

The deduplication is a low priority process, so host I/O will take precedence over dedupllication. However, all of the items above will effect the performance of the deduplication process itself. In general you can expect to get somewhere between 100MB/sec to 200/MB/sec of data dedupication from a NetApp controller.

The effect of deduplication on the write performance of a system is very dependent on the model of controller and the amont of load that is being put on the system. For deduplicated volumes, if the load on a system is low—that is, for systems where the CPU utilization is around 50% or lower—there is a negligible difference in performance when writing data to a deduplicated volume, and there is no noticeable impact on other applications running on the system. On heavily used systems, however, where the system is nearly saturated, the impact on write performance can be expected to be around 15% for most models of controllers.

Read performance of a deduplicated volume depends on the type of reads being performed. The implicit on random reads is negligible. In early versions of ONTAP the impact of deduplication was noticeable with heavy sequential read applications. However with version 7.3.1 and above NetApp added something they called "intelligent cache" to ONTAP specifically to help with the performance of sequential reads on deduplicated volumes and were able to mitigate the performance impact of sequential reads nearly completely. Finally, with the addition of FlashCache cards to a controller, performance of deduplicated volumes can actually be better than non-deduplicated volumes.

Deuplication Interoperability with Snapshots.

Snapshots and their interoperability with deduplication has been a hotly debated topic on the internet lately. Snapshot copies lock blocks on disk that cannot be freed until the Snapshot copy expires or is deleted. On any volume, once a Snapshot copy of data is made, any subsequent changes to that data temporarily require additional disk space, until the snapshot is deleted or expires. The is true with deduplicated volumes as well as non-deduplicated volumes. Thus the space savings from deuplication for any data held by a snapshot prior to a deduplication run will not be recognized until after that snapshot expires or is deleted.

Some best practices to achieve the best space savings from deduplication-enabled volumes that contain Snapshot copies include:

    •    Run deduplication before creating new Snapshot copies.
    •    Limit the number of Snapshot copies you maintain.
    •    If possible, reduce the retention duration of Snapshot copies.
    •    Schedule deduplication only after significant new data has been written to the volume.
    •    Configure appropriate reserve space for the Snapshot copies.

Some Application Best Practices

VMWare

In general VMware deduplicates well, especially if a few best practices in laying out the VMDK files are considered. The following best practices should be considered for VMware implementations:

    •    Operating system data deduplicates very well therefore you should stack as many OS's onto the same volume as possible.
    •    Keep VM swap files, pagefiles, user and system temp directories on separate VMDK files.
    •    Utilize FlashCache where ever possible to cache frequently accessed blocks (like those from the OS).
    •    Always perform proper alignment of your VM's on the NetApp 4K boundaries.
    •

Microsoft Exchange

In general deduplication provides little benefit for versions of Microsoft Exchange prior to Exchange 2010. Starting with Exchange 2010 Microsoft has eliminated single instance storage and deduplication can reclaim much of the additional space created by this change.

Backups (NDMP, SnapMirror and SnapVault)

The following are some best practices to consider for backups of deduplicated volumes:

    •    Ensure deduplication operations initiate only after your backup completes.
    •    Deduplication operations on the destination volume complete prior to initiating the next backup.
    •    If backing up data from multiple volumes to a single volume you may achieve significant space savings from deduplication beyond that of the deduplication savings from the source volumes. This is because you are able to run deduplication on the destination volume which could contain duplicate
    •    data from multiple source volumes.
    •    If you are backing up data from your backup disk to tape consider using SMTape to preserve the deduplication/compression savings. Utilizing NDMP to tape will not preserve the deduplication savings on tape.
    •    Data compression can affect the throughput of your backups. The amount of impact is dependent upon the type of data, compressibility, storage system type and available resources on the destination storage system. It is important to test the affect on your environment before implementing
    •    into production.
    •    If the application that you are using to perform backups already does compression, NetApp data compression will not add significant additional savings.

Conclusions

In general, NetApp deduplication can help drive down the TCO of your storage systems significantly, especially when combined with FlashCache in a VMware or Virtual Desktop environment. If best practices are followed carefully, the performance impact of deduplication is negligible, and the space savings for some applications can be considerable. Some careful planning and testing in the customers environment are necessary to ensure that maximum advantage is taken of deduplication, however the ability to schedule when the operations take place combined with the ability to turn on and off deduplication provide significant flexibility in to tune the environment for a customer's particular application profile.

EMC FAST and NetApp FlashCache a Comparison

2011-05-30T22:51:00.000-07:00

Introduction

This article is intended to provide the reader with an introduction to two technologies, EMC FAST and NetApp FlashCache. Both of these technologies are intended to improve the performance of storage arrays, while also helping to bend the cost curve of storage downward. With the amount of data that needs to be stored increasing on a daily basis, anything that addresses the cost of storage is a welcome addition to the data center portfolio.

EMC FAST

EMC FAST (Fully Automated Storage Tiering) is actually a suite made of of two different products. the first, called FAST Cache operates by keeping a copy of "hot" blocks of data on SSD drives. In effect it acts as a very fast disk cache for data that is currently being accessed while the data itself is being stored on either 15K SAS or 7200 RPM NL-SAS (SATA) drives.

FAST Cache provides the ability to improve the performance of SATA drives, as well as to turbo charge the performance of fiber channel and SAS drives as well. In general, this kind of technology helps to divide performance from spindle count, which helps drive down the number of drives required for many workloads, thus driving down the cost of storage, and the overall TCO of storage.

The other product in the FAST suite is FAST Virtual Pool. This is the product that most people associate with FAST since it is the one that leverages three different disk technologies, SSD, high speed drives such as 15K RPM SAS, and slower high capacity drives such as 7200 RPM NL-SAS. By placing only data that requires high speed access on the SSD drives, data that is receiving a moderate amount of access on the 15K SAS drives, and putting the rest on the slower, high capacity disks EMC FAST is able to drive the TCO of storage downward.

NetApp FlashCache

NetApp approaches the overall issue of improved performance while simultaneously driving down the TCO of storage in a different way. NetApp believes that using fewer disks to store the same amount of data is the best way to drive down TCO. Therefore NetApp has spent a significant amount of time developing storage efficiency tools to help their customer's store more data in less space. For example, they developed a variant of RAID-6 called RAID-DP which provides the protection and performance of RAID-10, while utilizing significantly less space. NetApp has also developed block level de-duplication which can be utilized with primary production data.

However, as with many technologies of this type there could be a performance penalty paid for it's utilization. Therefore, Netapp needed to develop a way to improve the performance if it's arrays while also supporting it's storage efficiency technology. With the advent of Flash memory, Netapp found a way to do this without any need for significant changes in the architecture of it's arrays. Thus was born FlashCache.

FlashCahce provides a secondary read cache for hot blocks of data. This proves a way to separate performance from spindle count, and thus not only allows workloads intended for Fiber Channel or SAS drives to potentially run on SATA drives, but it also addresses some of the performance issues with the storage efficiency technologies that NetApp developed. For example, with FlashCache utilized in a virtual desktop environment Netapp de-duplication allows many individual Windows images to be represented in a very small footprint on disk. However a problem arrises when a large numer of desktops all try to access their Windows image at once. However with the addition of FlashCache, most, if not all of the Windows image would end up being storage in Flash memory, thus avoiding the performance issue of a boot storm, virus checking storm, etc.

Conclusion

Both EMC and Netapp have developed ways to help both improve the performance, and drive the TCO of storage downward. the two vendors approached the problem is somewhat different ways, but in the end they have both solved the problem in unique and effective ways.

The NetApp technology requires that the user buy-in completely to the NetApp vision of storage efficiency. If the user ignores the advantages of de-dupication in particular, or has data or workloads that simply don't allow for the application of the NetApp storage efficiency technology then the TCO saving that NetApp promises will not be achieved. Utilizing FlashCache to seperate performance from spindle count is also critical in maintaining the performance of the array. This separation of performance from spindle count also in and of itself drives dwn the number ofd drives needed to support a workload, and thus also drives down the TCO.

The EMC technology requires a very good understanding of your application workloads, and careful planning and sizing of the different tiers of storage. EMC could do more to make the two sub-products work together so that a single solution could provide both the TCO and the performance improvements at the same time. However, EMC FAST is a product that provides the TCO improvement promised, and doe it with a clean and elegant solution.

Finally, a little on the future. With the cost of Flash memory coming down 50% year over year, it will soon reach the same price point that we currently see 15K HDD's at. Once that happens one has to wonder what role 15K HHDs will fill? If 15K HDDs are, indeed, squeezed out of existence by this reduction in the price of Flash memory, what purpose will 3 tiered automated storage tiering fill? Or, will the future simply be 2 tiers of storage, one that provides bulk capacity, and one that accelerates the performance of this bult capacity? if that predication is correct, then FAST VP will have a limited life, and FAST Cache and FlashCache will be the longer surviving technology.

Flash Storage and Automated Storage Tiering

2011-05-20T11:12:00.000-07:00

In recent years, a move toward automated storage tiering has begun in the data center. This move has been inspired by the desire to continue to drive down the cost of storage, as well as the introduction of faster, but more expensive storage in the form of Flash memory in the storage array marketplace. Flash memory is significantly faster than spinning disks, and thus it’s ability to provide very high performance storage has been of interest. However, its cost is considerable, and therefore a way to utilize it and still bend the cost curve downward was needed. Note that Flash memory has been implemented in different ways. It can be obtained as a card for the storage array controller, or as SSD disk drives, and even, as cache on regular spinning disks. However it is implemented, it’s speed and expense remains the same.

Enter the concept of tiered storage again. The idea was to place only that data which absolutely required the very high performance of Flash on Flash, and to leave the remaining data on spinning disk. The challenge with tiered storage in the way that it has been defined in the past was that it meant that too much data would be placed on very expensive Flash since traditionally an entire application would have all it’s data placed on a single tier. Even if only specific parts of the data at the file, or LUN level were placed on Flash, the quantity needed would still be very high, thus driving the costs of for a particular application up. It was quickly recognized that the only way to make Flash cost effective would be to place only the blocks which are “hot” for an application in Flash storage, thereby minimizing the footprint of Flash storage.

The issue addressed by automated storage tiering is that you no longer need to know ahead of time what the proper tier of storage for a particular application’s data needs to be. Furthermore the classification of the data can occur at a much more fine-grained block level rather than the file or the LUN as with some earlier automated storage tiering implementations.

Flash has changed the landscape of storage for the enterprise. Currently, Fash/SSD storage can cost 16-20X what Fiber channel, SAS, or SATA storage can cost. The dollars per GB model ends up looking something like the following:

However the IOPS per $ model looks more like this:

The impact on the tiered storage architectural model of Flash storage has been, in effect, to add a tier-0 level of storage where application data is placed that requires extremely fast random I/O performance. Typical examples of such data are database index tables or key lookup tables, etc. Placing this kind of data, which may only be part of an application’s data, on Flash storage can often have a dramatically positive effect on the performance of an application. However, due to the cost of Flash storage the question is often raised, how can data centers ensure that only data that requires this level of performance resides on SSD or Flash storage so that they can continue to contain costs? Furthermore, is there a way to put only the “hot” parts of the data in the very expensive tier-0 capacity, and leave less hot, and cold data in slower, less expensive capacity? Block based automated storage tiering is the answer to these questions.

Different storage array vendors have approached this problem in different ways. However, in all cases, the object is to place data at a block level, on tier-0 or Flash storage only while that data is actually being accessed, and then to store the rest of the data on lower tiered storage while the data is at rest. Note that this movement must be done at the block level in order to avoid performance issues, and to truly minimize the capacity of the tier-0 storage.

One approach used by several storage vendors is to move blocks of data between multiple tiers of storage via a policy. For example, the policy might dictate that writes always occur to tier-0, and then if that data is not read immediately it is moved to tier-1. Then if the data isn’t read for 3 months that data is then moved to tier-2. The policy might also dictate that if the data is then read from the tier-2 disk then it is placed back on tier-0 in case additional reads are required and the entire process starts all over again. Logically this mechanism provides what enterprises are looking for, minimizing tier-0 storage and placing blocks of data on the lowest-cost storage possible. The challenge with this approach is that the I/O profile of the application needs to be well understood when the policies are developed in order to avoid accessing data from tier-2 storage too frequently and generally moving data up and down the stack too often since this movement is not “free” from a performance perspective. Additionally, EVT has found that for most customers, data rarely needs to spend time in tier-1 (FC or SAS) storage, that most of the data ends up spending most of it’s live on the SATA storage.

Therefore as the cost of Flash storage continues to come down, the need for the SAS or Fiber Channel storage will continue to decline, and eventually disappear leaving just Flash and SATA storage in most arrays.

Another approach that at least one storage vendor is using is to avoid all the policy based movement and to treat the Flash storage as a large read cache. This places the blocks that are most used on tier-0, and leaves the rest on spinning disk. When the fact that the sequential write performance of Flash, SAS/FC, and SATA is similar is taken into consideration along with a controller that orders its random writes, this approach can provide a much more robust way to implement Flash storage. In some cases, it allows an application that would not normally be considered a good candidate for SAS or Fiber Channel storage to be able to utilize SATA disks instead. In general, this technique de-couples spindle count from performance thus providing more subtle advantages as well. For example, applications which has traditionally required very small disk drives so that the spindle could would be might (many, many 146GB FC drives, for example) can now be run on much higher capacity 600GB SAS drives and still provide the same, or better performance.

Overall, automated storage tiering is becoming a de-facto standard in the storage industry. However different storage array vendors have taken very different approaches to the implementation of automated tiering, but in the end the result is uniformly the same. The ability of the enterprise to purchase Flash storage to help improve the performance of their applications while at the same time continuing to bend the cost curve of storage downward.