I submit that CIOs today are looking for return on invested capital (ROIC) with an ever increasing “impatient” pace and I agree. Return on Invested Capital is a financial measure that calculates how well a company generates cash flow relative to the capital it has invested in the business.

When referencing ROIC you are in essence measuring how well a company is using it’s money to generate returns. I say why can’t or wouldn’t we use that as a measurement in our calculation of the success of a investment being made in the data center. I ask because we have all seen organizations invest significant capital in infrastructure only to see that equipment sit in boxes or in a lab doing burn in or waiting for some other integration component to be certified/tested for production.

The time this investment sits is time that it is not producing a return for the business. The objective should be to compress this timeframe so that the projects a CIO and their staff have planned can provide the business function they intended to serve. This is why I have taken the ROIC measurement and said, lets apply it to technology, specifically the data center and use it as a factor of determining the standard for success.

To articulate my point relating it back to an environment you may be familiar with, I need to introduce my long time friend Walt. I have worked with Walt for many years. Walt and I have probably worked together on 25 million dollars worth of IT projects over the last decade. Walt is a brilliant IT administrator with skills in networking, compute, applications, virtualization, storage and security. Walt knows his infrastructure well and is one of a handful of guys that his CIO goes to for projects where new services are being rolled out to his organization.

Walt is highly experienced in his trade and genuinely knowledgeable about how each of the technologies, interact. In the early years Walt and I had had some very spirited discussions and I cautioned myself from loosing my mind and telling him exactly how frustrated I was with his insistence that nothing my company made or any other for that matter could satisfy his requirements. Walt’s position was that the infrastructure he had was good enough because he and his staff understood how to support it. Walt wasn’t against change, he was against blind rapid change with no experience or trust that there was sustainability and support. He was also constantly implementing new projects because of the organizations rapid growth and lifecycle changes on products that were already in production but he was cautious to introduce technologies outside of a standard.

In the first years of our relationship, Walt regularly had equipment that was End of Life and End of Support (EoL and EoS) status. He was only replacing that gear months after EoS. That practice ended quickly after an outage of a EoS piece of equipment caused an extended outage where leadership said, we won’t do that again. From 2002-2006 Walt actually purchased new technologies 6-12 months following initial release, depending on the needs of his organization. After purchase, implementation would lag 3-6 months and in some cases greater. In the last 4 years virtualization has become a mission critical component of Walt’s data center and purchases of infrastructure have become synchronized and in some cases technologies are being purchased only months after initial release. Those purchases include, network, storage, compute and virtualization releases.

Prior to virtualization being a mission critical component to Walt’s data center, purchases were staggered based on lifecycle of the individual product. Today Walt’s organization makes investments based on the lifecycle of the entire virtualized solution. Walt’s challenge is manpower and time against the backdrop of the thoroughness needed to establish specific organizational standards for the solution being implemented. Solutions which are implemented must be classified as production and configurations must be standardized to enable mission critical operation. This production “mission critical” classification is beyond any individual product, it is a solutions classification that encompasses all aspects of function from operation to change/modification to troubleshooting and ultimately system recovery.

This is the lifecycle of operation that Walt is concerned with. Showing a demo of how something works never was good enough for Walt. Walt wanted to see it break because he and his staff needed to know how to fix it when it did break. Walt was reluctant to introduce new Vendors into production workflows without some experience with how well support and parts replacement processes would function.

Virtualization changed Walt’s world, in many cases it enabled him to do more things quickly. In other cases it created a situation where his ability to realize a quick return on the invested capital was impossible. Once the infrastructure was built, Walt could respond quickly, but it took significantly greater time to production classify the new platform because of so many new components coming online at once.

Now, Walt wasn’t focused on ROIC and his CIO wasn’t either, at-least in a form they realized. I remember numerous meetings where we would meet with the CIO and he would ask when the new virtualization project was going to be operational. Lets define what their most recent virtualization project was to get an idea of the scope.

Virtualization Project

• Compute – New Generation Servers
• Virtualization – vSphere 4 (current production is ESX 3.5)
• Network – New 10GbE Top of Rack Switches (current production is End of Row 10GbE)
• Storage – New Generation Storage Processor, New Disk Types and Size
• Application – Exchange 2010, Small SQL 2008, Virtual Desktops

In the past Walt would introduce new servers or switches to address smaller tactical needs in the data center with a server here or a switch there. The point is that the implementation of new technologies in the past was staggered versus all at once. This gave Walt and his team time to dedicate brain cells to operationalizing the new component.

Virtualization synchronized the technology spend and now Walt is building platforms inclusive of Compute, Network, Storage & Applications. He must learn and operationalize all new systems, applications and equipment while delivering the same or greater quality that he did when he was just focused on a single new technology.

Organizations which have dove into virtualization all see their technology investments cycles synchronizing because the solution is a platform which is dependent on other components to function more so now than ever before. If your organizations technology investment life-cycles have synchronized and you are implementing these solutions it is taking you more time to operationalize the entire solution because of the interdependencies of the solution as a whole.

This brings us back to Return on Invested Capital. When Walt’s CIO was asking him when the new virtualization solution was going to be operational, he was asking when the investment he approved was going to provide a production service back to the business so that he could communicate back to his leadership that the capital was at work.

A CEO in many cases doesn’t draw a correlation between the $150K firewall, IT invested in versus the $500K VMware project. The firewall was purchased and installed in a 1 month period with no professional services. The VMware project has $100K of professional services and it is apparently going to be difficult to complete in 6 months. In many executive minds the time to operate versus investment is out of sync for these projects. I submit that they are right.

They are right because IT staffs and integration partners are focused on the wrong thing. They are focused on the operational integration of numerous components of a solution that often have too many knobs to tune. Engineers circle on the best options and time circling is directly proportional to the time it takes to implement the solution. That time is exaggerated exponentially when technologies are either new to the market or new to the engineers working on the project. In many cases people are learning as they go, as Walt has done many times over the last decade. What paralyzes us all is that engineers (like myself) often teeter between choosing options that provide the highest performance versus stability. I was a fan of Home Improvement (TV Show), I believe in “more power”, faster is better. However, we all know that faster sometimes isn’t the most stable.

This is not to say that it is any IT staffs or integration partners fault. They are simply doing what is required to do in order to get the job done. In many cases they do it well but there are elements of the implementation timeframe which simply cannot be compressed because there are far too many unknowns. This is the manufacture of the components fault not the customer or partner, plain and simply put. After all don’t we as manufactures always push customers to buy the latest and greatest?

Enter the Vblock Revolution. Vblock is a pioneer in the technology industry where 3 industry leading companies (Cisco, EMC & VMware) have came together to make Walt’s and partner integrators lives easier. We do the integration and testing work, determine firmware, configurations and operating systems (on the infrastructure components) and sell all components as a single integrated solution. It is not a model airplane were assembly is required. Vblock is a fighter aircraft with your own custom cockpit and choice of engines and weaponry.

The intent is to quickly enable the deployment of that virtualization platform so Walt and his partner integrator can immediately go to operationalizing and building out Virtual Machines to support the business. Oh and I forgot to say that VCE has staffed the support organization with experts from all three principal companies, providing a single support number for Walt or the partner to call.

Walt and his partner now spend time on automation, provisioning and security policy, not integration and infrastructure deployment customization. Walt was previously spending time testing and building the infrastructure, playing custom integrator. In some cases decisions Walt or a partner were required to make for certain elements of Walt’s environment went against the best practices of one manufacture or another. This put the organization in the unenviable position of having a unique configuration with potentially limited support by one or all manufactures.

The Vblock architecture has made this dilemma go away for Walt. He knows that ordering a Vblock equals pre-determined configurations based on best practices which all three manufactures have jointly tested and support. Walt is not spending endless hours opening 4 different support cases with various support engineers making ready, shoot, AIM troubleshooting suggestions. I am not taking shots at any independent support organization because they all have the same issue, if they don’t have visibility into the entire system, they can only suggest configuration alternatives to solve the problem the system is experiencing based on their view of that problem. This is the revolutionary aspect to Vblock, it is viewed, supported, roadmaped and exists as a integrated product with certain options for configuration.

Now every criticism I ever had for Vblock and those I have ever heard is that its sizing guidelines or form factor doesn’t fit every customer situation. After my own education and understanding of the solution I have a new opinion. It is that Vblock simply can and does fit most any situation you can throw at it. There are always corner cases to the extreme but I am comfortable in stating this because of the following logic.

When you buy just about any solution, outside of the technology industry you might have the option to customize it but you are selecting one model versus another based on price and features to generally accomplish a required function specific to your need.

For example, I just bought a new coffee maker. It has 24 hour advance programing, I would have preferred a weekly schedule but 24 hour will do. I did like the thought of the shower head feature which evenly distributes water over grounds but couldn’t really tell the difference between coffee made with that feature versus one that didn’t. I assume that if I were to take the technology logic I would have asked for that feature to be removed because I didn’t really need it and it would save me some amount of money. In general coffee makers or products that fall into this sort of investment category provide a specific set of features, driven by a particular market segment and there are limited options when it comes to customization. There are customization options, there just aren’t a great number of them. This type of analogy is synchronous with the logic of Vblock type 0, it is designed to not be tweaked and tuned but more specifically fit a segment of the market with smaller requirements and ultimately fewer customization options. It definitely has options, just fewer of them.

Vblock type 1 and 2 are designed for larger infrastructures and lend themselves more to broader customization. To emphasize this example, about 8 years ago my wife and I built our dream house. That house was a custom home and more on the order of magnitude of what smaller companies may invest in their data center.

The custom home building process was an exciting one, decisions to make everywhere, yet all decisions were derived from standards the builder had established. Any deviation from the standard was okay but we had to go through a process of making sure it worked for the design of the house. I want to be clear, we chose the floor plan, the size of the house and what we wanted the style of the outside of the house to look like. The standards were components of the build from types of flooring, molding, lights, ceiling height, outside stairway to basement, spiral staircase etc..

Many items we accepted the standards, other items were logical upgrades “easy decisions”. Each Vblock has these sort of minor upgrades which are functions of additions or subtractions of certain components. In the case of subtractions our builder would not let us build a house without a roof, we won’t let you build a Vblock without one either.

At one point in our building process we decided we needed a three car garage and the decision to have that size of garage dictated a slight change to the roofline of the home. A little later in the process we decided that the powder room on the first floor was a little small and it needed to be bumped out. This change required all sorts of interesting changes to the house, foundation and roof. All in all the house was a perfect example of the solutions we are trying to build today in many data centers.

Data Centers and the infrastructure placed in them must be built proportional. You want a bigger garage, we need to expand the roof line. You want a bigger room on the first floor, we need to expand the foundation to support that room. If you didn’t take this approach you would have custom homes built that literally would fall over because of rooms that weren’t proportional to foundations and roofs. Now you can take this example to an extreme and tell me that there have been houses built on the edge of cliffs with balsa wood and they worked fine. I am sure they did but they weren’t built in mass, and if the wind blows too hard one day, that house is gone.

In the world of our data center the example may be a few hundred users and the apps they are accessing have some sort of event which spikes utilization and if the infrastructure is not designed proportionally to handle the spike, you could tip the entire thing over. The Vblock design principles are to simply avoid that scenario.

So for lack of a better description Vblock is a custom home and the choices you make for your environment are variables between the minimum and maximum configs within each Vblock. You may choose to upgrade from a standard, and doing so may dictate a change required elsewhere. This is to ensure architectural stability and guys like myself from Cisco (Solution Architects) and vSpecialists from EMC play the role of architect to make sure that your data center is architecturally sound.

It is my belief that we have entered a new day in consumption of data center architectures. That consumption is driven by the business necessity to establish a RAPID Return on Invested Capital (ROIC). Vblock is the architectural answer to the integrated virtualized data center problem. IT Administrators and the CIOs they work for understand this and welcome the approach. I for one am excited to be apart of a new conversation that spends less time talking about what and if and goes directly to how and why.

More thoughts and ramblings soon,
Trey

I am settling into my new position at Cisco and life is starting to establish a new cadence, this post is from inspiration gathered in a new perspective I have for competitive education. I say competitive education because at Cisco I am now exclusively dedicated to The Virtual Computing Environment Coalition (VCE) and prior to Cisco I was in the storage industry but working for a storage manufacture competitive to EMC, NetApp. Prior to joining NetApp I really didn’t have significant experience in the storage industry I had just used NetApp for a few virtualization projects and really liked the product. That thought hasn’t changed by the way, I still like NetApp products.

On the competitive education front, every manufacture educates its employees on the products in the marketplace that will be positioned against their product. That education from anyone in this forum is often laser focused on weaknesses and dismissive of strengths. I had a healthy heaping of this in terms of EMC capabilities so coming back to Cisco and having the potential to be dedicated to VCE was initially a uncomfortable position. I forced myself to look beyond that competitive perspective and said I must know for myself. I was also proud to return to Cisco, who I worked for in the 1990s and no one can argue that Cisco’s position in the history books is one that “could” rival General Electric’s contribution to business. Safe to say I felt very good about supporting a business endeavor Cisco was offering. So I have now been back at Cisco for a solid month and that time has had me heads down learning through “hands on education” and “original source manuals” versus excerpts posted in emails.

I write this to do my best at articulating how changed I am in establishing opinions without an “original source” perspective. To state more clearly I am forever against competitive education as the exclusive source of information about what a product or solution can or cannot do. Competitive education is critical to preparing for what solutions you are up against but in my case it can never be the sole source knowledge base.

I have always tried to focus on what strengths the solutions I represent can do but you often get into situations where the preverbal statement will be made “blank said that your X product can’t do this”. Anyone in the technology industry that says they have never been pushed to the brink of saying “That is complete rubbish and could you ask them about why blank’s Y product won’t do this”, is well not being forthright or they simply haven’t been in the business long enough. I venture a guess that there are a few exceptions but when making statements based on averages, I feel I am safe to make the previous.

The exchange described above is just an absolute waste of time in actually addressing the needs the customer has. Sure architecturally, if product X doesn’t do something that the customers requirements plainly state then there is a functional gap. However, there is usually always an alternative answer to satisfy that requirement. It is then in the customers hands to ultimately decide that the solution presented to them “in totality” can address their technical needs and business requirements.

I am proud to say that I have now had in depth hands on experience with two highly competitive manufactures and both products are excellent products. Products however do not address the needs of business and CIO’s which run their IT organizations.

This is the reason why I returned to Cisco and ultimately dedicated my focus to VCE. Customers want integrated, installed, serviced and architecturally synchronized solutions. VCE and VBLOCKs are laser focused on that objective and in my brief tenure but seasoned perspective this solution is wildly exceeding expectations today, with a future that is exciting.

More on why soon,
Trey

I am attending Cisco Live this week. It was nice to see so many old friends around the show floor last night. As the conference goes into full operation there will be several things to blog about. I hope to capture a few key partner solutions on the “World of Solutions” floor sharing with you their solutions and how I feel they will help us in our mission to data center transformation.

If you are at the conference please send me a email:trlayton@cisco.com or twitter:treylayton. I will be happy to give you a tour of our data center technologies that we have displayed on the show floor.

Also very excited for the presentation by our CEO John Chambers today. John is always a great speaker and I expect the main stage to be packed while our customers and partners listen to his perspective on our industry and Cisco’s perspective on how it is changing.

Look me up, very excited to chat….

Trey

Today’s message is about goodbyes and hellos.

For the last few years I have had the pleasure of working with a great sales team at NetApp with customers from around the world. I joined NetApp after a few virtualization projects in the mid 2000s which I designed data centers that incorporated the use of NetApp storage and Cisco networking with VMware being the core virtualization technology. Prior to that I worked at Cisco during several events which I have often referred to as convergence events. That is essentially the phenomena in our industry where two technologies combine with the promise of a new and ultimately better solution, once the solution and its foundational technologies have matured.

Previous Convergence Events

• Mainframe communications transport with DLSW
• Voice over Frame Relay (VoFR) and ultimately Voice over IP (VoIP)
• IP Telephony (IPT) and Now Unified Communications

We are in the midst of another convergence but this time it is with many technologies, all at once. Think about the many different convergence events we have going on in our industry.

• Server Virtualization – The convergence of workloads on x86 hardware.
• Unified Fabric – The convergence of IP and Fibre Channel on one wire to the server and ultimately to storage
• Storage Virtualization – The ability to abstract the relationship of volume/lun/datastore from a direct hardened identity within a disk subsystem, allowing for various means of non-disruptive and sometimes dynamic relocation and resizing.
• Network Virtualization – The use of some network partitioning technologies (VLANs) and/or Extending Ethernet LANs beyond traditional data center boundaries (Cisco OTV)
• Compute Virtualization – From I/O Virtualization, Combined Fabric, Network and Management to abstracting the physical identity of a server into a profile. This abstraction enables the assignment of server identity not something that a manufacture decides when they build the product but something an administrator creates at the time of design and applies as infrastructure needs dictate its use. (See: Cisco UCS Service Profiles)

Each convergence event produces its own ecosystem of partners, manufactures and integrators. Innovation typically accelerates because of the ecosystem activity being so fertile and if relevant to the marketplace, customer demand is high. What is different this time is innovation is happening in numerous technology areas and each of those areas are finding it necessary to partner with a broader mission and set of goals. These technologies are coming to the data center in a coordinated and often complimentary model.

These technologies and the use of them together has produced an entirely new category of technology conversation “Cloud”. It’s funny the word cloud on one hand frustrates many industry veterans and excites others. Cloud is more cerebral than it is tangible and I believe our journey to make cloud tangible for the worlds enterprises is the opportunity of the next decade.

I was recently a panel speaker at a CIO Summit in Philadelphia, PA. During this event the topic my panel was asked to discuss was cloud. Everyone at the conference was struggling for a definition of cloud. One manufacture presentation referred to cloud as a product offering for one of his product lines. I didn’t agree with that thought and in my session said that I believed cloud was the pluralization of virtualization, in the context of the data center, its infrastructure and the services it provides.

Cloud is coordinated, deeply integrated virtualization, in infrastructure and/or services from an external provider, deployed to an enterprise. It is in effect the attempt to drive efficiencies out of IT services in the context of the enterprise from a holistic perspective versus one technology element enabling those services. No one thing is cloud, but cloud is a thing.

When Cisco announced it’s investment in VMware several years ago then introduced the Nexus family and more recently introduced Cisco UCS, I knew that Cisco understood the many areas of convergence which were happening. I knew that they understood the cycle of innovation sweeping the data center landscape and they were innovating technologies outside of their traditional markets to capture the transition which was/is occurring.

I am happy to announce that I get to go be apart of that transformation. My final day at NetApp is today and while I leave behind high praise and gratitude for the company and people I worked with, I am excited to venture into a new opportunity.

Part of the new role will be working in the VCE Coalition and helping customers adopt these many layers of virtualization concepts and technologies in holistic infrastructure solutions. I received a message from a Cisco employee whom I have worked with for many years and he simply said.

“Welcome Home”

I have never been that far away but it does feel like I am going home.

Thanks to NetApp for all the great times. Everyone stay in touch and keep up the great work. Looks like I will also be at the Charlotte VMUG, which is shaping up to be an excellent event.

Trey…

It has been a while since I have posted anything. All Good reasons why, in that I have been kept very busy by my employer “NetApp” on some great work with customers on several very advanced technologies through the use of things we have worked very closely to integrate with Cisco and VMware, more on that in the weeks to come. I have kindof been under a information embargo and many of the things I want to talk about, well I just couldn’t/can’t. Finally, nothing else was interesting to me or sparked the passion meter, but the toys at home always do. So thanks to Duncan @yellowbricks for the idea.

In my normal dig through the very excellent technology blogs posted throughout our community I was visiting Yellow Bricks post “My Homelab” and thought to myself. I do alot of work on customer architectures and designs and most of that work is done at home, so it might also be kindof cool to document my contribution to my local power companies profits. I have some great tools to make sure that systems are off when I am on the road so I am not unnecessarily burning power. Now I don’t own all the gear, as much of the storage is on registered loan through my management at NetApp and is put to good regular use being recycled on occasion for a latest and greatest version. I have however purchased a decent amount of Cisco equipment over the years to include my entire phone system, which runs CallManager Express on a Cisco 2800 Series router with IP phones throughout the house.

Then there is the new stuff like a Nexus 5000 which Cisco has so graciously let me use for a brief period of time to work on some design concepts for some very key joint customers of ours.

When I built the house a few years ago my condition for building was that the builder let me do the cabling for everything in the house (except power). One weekend a friend and I, along with a few of our favorite beverages wired the house with over 10,000 foot of CAT6 and about 5,000 foot of RG6 satellite cable and a few runs of Fiber to my office upstairs. The other condition was building out a sub-panel for power and getting two dedicated 20-amp circuits to my communications closet. The builder thought I was nuts and the wife already knew I was so she just didn’t care < I love her.

Another piece that just worked out awesome was that we had designed the kitchen with an area that we just didn’t like once the kitchen started to come together. The redesign of that area produced an extra cabinet and a decent size piece of granite that fit nicely on the cabinet, no longer needed. It was a perfect addition to the communications closet for the token monitor, keyboard and Airport Extreme – access point of which there are now lots, even one mounted in the garage and another in the sub-ceiling near in the back of the house for those days when I want wireless coverage in the front yard or at the pool. You have to have outdoor coverage for Cisco Wireless IP Phones.

I am not a neat freak by any stretch, but cables are evil. So I mounted some hangers in the communications closet to serve as collection points for different types of cables because they seem to mate and reproduce but when hanging them, they tend to not get too wild.

As we venture to the my office it has so many direct connections to the communications closet that I am actually embarrassed to say, so we will just leave that one alone and just say that all of the rooms in the house have at-least 8 CAT6 drops. My office has accumulated quite a bit of equipment on its own and I have managed to have a dedicated 20AMP circuit for it as well but I try not to run much heavy equipment in there and use it to power a wall of monitors that I somehow can’t bring myself to the realization that I have enough. There are 4 24inch NEC’s, 2 of which are my my newly acquired Mac Pro and 2 for my Mac Book Pro or any other system I have in the room needing to run. I also have a PC hidden the desk for those moments when having a physical PC is required for whatever reason. I find that it is most often used when I call technical support for something and they tell me that they either don’t support what I am doing on a Mac or in a VM so I say fine, here ya go -brand new PC, doesn’t work there either please help me.

Now you have seen the RG6 cables for satellite and one may wonder why in the heck does he need that many satellite cables, so while it is not related to the lab it is still my pride and joy and yet another example of how perfect of a wife I have. She has allowed me to place in our great room a 65 inch LCD with two sister 50 inch LCD mounted below and a 100 inch project TV (not pictured) on the other wall in the great room. All of these are attached to my IP enabled HDMI matrix switch which allows all of them to play the same channel or serve stations from several DirecTV receivers. During the regular season for the NFL and March Madness my house turns into Man central and we just don’t miss a thing. Superbowl Sunday, coming up shortly, is also a very fun time and I can’t wait to crank the The Who at Halftime on the accompanying Denon THX System.

Finally, there is my buddy Peyton. He is a 18 pound domestic short hair cat that well, he could be a dog. Peyton will fetch certain toys and loves any Mac at which he finds himself always leaning on a keyboard and just enjoying the clicks as I type.

I hope you enjoyed the window into the the home-office. Gotta work harder to get more posts out on the site.

Until Next Time, Trey

A technology that I have heard about in the past but haven’t investigated deeply was SR-IOV.

The research done for the last two posts placed the technology in a more strategic light for me personally. I feel that this technology will soon be in the mainstream of enterprises and this technology will be added to our vocabulary of exponentially increasing technology acronyms.

SR-IOV (Single Root I/O Virtualization) is what I consider to be a revolutionary technology. SR-IOV is a Peripheral Component Interconnect, Special Interest Group (PCI-SIG) specification that many industry leaders are participating in.

PCI-SIG Board of Directors Includes:
IBM
Agilent Technologies
AMD
HP
Intel
LSI
Microsoft
NVIDIA
VTM
Sun Microsystems

This PCI-SIG group is committed to the development and enhancement of the PCI Standard . The SR-IOV specification allows for devices to be simultaniously shared across virtual machines in a physical server.

Okay so why would that be important? Let’s describe today’s implementation of I/O devices in the server virtualization space. I/O devices are our traditional Fibre Channel HBAs and Ethernet Network Interface Cards (NICs). The hypervisor abstracts those devices and creates a layer of virtualization to emulate virtual instances of those physical devices to virtual machines. This is most commonly seen with network cards.

Let’s take an ESX server and use 2 Quad Port 1Gbps Ethernet NICs. The eight ports on those two PCI cards are assigned to various Virtual Switches (vSwitches) in the ESX server. In each vSwitch we create Virtual Machine Port Groups, vMkernels, vMConsole, etc…

These “virtual ports” on “virtual switches” are part one of the abstraction layer. The second part is each virtual machine’s own vmnic. The virtual machine has its own NIC that the guest OS running in the VM thinks it owns. The administrator assigns the virtual machine NIC to the appropriate Virtual Machine port group which enables access through the physical ports of the server.

The picture below depicts this in that the left side of the figure are virtual ports and the right side are the physical ports which the virtual ports share for network access. This is our layer of abstraction.

The layer of abstraction affects performance by requiring the physical host CPU to sort and forward each frame, which traverses this layer. VMware has done an excellent job of making this abstraction layer very high performing. However, the greater load that we place on the abstraction layer, the more CPU resources we are going to use. This is typically not a factor in what I would estimate to be 70% of the workloads enterprises are virtualizing. There is another 30%, which represent the highest performing, most critical x86 workloads, in the enterprise. These workloads are SAP, Large Exchange Infrastructures, and Oracle etc….

This is not to say that you cannot virtualize the aforementioned applications, you can. However, there is typically always a conversation that starts with the application owners regarding performance when those applications become targets of virtualization.

The promise of SR-IOV or I/O virtualization takes the complexity out of that conversation. Intel and other chipmakers, in recent years, have announced technologies imbedded in their chipsets that provide hardware assist to virtualization workloads. Said differently, chipmakers have been developing technologies that virtualization workloads can leverage to run more efficiently. I/O Virtualization is simply another progression in that evolution.

So let’s get to this and the impact to storage. First let us take a new 10Gbps Converged Network Adapter NIC and place two of them, for redundancy, in a server. Now we install our favorite hypervisor on top of the hardware (ESX in this example). The converged network adapter is going to run Data Center Ethernet, thus this card is able to transport a FCoE workload as well as a traditional Ethernet workload (supporting IP based applications).

As we begin to build virtual machines we now take a virtual slice of each Converged Network Adapter and make two redundant Ethernet NICs. We take a second slice of the Converged Network Adapters and we make two 4Gbps HBAs for storage. This new HBA will be presented to the virtual machine to mount LUNs for a database we are going to be running.

To the PCI tree of the virtual machine, it thinks it has 2 Ethernet NICs and 2 4Gbps HBAs. Drivers are installed in the virtual machine operating system to recognize each HBA and Ethernet NIC. All of that is important but of most importance is that the operating system running the drivers for these I/O devices is speaking directly to the hardware.

Through driving the hardware natively, the operating system can take advantage of hardware features native to the card. There is no longer a need for a CPU to sort frames and emulate NICs. Our result is now the ability to run high performance enterprise application workloads and likely have them perform better as virtual machines, than as physical machines.

This technology also provides numerous other advantages like the ability to switch a device type.

Possible examples include:

Converting 1 to 10Gbps Ethernet or 4 to 8Gbps Fibre Channel.

Other interesting possibilities include the change from an HBA to an Ethernet Interface. The I/O is virtual so this is possible as long as the hypervisor would support it. Software features in hypervisors will evolve where hot insertion and removal could be supported.

What is maybe not completely appreciated from this description is that we can make these changes without changing a physical cable.

Beyond the one wire concept we also have a near limitless supply of I/O devices, which can now facilitate migrations from one storage array to another or from one storage protocol to another.

We also know that 40Gbps and 100Gbps Ethernet is around the corner. The application of I/O virtualization on 40Gbps and 100Gbps NICs present a practicle application of this speed of interfaces in the data center, on a server. The use of a 40Gbps NIC might allow a administrator to have several virtual machines running with 10Gbps NICs and others with 1Gbps combined with 4Gbps and 8Gbps Fibre Channel connections, all on a single physical NIC.

Now this is a topic for another time but what is so fundamentally exciting about Data Center Ethernet (DCE) is that it provides a service level architecture, which will enable the management of the inevitable oversubscription of the physical uplinks by Virtual I/O devices. Hey, it’s thin provisioning for I/O.

Possibilities are limitless, similar to the possibilities that server virtualization presented to the server of the past.

More on this in future posts but thought I would share what was top of mind.

Update: Neterion Releases I/O Virtualization 10G Card

Trey

The Cisco UCS system is proving to be loaded with a few thought leading technologies.

Official Product details appear to be scheduled for launch on March 31st, 2009
http://www.thedcofthefuture.com/

Much of this detail, minus a few specifics, can be gleaned out of the At-A-Glance data sheet. Taking that data along with a nugget or two here and there I have managed to assemble the following. I believe this provides a simple description of the product and until Cisco truly pulls the covers off of it, I am not sure we can assemble more. If you see something wrong or something to add, let me know. One thing I am trying to research more is the UCS manager, nothing more than 100,000 foot dialog about it, so it is not included in this post.

Cisco UCS B-Series Blade Servers:

2 Blade Servers Available on First Product Ship

Half-Width Blade: 2 Intel Xeon 5500 (Nehalem) Processors supporting 96GB of DDR3 Memory, two optional SAS drives, and a single converged network adapter (CNA) mezzanine slot for up to 20Gbps of I/O throughput.

Full-Width Blade: 2 Intel Xeon 5500 (Nehalem) Processors featuring what is being called “Cisco Extended Memory Technology”. A single full-width blade will support 384GB of DDR3 Memory, two optional SAS drives and two mezzanine slots supporting up to 40Gbps of I/O throughput.

The memory technology is of key importance, apparently this technology takes groups of 4 – 8GB DIMMs and presents them as a single 32GB DIMM to the operating system.

Cisco UCS 5100 Series Blade Server Chassis:

The UCS 5100 Series Chassis is a 6RU blade chassis that will support up to eight half-width Cisco UCS B-Series Blade Servers or up to four full-width Cisco UCS B-Series Blade Servers.

Configurations will also support various combinations of half and full-width blades.

Knowing Cisco the way I do, I expect there to be a few different series blade servers of various socket configurations and blade slot dimensions. Given the architecture you will read about below you will see that there is nothing preventing them from releasing blades which consume two horizontal slots (for a single blade). There is also nothing stopping them from releasing a blade that consumes every slot in the chassis. If they do we will need to nickname it Baby Blue Gene C

Each chassis will contain four redundant power supplies and two Cisco UCS 2100 Series Fabric Extenders. The physical chassis has what is being described as a passive midplane (UCS 6100 Series Interconnect). Power entry appears to be from the rear, with the units being swappable from the front panel. The two fabric extender slots are accessible from the back panel. All server slots are accessible from the front panel.

Cisco UCS 2100 Series Fabric Extenders:

Each UCS 2100 fabric extender has a single 10Gbps fabric connection to each half-width slot. Thus placing two UCS 2100 fabric extenders in the chassis provides two fully redundant 10Gbps unified fabric connections, to each half-width blade. A full-width blade will have four total 10Gbps fabric connections (2 to each UCS 2100). The UCS 2100 will have 4 SFP+ 10GBASE-KR connections to the UCS 6100 Series Interconnect. This basically means that each UCS chassis is capable of having 80Gbps of line rate connectivity to the UCS 6100.

The UCS 2100 receives all configuration and even firmware from the parent UCS 6100. This is similar to the Nexus 2000 relationship with the Nexus 5000. It is assumed that you would likely acquire dual UCS 6100s, to avoid any single point of failure.

Cisco UCS 6100 Series Fabric Interconnects:

The UCS 6100 is the central nervous system of the UCS platform. The UCS 6100 will ship in 20 and 40 port models. These systems are a re-packaged Nexus 5010 and 5020 with a few tweaks for UCS. The UCS 6100 20 port version will have a single expansion slot for additional 10Gbps DCE ports or various combinations of Fibre Channel. The 40 port version will have two expansion slots. See the Nexus 5000 Series for the specific expansion modules, as they are described as the same for the UCS 6100.

This brings up the point at which we begin to reflect on a few things we have learned and we recognize that the UCS 6100 will be the point at which we make FC, FCoE and IP Storage connections. For those systems supporting native FCoE target cards, you will place the storage systems native FCoE ports directly in the UCS 6100 10Gbps DCE ports. Those systems which do not support FCoE target cards will attach via 4Gbps Fiber Channel from each UCS 6100. Once again, expecting there to be two UCS 6100s, in most data centers. There is no 8Gbps FC card for the Nexus 5000 today, thus none for the UCS 6100, but I expect that to be released at some point for arrays which have 8Gbps FC.

The UCS 6100 is also the point at which you would interconnect into your production network via 10Gbps Ethernet. For those environments that do not have 10Gbps Ethernet, 16 ports in the 40 port UCS 6100 and 8 ports in the 20 port UCS 6100 can be down-throttled to 1Gbps Ethernet and a 1000-BaseT SFP can be inserted, once the port has been “down-throttled”.

Observations:

So my first thoughts are wow, alot of power in this system. Starting off with the ability to get 384GB of memory in a single server. This brings an interesting VM to server ratio possibility into play. In most virtualization projects I have been involved with, memory has been a key factor in virtual machine density on a single server.

The use of 802.3ap 10GBASE-KR ushers in a new and exciting approach to backplane engineering. This architecture is the nearest to wire-once we have ever seen. Meaning that the unified fabric transports management, diagnostics, backplane, storage and user access. You simply need to connect the UCS 6100 to your storage infrastructure via FCoE, IP or FC (or all of the above) and then interconnect your production IP network at 10Gbps or 1Gbps. Your infrastructure will grow and applications will move from supporting FC storage protocols to supporting native FCoE or a high performance NAS protocol, such as NFS. Moving from FC to NFS or the opposite does not require the change of a cable. This simply requires the change of a setting in the system (specifically in the software that is managing the CNA). Once the change is made the device is presented to the PCI tree as the device type you need for connectivity (see previous post -3rd CNA).

No solid word yet on OS support but interesting possibilities come to mind with the Nehalem chips (another worst kept secret). Since these chips are in the Mac Pro maybe we can run OSX in this bad boy and I can find some way to jam this into my basement.

Exciting possibilities with this platform and I am excited to see what is next.

Hope you enjoyed the read, more on another topic soon…..

It has been a long time since I have seen so much discussion about a technology announcement. It seems as though every corner of the connected world is discussing Cisco’s Unified Computing System. In a previous post I discussed more of what it means to the modern data center. As the onion is peeled back and we are afforded the opportunity to learn more about what is in UCS, I found that one of the most interesting things were the CNAs. These are the CNAs that are going to be sold with the product. Cisco is calling them Unified Computing System Adapters. We are going to stick with CNAs as the naming convention for this post.

The first CNA is a two port 10 Gigabit Ethernet Adapter utilizing Intel Oplin silicon presenting two 10 Gigabit Ethernet NICs to the PCI device tree of the blade. If one chooses to run FCoE on this NIC you are required to install software drivers to enable the virtual HBA functionality. This will result in likely the least amount of support with storage manufactures, not that it won’t be supported but it will likely not be high on the testing list, read on to understand why.

- The above card is designed for blades that will primarily run workloads that have IP based storage attached. Meaning servers that will use iSCSI or NFS as the primary storage transport technology. These will also be the least expensive cards in the system.

The second CNA is a two port 10 Gigabit Ethernet Adapter utilizing Intel Oplin silicon, yet providing the option of factory installing a Qlogic or Emulux HBA chipset. Allowing the consumer to continue to leverage their certified Qlogic or Emulux HBA drivers as virtual HBAs over FCoE in the system. This CNA presents two 10Gbps Ethernet interfaces and two 4Gbps FC HBAs to the PCI device tree. In a nutshell the native OS running on the blade thinks that it has a Qlogic or Emulux HBA installed but it is actually a virtual HBA encapsulated in FCoE and transmitted out of the Oplin 10Gbps interface.

- This card is designed for those organizations who must stick with a certain compatibility matrix and are committed to continuing to utilize Qlogic or Emulux HBAs.

The third CNA is the most exciting CNA to hit the streets and will be the source of many “how did they do that conversations”. This CNA is also a two port 10Gbps ethernet card but it is being called a Virtual Interface Card (VIC) because of its ability to present up to 128 virtual devices to the PCI device tree. 8 of the devices are used for UCS systems management which brings me back to the other 2 CNAs, each and every one has a management pathway so that firmware and bios can be upgraded and managed through the UCS Manager, more on that another time.

So we are left with 120 virtual devices. These virtual devices can be any combination of HBAs or Ethernet Interfaces. Why would I need virtual devices? Here is why. Today in a ESX server we have physical interfaces, those physical interfaces are virtualized through the ESX Hyper-visor and the virtual interfaces are essentially emulated in software. The emulation process consumes resources and impacts performance.

The golden goose of high performance virtualization is actually inventing a technology that allows you to present a virtual device that the virtual machine thinks it owns and actually runs drivers in the virtual machine to drive that interface. There are many forms of this but this version is ultra portable and works with vMotion and thus DRS (VMware’s Dynamic Resource Scheduling).

The adapter also supports Cisco VN-Link technology (all of the CNA’s actually do). This basically means that when you vMotion the virtual machine (while in-flight) the traffic statistics, security and QoS policies will travel with the virtual machine. Allowing you to manage the virtual machine as though it was a physical. The Virtual I/O technology will allow the virtual machine to perform as though it was a physical while still allowing you to vMotion the virtual machine. This UCS virtual I/O technology is written in compliance with the SR-IOV specification and represents a entirely new and innovative approach to high performance virtualization.

I am very excited to see what the other server manufactures announce or counter with. There are other technologies which are similar but this is really the first designed for VMware.

Next post will include more details on the blade servers and the fabric interconnects.

stay tuned….

Cisco’s Unified Computing System has caused quite the buzz and while many are working on the specifics of what hardware component does what, I figured it would be worthwhile to write a post about what the promise or reality of UCS is. This is essentially a major transformation or technology shift that is occurring in front of our very eyes. I say this because I watch in amazement as this company Cisco Systems has truly transformed itself and an industry in a few short calendar years.

About ~4 years ago when ESX 2.x was out and true enterprise server virtualization was just starting. There was possibility of what value virtaulization could provide to an enterprise, but a very long road to realize it. ESX 3.0 was released and made the time for virtualization now, as the slogan went. ESX 3.5 was released and the world really took off with manufactures and customers really beginning to comprehend what virtualization really meant for the future of computing.

At VMworld this year VMware discussed the next release of ESX and it prepared us for how the next step, in our dreams for virtualization would be realized. Well I can say that back when ESX 2.x was released, Cisco really didn’t have much of an idea what VMware was and what it meant to the data center. I would say that extended into the initial release of ESX 3.0 for a large segment of the Cisco population. Something changed about 18-24 months ago. Cisco and likely even it’s own IT organization began to wrestle with the realities of what that times virtualization capabilities were. While they wrestled a few smart people were able to comprehend what the the possibilities of this new technology could bring.

Cisco saw a great problem in today’s data center, in that problem they saw opportunity. I previously stated that I am watching Cisco in amazement, I said that because I watch as they execute as if they are lead by the most brilliant generals on a battlefield. The battlefield of technology, where ideas are the smart weapons.

The problem with virtualization has been the compute infrastructure didn’t scale or adapt to what you needed it to, physically. Furthermore, physical didn’t translate into virtual. Think about it, we basically installed ESX on a physical server to fool the underlying operating system into thinking that it was running on it’s own piece of hardware. The process of fooling the physical server left a chasm of masked problems. The hardware infrastructure was dislodged from the virtual infrastructure.

In the past – When we bought a server for virtualization we wanted lots of I/O slots to handle network, SAN, memory and CPU. So we went and bought rack servers. We disliked the rack servers because they took up lots of space and cabling was a nightmare. The replacement of a failed component required two able bodies to remove the device from the rack and replace the failed component.

Blade Servers were our answer to the cabling mess and they handled some of our power concerns. That legacy blade enclosure was pre-wired with power but the lights in the data center dimmed when you powered up that enclosure. In complete fairness the newest of blade enclosures are much more energy efficient. Another problem with blade servers were that they didn’t have the I/O capacity that we needed. Finally, we were nickeled and dimed for every management feature in the enclosure. Things were good because VMware enabled this mishmash of sheet metal, plastic and silicon to be managed through it’s inefficiencies. There was not a choreography of integration between the virtual machine and the infrastructure it was running on. That integration was left completely in the hands of VMware, until today.

The past 10 months have seen some very interesting announcements from Cisco. All chess moves in the technology war.

The products and technologies leading up to UCS combined with the capabilities of the next release of ESX are what make it so powerful.

So, back to the basics, if you buy a server from pretty much anybody you will buy a thing that has a processor and some memory and disk. If you want management, open your wallet and spend some more money. Management is not embedded in the traditional server solution. Management for UCS is embedded, it’s part of the system. The Cisco UCS was designed from the ground floor to all work together and be managed from a single point.

The next thing that Cisco has done is basically take the physical state out of the infrastructure. That is the wire. Cisco UCS presents a world where the customer will “wire once”. The ability to invest in cabling the way you want it, labeled and perfect, without having to touch it again. It brings back memories of how organized things were in the data center when we had mainframes that were the primary computing platform. More on that thought another time……

The next key component of UCS is simply that it is optimized for virtualization. This is precisely what I was leading to before. I am not saying that ESX server just runs faster, research VN-Link for more info. Nope, what I mean is truly integrated and virtualization aware. The ability to manage to the virtual machine, as though it was a physical machine. Customers still struggle today with challenges the virtual world presents because there is no way to get data out of that virtual machine or apply a true end to end security and quality of service policy to it.

Customers are now afforded the ability to customize the compute infrastructure, it will be sized for their workload. Workload is a key word that will resonate through the future compute lexicon. It is used today, but loosely. Cisco UCS has afforded Cisco the ability to have a consultative conversation about a customers application workload. VMware’s next release of ESX server combined with Cisco’s UCS enables that workload to be created, managed and monitored from a single point. Through integrations with leading storage manufactures, Cisco UCS management extends to the storage array.

Today we size storage workloads based on disk I/O, the data center consultant of the future will design the data center based on the compute I/O.

This formula will incorporate:

All things that make up the data center organism of the future, working together to run a high quality business workload on a single unified fabric.

Through, the vast amounts of workload information Cisco collects, they will gain intelligence on how to make their network equipment and unified computing platform better. Cisco’s compute infrastructure will evolve into the reconstruction of the mainframe, in a modern form that scales and adapts dynamically to the needs of the business to which it is serves. CIO’s will have the ability, through the BMC integration to define service level agreements for workloads and guarantee them end to end.

The world just changed and we have a front row seat to history.

FCoE Fibre Channel over Ethernet promises a great new future for block based storage protocols. I have found that in my conversations with many customers/clients that there is a significant amount of confusion around what FCoE actually is.

This post is an attempt to clear some of the basic confusion up. The FCoE technology in itself can spawn many conversations but this one will attempt to stay on track with the basics.

So how did FCoE come to be?

On April 5th, 2007, a group of companies published a press release entitled “Leading IT Vendors Propose Fibre Channel over Ethernet (FCoE) Standard”

The press release announces the concept of creating a new protocol specification to transport Fibre Channel storage traffic over native Ethernet Frames (FCoE) and was presented to ANSI on June 6th, 2007.

So What is FCoE?
If you are really interested take a look at the full specification.

FCoE is a protocol specification that maps Fibre Channel over Ethernet that is independent of the native Ethernet forwarding scheme. Thats alot of words but very important to understand what that means. This means that a FCoE frame traverses the ethernet fabric without utilizing the traditional ethernet process, utilizing broadcasts etc…

This architecture allows FCoE to seamlessly integrate with existing Fibre Channel. The forwarding mechanism is in essence the same as Fibre Channel plus all the zoning rules. This enables you to integrate a FCoE fabric with an existing SAN fabric. The two fabrics are interchangeable and all of the new switches which support FCoE (discussed below) also have card slots for FC connections to enable that convergence point between the two fabrics.

The new ethernet fabric, which was designed for FCoE, is called a couple of different things. The names generally refer to the same principal technology yet the names are usually tied to a manufactures marketing.

For Example:
DCE – Data Center Ethernet (Cisco)
CEE – Converged Enhanced Ethernet (Brocade, IBM)

We will use DCE for the purposes of this post.

DCE includes a few new features that are leveraged by FCoE and can be leveraged by other protocols as they are developed or re-engineered for DCE.

Features:
Priority-Based Flow Control IEEE 802.1Qbb - Provides class of service flow control. Traditional ethernet has flow control but it is one size fits all. Class of Service flow control allows us to give a preference to things being transmitted. FCoE, for example, uses a Class of Service flow control which will guarantee it’s delivery and preference in the event of congestion.

CoS Bandwidth Management IEEE 802.1Qaz - This new specification allows grouping of classes of traffic into lanes of bandwidth, once again ensuring that FCoE or other transmissions are given the preference they need.

Congestion Notification IEEE 802.1Qau - End to End Congestion Management for a Layer 2 Network. The ability to actually have the network notify as points in the network become congested, at the switch port or host.

L2 Multi-Path for Unicast & Multicast - This effectively can allow for the elimination of spanning-tree, in certain environments. Today when we create multiple links between two devices those links are not load-balanced completely or at all. In the case of switch to switch links spanning-tree will block one port and it will not be used for data traffic, unless the primary link fails. In the case of switch to host traffic you effectively need to bond two or more ports together to load-balance via a higher layer protocol. Which brings us back to switch links, you can bond switch links together via etherchannel yet these ports are load-balanced based on source and destination pairs. If you have 4x1Gbps connections bonded together, then one source and destination pair can only consume the bandwidth of one physical link. More on that some other time. L2 Multi-path is what Fibre Channel networks do today. Basically if you establish two connections on a switch to a host, FC will load-balance. L2 Multi-Path is a game changer and not alot of people understand it yet.

Lossless Service - This is key to FCoE being successful. Traditional ethernet discards frames when congestion occurs and the higher layer protocols will take care of ensuring the data gets retransmitted. Said simply, you can’t do that with FC traffic. The fabric (in this case Data Center Ethernet) needed to have an ability to enable a service that could guarantee the transport of data.

What do you need?

The above are the features of DCE now what do we need to run DCE and ultimately FCoE over DCE. Basically, you need new hardware. That hardware is detailed below.

Switch hardware - Cisco has released it’s Nexus product line. The Nexus 5000 is a Data Center Ethernet class switch which can deliver the lossless service and has ASICs on board that can forward frames with latency numbers that are far better than we have seen in any ethernet switch ever released. Brocade and others are planning releases of FCoE capable DCE/CEE switches.

NIC hardware - Making FCoE work requires supporting DCE features, end to end, and this obviously includes the host connection. A new class of NICs has emerged and those have many different marketing names as well. We are going to call them CNA’s (Converged Network Adapters). A CNA is 10Gbps Data Center Ethernet adapter (supporting features mentioned above) and designed specifically for transport of FCoE frames. Now lets crack open a CNA card and have a look under the covers, in a simplistic manner.

Here is basically what it is.
1.) 10Gbps DCE Ethernet NIC ASIC
2.) Legacy Ethernet NIC ASIC
3.) Legacy FC HBA ASIC

Basically, you pop this NIC in your server and install the drivers. One set of drivers will install a Ethernet NIC another will install a HBA. Traffic that comes from the “virtual HBA in the server will then be classified and transmitted properly out the 10Gbps connection adhering to the forwarding rules defined in the fabric. World Wide Names (WWNs) all that stuff remains the same from the FC world. IP will run on the legacy ethernet NIC and it will be transmitted out the 10Gbps DCE Ethernet NIC with the proper class of service for IP.

Which brings us to the thing that I find so many people confused about. That last paragraph was the first time I mentioned IP. IP is not apart of the FCoE stack. No IP addresses exist for FCoE components. The only time you need IP is if you want to actually run IP traffic on the CNA adapter. The CNA adapter will however ensure that IP traffic does not get in the way of FCoE. This brings me to a final point that people are often confused about. I have not once mentioned routing. FCoE is completely layer 2, meaning switched traffic, no routing is involved. Remember the FCoE specification allows for an independent means to forward traffic. Our forwarding mechanism for FCoE are the zones we define in the DCE/FC fabric, the same way we use to with FC. IP transmissions can routed on the same wire but that traffic is classfully separated from our FCoE traffic.

Cisco FCoE Video
IBM Redpaper – Introduction to FCoE
Convergence with FCoE and Lossless Ethernet
NetApp First to Announce Support for Native FCoE Storage