State of Data Center Operations

Power in the Data Center and Balancing 3-phase power - some questions and answers

2011-11-18T15:38:00.001-08:00

First, clearing up some misconceptions I've heard and giving out some basic data:

This conversation is for power typically used in the US.
120V power is one phase wire to neutral wire.
208V power, across 2 phase wires, is still considered single-phase.
There is no "two phase" power in common use these days. Trust me.
It is really important to balance use of the phases of 3-phase power.

Most data centers use large UPS systems for the room (before distribution of power to power panels and racks). Smaller server rooms sometimes use rack-based UPS systems (after distribution to racks).

The step-down transformer sometimes used taking 208V down to 110V wastes energy. I ballpark it at 10%. In most server rooms 208V power is used for equipment like servers and storage. Here are a few examples of how that power is often distributed and how to balance power across the phases in each case:

A 3-phase power panel with 208V circuit-breakers wired to outlets near each rack. This method is the usual method in older data centers, and requires an electrician for every change, which can be expensive over time.

Balance power by placing gear in racks using under-utilized phases. For example - Phase A is under-utilized by B and C phases are over-used. Use racks with AB and/or CA to boost use of A more than B and C.

3-Phase from Starline Busway or equivalent to 208V plugin modules which provide outlets. This type of distribution is quite flexible and a bit more expensive up front. No electrician is needed for minor power distribution changes over time.

Balance power with a short outage and switch modules to change phases in use for the outlet.

3-Phase to the PDU/power strip in the rack. The equipment to do this is typically more expensive than other power distribution equipment.

Balance power by changing which outlets you use in the rack. Power must be balanced in each rack.

The real question in balancing 3-phase power is this: how much out-of-balance is too much? Our standard is that power must not be more than 10% out of balance. It is "ok" if a very temporary migration takes power use 15% out-of-balance for less than a few hours.

I have seen power panels 25% or more out of balance. This typically causes some problems:

Heat - this condition is detectable as heat
Circuit Breaker outages - circuit breakers can flip
Waste of power - Power will be induced in the neutral wire and be wasted
Loss of capacity - this wasted power is lost and cannot be used, lowering the maximum capacity available.

Radical Automation

2011-10-20T22:29:00.000-07:00

There are a lot of tools out there to help out operationally.
They help with numerous areas, including but not at all limited to:

config file management
patch and package management
server builds
remote management (even for virtualized/remote console)
monitoring/alerting
data gathering and data graphing (can be separate from monitoring/alerting
CMDB and inventory management
capacity planning/management
performance monitoring/management/planning
security monitoring/prevention/policy enforcement
correlation engines
release tools
lifecycle management (hardware, software, etc)
control management (automated error correction like restarting a failing application instance)
reporting tools

Each of these items is useful in its own right.

When you have enough of these systems, they can seem like a lot to look at, and a lot of overhead. But without them you are blind.

The solution is radical automation.

I've worked in a radically automated shop. This was in the days before so much open source software, in the 90's. We wrote systems that handled most of these issues end-to-end, and were written flexibly to be extensible. We had what I think of as code-perfectionists who wrote self-documented code in perl, no less, that anyone could modify. I started helping do it, and then others who were better at it and more dedicated to this space took over. But even years later I could go and modify our code and automation because it was written so well. But I digress.

Having all of these systems very tightly integrated did many things for us:

Automation means less administrator-caused downtime. Also, less experienced folks could take on more complex tasks that were "wrapped" in automation that safeguarded against most stupid mistakes and "fat fingering" problems.
We had the data we wanted now, and on an ongoing basis. And seeing and using that data, sharing that data, was easily doable.

Weekly reporting that could be changed week to week but otherwise be consistent

We could tackle enormous tasks. When building and deploying a server is entering a tiny bit of data in a database about it and its role, and plugging it in and waiting, it is not hard to deploy hundreds of servers in a day with a few people.
Budgeting and planning were eventually largely based on rules from automated capacity planning.
We knew where and what and how many of everything there was and why.
We did a LOT more to focus on vendor management when we were not so busy staring at our navel.

There are a ton of frameworks for automation, but none replace rolling up your team's sleeves and getting to work on writing some automation. You can start small and grow from there. Pay your sysadmins to be programmers and not just sysadmins.

Flash storage in the data center

2011-10-05T08:24:00.000-07:00

Flash storage in the data center is here for many of us and is coming to the rest like a freight train. There is a dizzying array of options, including bypassing flash and using instead a pile of RAM to consider.

What roles is flash used for, generally:

Primary storage - when you always need blazing fast read performance. I think hadoop clusters and databases when I think flash primary storage.
Caching - when you need a boost to traditional storage performance, especially when using SATA storage (although some use it against 10K RPM disk as well). This is very common for virtualization uses and is also to improve general performance of centralized storage.

Flash is used in 2 form factors:

SSD (solid state disk) - replaces traditional disk in the form factor of a disk. This can be used locally or in a storage appliance separated from a a server.
PCIe - bypassing SAS protocols and throughput limits, flash storage is accessible to your server's motherboard

There is one more dimension of flash storage use: where do you put it?

Locally - put it right in a server
Remotely - put it in a central storage appliance

Both form factors and roles can be involved with both locations of storage.

Flash is generally not so great at sequential writes. Dell is selling SSD storage and has a fascinating write up useful for thinking about flash in general, showing flash is generally not worth the money for sequential writes. As I think about databases and convergence, I think about database storage history. First we used to separate all different database uses for storage into separate physical disk. Then having a big pool of disk became more important. Now we're on the verge of separating out those uses again! But I digress...

Here are a few options I have been involved in looking at or have been hearing about:

FusionIO - mostly used as caching PCIe local storage

They did what every hardware startup wants to do - got past several hurdles to success! FusionIO lined up a first MAJOR customer (rumored to be Facebook) that helped them get to volume pricing, and then lined up an OEM agreement with a major manufacturer of servers (Dell). It clearly works quite well for many/most folks, but does NOT work well for some folks using it as primary storage for Oracle...

Dell MD1220 with 8 SSD's - Local SSD primary storage

Dell's write up mentioned above is interesting! 600GB usable storage with mirroring data protection.

Tintri - A network appliance using non-PCIe storage as a cache

this is via NFS, it is fast, low latency...

Violin Memory - flash and RAM in an appliance hooked up via fiber channel or on the network via iSCSI
Network Appliance - This appliance can use the PCIe-type approach as cache in Flashcache as well as the SSD approach.
Pure Storage - Appliance using flash as primary storage over fiber channel

VMWare Vsphere 5 - a few thoughts

2011-10-03T20:49:00.000-07:00

VMWare has announced the features, and pricing structure, for vSphere 5.

The first item being heavily discussed is the new cost per-used-RAM. We use Enterprise Plus, which gives 48GB used per CPU. If you have an 8-core CPU, then that is only 6GB/core for your license! That sounds bad to some folks. Here is the other side of the story, however. If you have a redundant configuration as many of us do, then the unused RAM is not counted. We found that when we added up our used memory we needed to buy no new licenses.

Update: vmware moved to 96GB/CPU for "old" enterprise plus licenses after the uproar from customers... We became unfazed as our calculations led us to conclude it did not affect us.

RAM prices at a low: It is a better time than ever to use server virtualization!

2011-10-03T20:46:00.000-07:00

RAM: Over the years I have watched as server RAM prices have gone down. However, when I recently priced out a virtualization server with 192GB RAM and the RAM was half the price from 9 months ago, I thought WOW! My purchase power to implement virtualization has increased dramatically! With 2-socket servers with 192GB RAM, there is more compute power than most companies really need for most tasks in total; virtualization of some kind is required for most enterprises to make use of the hardware on the market.

There is no better time than the present thus far to implement virtualization.

New ASHRAE standards, energy efficiency, and HPC

2011-10-01T12:13:00.000-07:00

The new ASHRAE standards, when vendors support them, allow for servers and storage that can be run continuously at a maximum of 113 degrees Fahrenheit. Your support contracts and warrantees are not voided when this is supported.

Dell is at the forefront of supporting these standards for the public. They have made public statements supporting these standards. Most of us have heard about the military running servers in the desert without air conditioning. Milspec is not for most IT organizations, but is paving the way for energy efficiency for the rest of us. It means in many places in the US, ambient air is plenty cold 100% of the time. No more cooling server rooms except as needed for people doing work in server rooms.

I have heard jokes in the past about hanging winter coats in server rooms in warm climates as people get very cold working for long periods in chilled rooms. Now the new joke is shorts, flip-flops, and tanktops on a rack by the server room door as we might start running these rooms much hotter.

How does this affect HPC?

Air cooling at any temperature is only good for racks holding up to about 20KW of power-using dense equipment. Any more than that and you just cannot get the heat out fast enough without help from a more dense material than air or by moving air so quickly that it is not cost effective. If I can run data centers on ambient air cooling, then running air cooled racks is getting cheaper and cheaper. Highly dense racks requiring chilled water or other solutions are becoming more expensive compared to these air cooled racks.

It has been the trend to try to use smaller and smaller server rooms, and get more and more dense. The benefits seem to now peak to 20KW/rack and go down from there. If you do not really need extreme HPC density for performance reasons, extra density is not cost effective.

Methods to retro-fit ambient air cooling into existing data centers are critically needed. There is not nearly enough action in the market in this area.

The cloud: hype and reality

2011-10-01T12:04:00.000-07:00

The cloud. Everyone in IT is in a buzz about it. It means many things to many people, and sometimes seems to mean anything anyone wants it to mean.

First let me say that, if it is not obvious already, this is not a one-size-fits-all world. As businesses tackle the next buzzword concept, it is very important to keep that in mind. One great was outsourcing, which is great for lots of situations, but definitely not all. Now the cloud.

I was asked to ignore all "business problems" and evaluate moving all servers my department runs in the cloud. It was going to cost twice what it currently costs us in terms of costs like equipment replacement and support contracts. Running servers in the cloud is great if you are small and it helps you avoid hiring your first systems administration team, and it is great if you are huge and need help not buying equipment for rare peaks of capacity utilization, especially if you also have to buy equipment for a DR site for those peaks as well.

As far as SaaS goes, it is a clear winner for many cases. If you do not need to run a local email system, sure, outsource it to a "cloud provider". Salesforce CRM and more can be great instead of running it yourself. Examining every service you run and seeing if it could be more effectively run by others is a sensible thing to do. Sometimes software makers move to a hosted model that is more effective.

Running your own "local cloud" can be effective too, assuming you have appropriate control of and planning for your infrastructure. Moving into server or desktop virtualization when you may not have the network bandwidth or central storage IO to handle it is a disaster.

As always, if you do what makes sense, things go well. Do things because "they are the hot thing to do" rarely goes well for anyone.

Blade servers - your time may never come

2010-10-09T17:10:00.000-07:00

I know many people use and like blade servers. However, I have repeatedly evaluated them and found them not quite matching needs. Blade servers can provide high density compute capabilities. Let's look at a common product - a 10U 16 blade chassis. You can have up to 32 CPUs, 3TB RAM, and up to 32 disks - 2 per half-height blade, all in a 10U chassis.

The problem for most enterprises is that this is more compute power than they will ever need! If you are a Google/Ebay/Yahoo/Amazon/Facebook kind of company, of course that is not true. But these kinds of companies often use alternative products for density, such as multi-server chassis that strip out the management bits of blade computing for a cheaper solution.

I am running Data Center operations with about 200 virtual servers on about 15 vmware ESX servers. But when I look at the details, and the future, I could imagine running them all on one box within a year or two. Most of these ESX boxes run pre-nahalem intel CPUs, where 4 physical CPUs are easily outclassed by 2 nahalem-based CPUs. With an 8-core CPU (or a 12-core AMD cpu) perhaps we could go 4-1, roughly 2 years later. My old 4U servers with 128GB RAM are being replaced by 1U servers with 192GB RAM. If this trend continues, I'll have 1U servers with something like 512GB within a couple years, maybe even 1TB of RAM. Large enterprises need this kind of power for very few needs other than a virtual server infrastructure.

New single high-power systems are replacing use cases for the blade chassis of yesterday.

VMWare Vsphere 4 (enterprise) - is it a big deal?

2009-06-27T18:30:00.000-07:00

One feature of vsphere 4 is a game-changer - EVC - enhanced vmotion compatibility. It allows vmware to use new CPU features for virtualization management, but uses lowest-common-denominator features for running operating systems, to enable vmotion between modern Intel cpus, OR modern AMD CPUs, but not between the two. Without this feature, I felt forced to buy more old CPUs. Now I'm free to buy new 6-core CPUs, for example. Once we test the new CPUs, I'll post about it.

New technologies benefiting the little guy - power efficiency in small server rooms

2009-03-15T21:54:00.000-07:00

Small server room operators have many capabilities:

hot and cold aisles
plastic aisle separator curtains
Liebert XDP/XDV and other in-row cooling solutions

I believe that in the newly popular PUE-type evaluations, small operators can be competitive with the big colos. Moving air big distances takes energy. I buy into the concept that fluids are much more efficient for moving heat around than air. Raised floors in closed systems are no long er the efficient way to do things.

What is ITIL? What is a CMDB?

2008-03-02T07:29:00.000-08:00

I was working with some ITIL-type concepts before ITIL was well known. Pretty much all of us are, although many of us do not know it. I think of ITIL as a framework to not forget about some of the stuff that should be considered as we manage our IT and ops stuff. If you get ITIL training, part of it is about using terminology differently - we all know that people using the same word to mean different things can lead to confusion, especially in the tech. industry, so this is important. The other part is getting a list of all the stuff you should pay attention to - and it is meant to be a complete list.

CMDB is an inventory of stuff. A good inventory. For example - I have a list of all of my servers, and what they are for. Information about when they were deployed, and what their service contract is, if any, is in there. Then, instead of checking all the servers, we check the CMDB. It is just the database of record for what I choose to put there. It can be in several places - we use a separate database to track hardware problems. In the grand scheme of things, it is part of the CMDB too - the database of record of what is what.

Compute Power + Virtualization = Revolution

2008-02-27T04:19:00.000-08:00

I've jumped on the virtualization bandwagon. My team now uses macs and vmware fusion. We're virtualizing all IT systems we can.

What we're virtualizing:

OS's: windows, solaris x86, linux
Applications: Oracle for development, IIS/ASP, Filemaker Pro, Operations management tools virtual center and Netapp DFM
Virtualization technology used: VMware ESX/Infrastructure
Features to use next: capacity planning
Favorite new feature: storage vmotion - we change what storage is behind a virtual server instance on the fly, live!

Benefits:

Unified HA: for an IT shop with a bunch of stuff with no HA solution, and 15 things with 15 ways to fail over, we unify the HA capabilities of our infrastructure, using the same HA metholodology across the board.
Capacity utilization: Our IT infrastructure was very lumpy in how much servers were utilized. Now we can pack in virtual instances until utilization is high across the board. VMware DRS keeps us from having over-utilization problems.

Where do we not use it:

Where we need a direct piece of external hardware: currently backup tape libraries
Where we want a server separate from our infrastructure: some monitoring server instances
Where vendors tell us not to (production Oracle, some other software...)

Virtualization - why I expect it to become a bigger deal, oh and who doesn't care?

2008-01-02T08:55:00.001-08:00

I hear lots of folks talking about virtualization. However, there is one primary reason why I think it is about to become a bigger deal that people do not talk about much: hardware. First, the price of RAM has dropped off the charts - it is MUCH much cheaper than it used to be. The second is the small servers (1U tall) with huge power. 2 fast quad-core (with appropriate memory for caching) low-power-use CPUs with a mountain of memory - 32-64GB are available today. This is more CPU power than most companies need for any single purpose. There is talk of going back to a mainframe mentality - never before has so much compute capability been available for so little cost in so little space needing so little power. Ancient mainframes took up rooms or buildings - if these new virtual servers are like mainframes, they are the first to fit in 1U of rack space.

It is best for virtualization infrastructures to be on a central storage platform. I don't plan on talking much about that at this juncture, other than to say I am running vmware servers off of NFS quite successfully.

IT shops can get many bangs for the buck running vmware infrastructure with virtual center - take weird old servers, with only internal redundancy (if that) and gain:

-- capacity planning for virtualizing new servers

-- automated help virtualizing those new servers with minimal downtime

-- as much storage redundancy as the shop can afford and manage in central storage

-- options to use central storage backup capabilities to integrate with the IT shop backup strategy - have vmware cause a central storage snapshot with vm disk use quiesced, with no interruption in service - thus no longer needing a backup client license on each (virtual) server.

-- HA - clustering and failover - and this for services that did not have it before - makes an IT manager's service uptime much better

-- Easy server upgrades - just fail server instances off the physical server you want to replace, and replace!

So who does not care about virtualization? The big web ops shops don't care. They need CPU, RAM, and networking. These new cheaper more powerful servers using less power are still awesome, but not as part of a virtualization platform. They might have tens or hundreds of servers fulfilling one purpose, not part of an old server. I'm talking about the likes of production web operations of companies like Yahoo!, Google, Amazon, Ebay, and the like. They bind many servers together (for example, behind a load balancer) to act as one.

However, big web operations like these, more and more, are creating virtualization-like capabilities in the opposite direction - making large banks of servers be able to flip-flop repurpose as needed for other purposes. This is the same problem, kind of. If virtual servers can be very efficient, but multiple servers could bond together to act as one server in some ways, then

Initial Post

2008-01-02T08:47:00.000-08:00

I should start by defining data center:

Facilities management, network and systems (and applications) administration, database administration. I include security type administration as well. This includes all aspects of IT and internet operations management. ITIL is a framework for... I'll say not forgetting about some of the important aspects of this area.