Ultimately, these tools cost money. If the department budget gets cut, new implementation of these tools is at risk.

An investment in DCIM and branch circuit monitoring would provide the ability to easily balance power and to quickly identify where we can add additional circuits. The return on this investment is complicate to quantify. There are several factors to consider:

1. Staff time required to determine best placement of a new electrical circuit is greatly reduced. We go from manual data gathering and calculations to instant data availability
2. Staff time to find an ideal location for new IT equipment is greatly reduced. Again we go from a manual process involving spreadsheets to instant data availability. Several DCIM packages include tools to automate the process based on power and space availability
3. We shift from a planned capacity model, where estimated loads are calculated from manufacturer information (usually peak demand), to an available capacity model, where we know exactly what is left on a circuit, panel, or PDU. This allows us to make greater use of the capacity we have and extend the timeframe before we have to start planning for increasing our electrical (PDU, UPS and generator) capacity. This could postpone several million dollars of capital expenditure.
4. We gain instant PUE/DCIE availability and the ability to trend this over a multiyear period. This means we will be able to have have realtime feedback on efficiency changes. No spreadsheets to calculate data that is manually gathered.

If DCIM and branch circuit monitoring allow us to postpone investing in electrical infrastructure – specifically generator and UPS – for several years, it would pay for itself in the first year.

When we conceived the facility over five years ago, DCIM was in it’s infancy and Senior IT Management at the time did not see the value proposition for the large investment amount. As a result, we have a building management system (BMS) that provides us some insight, but doesn’t treat the electrical systems as a critical monitoring and reporting point. In order to hold down our construction budget, we did not install PDU’s pre-populated with branch circuit monitoring. Now, it is an expensive proposition to add this capability after the fact. I heard that the increase cost of each PDU would’ve been around $5000 per PDU or roughly $120K. To add branch circuit monitoring for our PDU’s and a few additional electrical panels will now cost us $500K. DCIM is still in an RFP process, so I can’t speak to the costs, but there is a wide range of options and costs. We now find ourselves in the position to really need DCIM and branch circuit monitoring to properly manage our capacity.

Every company has events that cause it to reevaluate set budgets to hold income vs expense rations. Sometimes this means that your budget gets slashed with a red pen – cancelled or postpone until the next fiscal year. Given a budget with DCIM, branch circuit monitoring and a generator, I would defer the generator since it will be easier to justify after real capacity reporting is available.

REDACTED

FWIW, I’m writing this post on my personal laptop, connected to my personal wireless connection.

1. This site needed a facelift. New theme, new header graphic, maybe more if I get ambitious.
2. I need to either post here or abandon the site. I’m going to try to post more. This year I’ll be deploying a DCIM system for work and I figure that the experience will make for some decent content. I’ll also be working on several other large projects that should make for some interesting posts.
3. Posts here will be mostly technical. But, amazingly many things are technical – data center stuff & IT stuff is obvious, but parenting, home repair, cooking, and many other things are extremely technical too…

Here are a few planned items that I hope to post about in the next 30 days (it helps to have a plan!):

• DCIM (Data Center Infrastructure Management) software – evaluating and selecting
• Data Center Management – Facilities vs Information Technology?
• Leaving IT and then working with them as an internal business partner

Another thing adding complexity to our environment – we moved the ownership of the data center from IT over to Facilities and formed a new team responsible for data center health. In essence, we’ve changed roles from an IT group to a Co-Location provider. This has changed the perspective from which I look at the data center.

Let’s start with the current challenges:

1. Currently, we have little to no power monitoring in place. We have little data being collected that can be used to forecast the future. Current systems limit monitoring and data collection to IP connected CDUs (cabinet level power distribution units), UPS systems and incoming utility feeds. No systems are in place to monitor all the branch electrical circuits feeding the IT load, the cooling loads or general building electrical (lighting, office space).
2. Disparate IT systems are in place. The initial foot print was multiple rack unit servers. Over time, we’ve increased the number of blade servers in use – 6 years ago we had a foot print of roughly 56 blades in four enclosures, today we have over 200 blades in 17 enclosures. Recently, we’ve installed a VCE VBLOCK solution as a proof of concept and I’m hearing rumblings of at least six more systems spread across our two primary sites. This poses a few challenges:
   • Differing power density requirements and power connection requirements. Low density cabinets use single phase, 208 VAC, 30 amp power with NEMA L6-30 connectors, high density cabinets use 3-phase, 208 VAC, 60 amp power with IEC 60309 connectors. Storage and blade enclosures often use multiple pairs of circuits or may have specialized requirements (i.e., placement of multiple blade enclosures in a low density cabinet may trigger the installation of a 3-phase, 208 VAC, 30 amp circuit pair with NEMA L14-30 connectors in order to prevent overload).
   • Space allocation contraints. Do you dedicate space exclusively to standalone servers and space exclusively to blade enclosures? Or do you commingle standalone servers and blade enclosures?
3. Storage resources are physically segregated from computer resources within the data center. This has been done with expectations that future requirements will drive additional physical security for storage resources that will not apply to compute resources. Storage resources also require that space be allocated for future expansion so that downtime will not have to be incurred to expand storage frames.
4. Currently, no aisle containment solution is in place. Both hot aisle and cold aisle containment are possible based on our design, but a decision needs to be made on which would be more effective and which would provide better results.
5. Some top of rack equipment has a reversed airflow (exhausting hot air into the cold aisle).
6. Limited blanking panels are in place. This causes more air mix between the hot and cold aisles than is acceptable and definitely impacts our ability to fully optimize the space. Fortunately, this is relatively inexpensive and quick to fix – just order and install more PlenaFill blanking panels.
7. Time and staffing levels. While the data center has been operational for two years, IT staffing levels have prevented the implementation of blanking panels and other simple solutions.
8. Budget. Budget constraints and lack of vision and understanding have kept necessary tools out of reach.

At current time, we lack the ability to truly determine our PUE (a very controversial metric, but at least somewhat useful). A rough calculation gives me a 3.2 PUE – highly inefficient. Obviously, we have a long way to go. Getting to a PUE of 2.0 would decrease our electrical utilization by 30% and our electrical costs by over $300,000/year.

Looking at the Open Compute Project’s website, I see a lot of innovative ideas. I like the “triplet rack” concept. It is a great idea when you have clusters of cabinets that create your rows. I’d have used a “doublet rack” in places had I thought to ask our manufacturer for something like it. The concept of splitting power between “clean utility” and UPS power is not new, but the way that Facebook is implementing it is interesting. My conversation with myself today will focus on power.

First, no server major commodity server manufacturer (HP, IBM, Dell) offers a 277v power supply. This means either the manufacturer would have to create one for you, or you would have to source from a 3rd party. I see problems with this, especially if you leverage manufacturer support for your maintenance. The biggest being you would have to stock a fair number of replacement power supplies since they are a non-standard part. I don’t think that would fly in my organization, I doubt it would fly in other organizations.

Second, while they have two power paths, a 277v AC path and a 48v DC path, they terminate into a single power supply. In a scale out computing scenario this is fine as the loss of one individual node is usually trivial, but if you are hosting mission critical databases you are going to want to make sure that you don’t lose a node because of a failed power supply.

Third, it looks like Facebook has a fairly homogeneous server stack. This has not been typical in my experience, and I expect that most others in my position would feel the same way. A homogeneous server stack allows you to perform a number of optimizations that you wouldn’t be able to otherwise. The typical cabinets in my server room have a variety of 1U, 2U, 4U and blade enclosures to service the disparate needs. Since our data center group does not specify servers we have very little control over what comes in as long as there is capacity (space, power, cooling) to support the equipment. This often means that we have to support servers that are purchased as a proof of concept or because of an unbelievable discount from a vendor trying to find footprint in our data center. This means that any group of three cabinets I have in my data center my not match any other group. Our initial design criteria for our current data center was 10kW per server cabinet for the first phase, growing to 15kW with the second phase and 20kW in the third and final phase. Right now we are in the 6-8kW range for most of our cabinets, but increasing use of blade servers will move us closer to 15kW very soon. Facebook is looking at 56kW for two of their three cabinet pods, that puts each cabinet at around 9kW. For enterprise data centers with a mix of server types and a industry push towards virtualization and higher density platforms, how well does this play?

I’m sure I’ll have more to say as I continue to digest…

At home:
Last weekend was a whirlwind. Katie and I decided to paint one of the guest rooms and added the downstairs bathroom at the last minute. I am convinced that all the home improvement shows have us suckered into believing that painting a room is a quick, easy weekend task. It’s not really as quick and easy as it seems… There is the taping, the wall repair, choosing a color (actually the easiest part for us) and the painting itself.

I supposed it wouldn’t be too bad if our walls weren’t as textured as they were. They seemed to soak up the paint readily but leave a multitude of tiny white spots showing through. Multiple coats with rollers and brushes were required. Even then, we have to go back sometime soon and fix the small areas that we somehow missed.

Yeah, that quick and easy weekend project ended up extending into Monday and Tuesday night! The results are stunning though. We chose a great slightly green grey color that fits both rooms well. I installed a new lamp, towel holder and toilet paper holder in the downstairs bath. New blinds and a curtain were installed in the guest room.

Wednesday was a long night as Katie and I finished getting ready for our guests to arrive. Floors had to be swept and mopped, the downstairs bathroom had to be put back together – none of the pictures or furniture had been put back.

Thursday we just hung out with Katie’s friend Vanessa and Daniel hung out with her daughter Judy.

Friday, finally it’s Friday! Home… watching Daniel and Judy so Katie and Vanessa can had some adult time together. Nothing like to old friends out on the town together!

Work, work is another story… lots of tasks, lots of projects, lots of things going on. I’m sure that it will smooth out soon, but it ended up just being an extremely long week at work. Sometimes it feels like the day will never end! Add on top a major activity scheduled at the office on Sunday… at least I have tonight and tomorrow!

Have a great weekend everybody!

Should I tear this down or give it more of my time?

I’ve decided to resurrect the blog. We’ll see how it goes…