http://americanbuildersquarterly.com/2015/rosendin-electric/

In exchange, she suggested a few topics as a guest blogger.  This is going up on her site first, with my post trailing by about a week.  Blogs typically discuss the nuances of what UPS system to buy or what is a better widget to employ in your data center, and I will confess that the Data Center Guru blog is no different.  So, I wanted to step back and speak on what really makes a data center successful.

The two key elements to data center development are:  availability & meeting and beating target benchmarks that are dead-matched to your business and goals.

Hersey!   No doctoral thesis on game theory and the mathematics of reliability!  No discourse on differing battery technologies!  No boring diatribe on something technical!  What is the world coming to?  Sheer madness.

The fact is a data center is simply nothing more than a factory that makes information.  Virtualization?  Nothing more than industrial optimization, albeit one you can’t touch.  PUE?  Merely using less energy or skylights in the old days.  Compaction?  Process reengineering.

And the purpose of the information factory is to make information continuously at the lowest cost per transaction or activity, just like any factory-made product.

When we embarked on the ANSI/BICSI 002 Data Center Standard in the early 2000’s, one of the glaring needs that had to be met was a performance-based set of metrics for the data center.  This would be for all four aspects of the data center – the facility, network, platform and application.

In all cases, we should cease to care about the constituent pieces of the ecosystem and start worrying about the end-to-end ecosystem.  What we all worry about for data is continuity – the continuous flow of data or the protection of stored data.  While most facility solutions focus on the particular engineer’s or owner’s design, the key thing to remember is that we have redundancy built in to all the major layers (facility, network, application and platform).  Redundancy is there, for the sole reason that things break, fail, don’t load right, somebody digs up your fiber with a backhoe, etc.  In short, redundancy allows for when things must be maintained or do fail.

Business-wise, executive leadership speaks to the output.  Technical-wise, stakeholders speak to the components.  There is vast difference.  The truth is what really matters, and what most of you are compensated on is the continuous operation and not the level of system availability in the facility, network, platform or application.  Redundancy is simply the manifestation of the failure analysis and business recovery model you are choosing to operate.

Back in the early 2000’s, I was commissioned to review the data processing facilities for a major high tech company.  Like many firms at the time, they were data center space-rich but infrastructure-poor.  Upon review, they had a series of solid Tier II and one Tier III facility.  The goal was to study the facility piece of the enterprise, to determine the work require to bring six live data centers to a Tier III stance.  Yep, all six, retrofitted live, to the tune in excess of $110M.

So, we asked the basic question, “How much space do you need?”  The reply, “For this program, not as much as we have.”  Ok, now we’re getting somewhere.  We didn’t ask about availability, yet.  Our observation was that this client had good application and platform diversity, older but well-maintained MEP systems and plants and a serviceable network.  We simply asked could you achieve the same result by hardening and enhancing the network, which had an added benefit of turbocharging their disaster recovery program.  In short, the failover would be more DR-ish on the network than forcing each site to stand-alone.  After a three-month period to mull it over, the answer was yes.  That day, we coined the term Enterprise Reliability (ask Jeff Davis).  While we went home without a contract for the design, we earned a life-long client.  And we saved them $95M in the process.

The contrasting allegory today is the large-scale social media, entertainment, portal, e-commerce and search firms (did I miss anyone?) approach things differently from many, as they have a processing, storage and network scale of great scope.  When an IT program grows to a specific scale, the failure analysis allows for portions of data centers, even entire sites or regions, to fail, without compromising SLAs to their customers.  What’s occurred is that all systems are now considered collaborating and integrated to the end result of the application availability.

Once you plow through what your internal and external SLAs need to be, the benchmarking, sell and price targeting are then brought to bear.  In the industrial case, you first have to decide on the product then how you are going to fulfill it.  Here’s the rub.  Unlike 15 years ago, where the only options were build-to-suit/operate-your-own and a limited amount of managed services, today there are a host of options.  Heck, 15 years ago, SaaS was mistaken for a Swedish car or British Tier 1 commando unit.  What may occur in the benchmarking phase is that the solution for the facility, software, network or platform solution may take a sharp turn from the initial assumptions and make you reconsider how you are going to achieve your solution.  What should not change is the cost and sell baselines or the SLA, only how you got there.

You might run down to see my friends Chris Crosby or Jim Smith for a facility solution.  You might call any of the cloud service providers for an IaaS solution on the virtual side of the business.  Or VM Ware might hop in and tune up your platforms, saving power, space and hardware.  You get the picture.  There are many tools, and the trick is which one you pick up to solve your problem to your specs.

The benchmarks are levels that your business expects to execute, profit and pay for services.  You start with the business model you’ve undertaken, then you work through the templates on benchmarks to narrow the cost/unit, cost/transaction and the operating carry.  The trick is don’t get to enamored with your first approach, as it’s likely to change.  Remember Sal Tessio’s line at the conclusion of The Godfather, “It’s was never personal, it was just business.”  But also remember when he said it that he was being driven off to be executed, because he did not stay loyal to the family business.  That is the result of misalignment between goals, requirements and expectations.

We’ll get into the benchmark builds in the next blog.

Salute’ a tutti!

We have seen a monumental leap forward in battery technology in the past five years.  Some of this is a result of military technology being passed down to the industrial and commercial sectors, and some of this is related to the stored energy needs for hybrid cars and wind and solar distributed generation power systems.  While we now enjoy the luxury of batteries that can tolerate higher and lower temperatures than the traditional 77F; offer varying discharge and charging profiles; and can now offer a wide mix of weight to kW ratios.  Super!

At the end of the day, we’re still dealing with a chemically-based stored energy system.  And the traditions that we have used for years in data centers to prevent the exact types of failures that occurred on the 787 looked to have been missed or inadequately provisioned in the aircraft.  This blog is not about the indictment of the 787 design team.  I’m certainly not skilled enough to develop a design that can fly 10,000 miles at 590 MPH under the unwavering scrutiny of the FAA.  But some of the old battery rules need to be followed for the same reasons they always have – you don’t want a battery fire.

There are two basis approaches for battery management – voltage and impedance.  One of the great adages in our industry is we design for redundancy when failure is not acceptable, and the systems that are subject to failure or maintenance are offered with predictive maintenance or failure precursors that allow for any operator to intervene at an appropriate time.  One of the great benefits of batteries is that they tend not to failure quickly and catastrophically, if they are properly monitored.

Of paramount importance is the monitoring and controlling of the battery charger and well as the health of the individual cells, both for impedance and voltage.  While the failure of many battery types tend to be open-circuit resulting in the loss of the DC buss, the failure should never be so severe that it results in a life-safety hazard or a hazard to craft or building.

Sometimes, the old school is the best school.

What happens is topology and cost creep, where nuance, detail and pathway are piled atop each other in search of the system that is resistant to every conceivable circumstance.  This is admirable in intent, but typically a massive failure in cost and complexity.  What really matters is the system availability over time.  As we discussed with maintenance in the previous blog, simpler tends to be better.  The rub here is that ultimate system size has a large impact on the topology that you select.

Before we get to system architecture, let’s talk about the parts that we typically use, namely switchgear, generators and transformers.  When you look at the industry over the past ten years, a few habits begin to emerge:

•  Main switchboards are evolving to match substation transformer and generator sizing, in the 2,000 kW to 3,150 kW range.
• Arc flash hazard mitigation is forcing some changes to systems via the reduction of energy present in any given breaker cubicle.
• Data centers are getting physically smaller on a per-unit load basis when viewed by kW/compute cycle and kW/TB (and a great friend of mine reminded me that it’s going to be the  kW/PB or petabyte as the new normal where storage is king).
• Servers are sexy, but storage pays the bills.  You have to put all that data somewhere.
• Data centers IT environments are usually isolated by load group and consist of several rooms or single user facilities of up to 10,000 SF raised floor each, with a few notable exceptions in the large-scale colocation, social media, content and search communities.  Smaller rooms allow for compartmentalization of systems, and this lends itself to the smaller “footprint” noted above.
• The amount of large, one-of-a-kind facilities is plummeting.  Organizations tend to find what works and then stick with it.

Let’s be honest.  In the past ten years, the wholesale and retail data center and the search/media/social networking folks have pretty much built the lion’s share of the facilities on the planet.  This has driven a couple of fascinating phenomena.    First is the de facto industry standardization on the 2N topology in a wholesale facility, and the N+1 standard in most larger data centers, save for transaction processing.  Those who disagree might have forgotten that the first prescribed Tier IV system by The Uptime Institute was system-plus-system, 2N or 2N+1.  Second, with the amount of systems and data center space built to this wholesale 2N standard, there are now millions of hours in operating time on this 2N architecture over tens of millions of square feet of space.  And the performance data, excluding human factors, appears to favor wholesale’s business model.

The rules of thumb for the larger critical power loads (those exceeding 5 MW, the old school break between low voltage and medium voltage systems), a design might utilize multiple and paralleled systems, looped distribution and utility systems that appears more “grid like” in the topology.  In this case, those several utility, generator and UPS services collaborate to supply a given load any may offer multiple connections to loads thoughout the facility.  At that scale, this “grid” or “looped” topology is likely to be far more cost effective and operationally efficient to the more “siloed” 2 MW segregated rooms and load blocks.  The key is that both of those choices are mapped to a business’ need that exceeds your typical 1,100 kW-1,500 kW / 110 – 150 W/SF requirement in that 10,000 SF space.  The key to larger builds is that there is a known and disciplined connection of this type of system architecture with the business it supports.

Moving back to the wholesale data center model “2MW” system, they are nearly exclusively 2N or N+1.  One interesting point is how we define 2N.  For these smaller system, isn’t it simply 2N = N+1, or 1+N?  Sometimes it is, often it isn’t.  To slice through the semantics and to borrow from ANSI/BICSI DC Standards 002:

•  2N is a fully redundant architecture consisting of two equally sized systems, each capable of carrying the total system load.
• N+1 implies an N system capable of carrying the load, with a single system able to replace one of the systems comprising the “N” capacity in case of maintenance or failure.  When”N” =“1” and there are two systems, you have then have 2N.
• 1+N implies either 2N or N+1, depending on system size.  For this reason, this term is not commonly used in the industry standards, TUI Tier Rating and ANSI/BICSI 002 Class Standards.

 The ability for a system to respond to failure or maintenance modes of operation is what determines its Tier or Class rating.  Recalling ANSI/BISCI 002, where performance-based criteria was first introduced for electrical and mechanical topologies, Class III allows for a single failure or maintenance event to render the system to an “N” state.  The wholesale data center industry has evolved to this Tier or Class III model, where the critical power architecture is 2N, allowing for complete redundancy from the UPS input switchboards to the PDU output, while using N+1 in the mechanical and supporting systems.  This did not happen by accident. 

 When examining the wholesale data center business model, what was introduced to the critical infrastructure building business was a direct connection between the capital cost to build and the business.  Wholesale data center, being essentially a high-tech real estate business, holds the sensibilities of cost discipline and functionality.

 When examining how to achieve cost effectiveness, one typically chooses the cheapest part, with the caveat in the data center business that the part also needs to be reliable.  That’s why we see 2,000 or 2,500 kW generators, 3,000A or 4,000A switchboards, 750 kVA UPS modules and Trane Intellipaks everywhere. 

While solid reliability at the component level is mandatory, it’s how those parts are procured, arranged and connected that result in system availability at the lowest.

Three years later, the city is replacing all of its street lights with LED systems, primarily due to avoided lamp replacement cost and the massive energy savings they would realize.  Life cycle costs clearly won out here.  We also see some resistance in the built environment versus the more utilitarian applications such as street lighting or exterior lighting.

We find it both rewarding and amusing that the city got religion on this issue.  The case in point is that LED offers a significant energy savings over fluorescent or HID lamp systems for continuously-lit systems.  LED is challenged in areas where the lighting is activated by occupancy or where it’s not in continuous service.  Our experience has shown that you may realize up to a 40% reduction in input watts for an LED system over your standard T8 fixture, at the identical lumen output.  For incandescent fixtures, this wattage reduction could be as large as 90%.

So what about price?  We attribute the current price point for LED OEM systems to be a cost-recovery exercise by the fixture manufacturers.  LEDs are widely used in a host of industries, where the LED themselves are a commodity item, even for the high-end diodes.  We have also see LED lighting systems be gamed by distributors, manufacturing reps and the manufacturers themselves in quotes and fixture orders with the purpose of packaging to a certain manufacturer.  The key here is to get the line item costs for your fixtures. 

What are the drawbacks for LEDs?  First, standard fluorescent and incandescent dimmers don’t work well or at all with LEDs.  Second, the color rendition is a bit on the cooler side and into the blue spectrum from pure white light.  That being said, LED color indexing is being rendered closer to daylight and now offers a few more color options, approaching the offerings we see for the fluorescent lamping in today’s market.

What’s the proof in the pudding?  About six months ago, my brother-in-law and sister-in-law gave me a call.  My sister-in-law operates a thriving college counseling practice out of their home in Silicon Valley.  As a result, the public area lighting in their house is on about 12-14 hours per day.  The lamping package was exclusively PAR- and MR- type lamps, 42 in total.  They have also experienced significant lamp mortality, especially on the MRs, due to both overall hours on the lamp as well as the per-day run time, which well exceeds 10 hours per day.

After a $1,050 lamp retrofit and a single new LED-appropriate dimmer, the utility bill was cut by one-third, or about $300/month, resulting in a three-month payback on the initial $1,000 lamp investment.  The CR on the MR and PAR retrofit LED lamps are identical, so now we have identical color throughout the spaces.  The only rub was a single dimmer that had to be replaced in the dining room.

Five months later, my brother-in-law has not replaced a single MR-type LED lamp yet.  We did splurge and choose the 25,000 hour life lamps, a four-fold improvement over the published MR-16 lifespan systems they replaced.   When looking at the MR lamp mortality, we never realized a great than 4,000 hour run time on those lamps, so the lifespan increase further reinforces the life cycle payback.

And my brother-in-law is thrilled to NOT be up on a ladder twice a month to change lamps.

This all started about three months ago with David Shirmacher, the SVP of Technical Operations at Digital Realty Trust over a casual lunch (as if Dave and I have ever done anything casually).  Dave is one of my favorite guys in the business.  We’ve worked together for a long time, and he’s got one of the best minds in the field.  The subject of reliability and maintenance frequency came up during lunch at the Salt House in San Francisco.  The debate focused on both maintenance activity frequency and the complexity of that specific, multi-step activity.    That got me going on where did this start and what is the quant to back up the industry’s well-ingrained behaviors.

 While pundits sit and preach about system reliability, they fail to recognize one salient point:  It’s not about reliability, it’s about availability. 

 No one cares what happens behind the curtain in Oz if your data keeps right on flowing.

 An old saying in the data center business is that you don’t design systems for how they work – you design them for how they fail and how they are to be maintained.  While individual system focus has driven the reliability of components to very high Sigma levels, it’s how applications, platforms, systems and facilities are brought together, interact and are changed that drives your application availability.  So, what are the arguments and where did this all begin?

 Reliability calculations were born in 1966 during the Space Race in the form of MILSPEC Handbook 472, at the behest of the aerospace programs of military.  On page 2, 472 defines the tenets we use today – Mean Time Between Failures (MTBF) and the Mean Time to Repair (MTTR).  Go ahead and Google “MILSPEC 472”.  The handbook is available as a free PDF download on several websites.  Another excellent resource for the most modern version of 472 is the BICSI International Supplemental Information 001, Methodology for Selecting Data Center Design Class Utilizing Performance Criteria.  It’s available on the BICSI website at https://www.bicsi.org/dcsup/Default.aspx  (it’s free, but you have to fill out a request form).  Here, we’ve moved to performance-based metrics backed by the mathematics in this article.

 MILSPEC 472 was the first effort to develop the basics of system and component uptime.  It remains a  seminal work.  While the scorecards, matrixes and the network analysis methods are still quite relevant today, 472 does have its drawbacks.  It focuses on the examination of one system, not a series of physically separated but logically- and operationally-joined systems.  Nor does 472 consider concurrent maintenance and operations risk, where in the data center world, routine human intervention is both required and offers a varying risk proposition.

 Our availability assumptions have been reinforced by the historical performance of a variety of system architectures and features.  This manifests itself in the anecdotal Sigma levels you see published for a variety of electrical and mechanical system architectures.   However, these Sigma levels only consider the MTBF and MTTR for the system, with no outside influences, such as manually-initiated maintenance operations.  While 472 was developed for electronics, aircraft and spacecraft, a majority of system failures for those assemblies, short of a catastrophic failure from a series of events or force majeure (like a missile to the fuselage), simply means the system is offline, to be replaced by a squadron spare.  In the data center business, that squadron spare is that instantly-available redundant system that stands ready to assume a processing, storage, network, power or cooling load should a platform, component or connection fail.

 When you actually look at the probability’s mathematics on Page 1-16, it’s a compounding multiplier of:

 P(A1) x P(A2) x P(A3)…

 While this drives the steady-state to high very Sigma levels and a near zero theoretical outage value, it says nothing about manually-induced incidents, resulting either from maintenance or modifications, or in the case of software, new code loads.  And it’s in that frequency of change, coupled with the competency of the effort that has the most profound effect on your system availability. 

 Most facility guys happily follow the manufacturers’ edicts for maintenance frequency.  Most facilities will run their generators monthly.  But if you experience a real power outage, does that count for the gen test?  If you test weekly, would it better to test monthly or visa versa?  If you test monthly, would it better to test quarterly, assuming your batteries, fuel and other support systems appear to be in excellent health?  The fact is that every time you have to maintain a piece of equipment, you change the facility’s steady state operation. 

Let’s swing into the math.  You may possess exceptional topographic redundancy in your system and find comfort in the Law of Large Numbers (LLN) when looking to minimize incidents in your facility.  LLN does tangentially recognize the fact is that one activity in any large number pool will simply go haywire, the dreaded Black Swan event.  Or as I call it, Bad S*** Happens to Everyone at Least Once.

LLN clearly speaks to normalized returns over a large data sample pool and a long period of time.  While the Black Swan event is not likely to occur, best practices speak to changing our behavior and design choices to reduce your exposure to the “Swan” as much as practical.  Don’t get too comfortable.  Let’s now throw in the Gambler’s Fallacy.  The Gambler’s Fallacy, or the fallacy of the maturity of chances, is rooted in the false belief that deviations from an expected behavior or outcome observed in a repeated independent trials or events (of some random process like a dice roll), future deviations in the opposite direction are more likely. 

 Take a moderately large set of repeating events, like a 16-sided dice rolled 16 times for a single win or the 16 quarterly maintenance cycles on your 4 UPS modules in a year.  One would expect that if you had not had a failure during that set number of transactions, the probability of the failure would decrease as you proceed through the finite count.  That’s not true.  Each event is unique, and the outcome of the succeeding event is not known until it’s over. 

 That 16-sided die is the requisite, but separate maintenance or switching activities you undertake in your data center every year.  The chance of success in this series would be:

 1 – {number of successful events/number of total events} ^{exp of the event in the sequence}

 The math of the dice versus the UPS modules are very different for one reason – the number of transactions within the activity.  The chance of a successful event on the first attempt in a 16-series dice roll would be 1 – {15/16}¹⁶ = 64.39%.  On the succeeding event, the chance of a successful event is:

1 – {15/16}¹⁵ = 62.02%, as so forth.  But when you look at a single UPS module undergoing quarterly maintenance, you have to consider the individual steps (transactions) within the activity as noted in the MOP.  It’s not a single event.  Assuming 10 major switching or maintenance steps per module per activity, or 40 activities per year, this would result in: 1 – {39/40}⁴⁰ = 36.31% for the first attempt, and 1 – {39/40}³⁹ = 37.31% for the second activity.  When you reach the last activity, the percentage of success has dropped to essentially zero.  Herein lays the math verifying that less maintenance is better.

We all acknowledge that zero maintenance is not acceptable, but neither is over-maintenance.  The balance must be struck between component count, maintenance frequency and system complexity to yield the lowest risk to uptime.  Predictive maintenance certainly addresses activity frequency, where services is applied only when the systems requires it.  Less frequent but necessary maintenance or having less components to maintain might actually be your best approach.  Regardless of when the maintenance activity is undertaken, each activity poses a risk.  And the way the mathematics work, the total count of these maintenance activities has the most significant impact to system availability.  The fact is past success or failure does not promise future success or failure, just that each is possible in independent events.  That one event might be the one that drops only the “+1” of the service, or it may be the one that drops the load or facility.  The Black Swan has come to roost. 

 Some low-risk activities, like checking oil on the gens or non-invasive IR scanning, pose little risk to the concurrent maintenance and operation of the system.  When you look at taking a UPS module offline, it takes dozens of steps in an MOP to successfully complete the activity.  Each of those tasks in a given activity carries different weight on the uptime calculation.  Performing a close-transition key interlock circuit breaker transfer poses a far greater risk than operating a “UPS Module to Static Bypass” button (as long as you choose the correct module).   The key point here is to reduce both the total number of activities or changes of state on any given system and the more risky activities if you wish to radically reduce your uptime risk.

 Let’s look at a Tier/Class III 2N UPS data center with a single generator and no swing generator.  Assume:

• One dedicated generator plus one redundant (N+1) generator.
• Two UPS modules per “A” and “B” side, for a total of four modules, distributed parallel.
• Quarterly UPS module invasive maintenance.
• Annual invasive UPS module maintenance.
• One UPS module automatically going to static bypass annually.
• Monthly non-invasive generator maintenance.
• Quarterly non-invasive generator maintenance.
• Annual invasive generator maintenance.
• One Utility outage per two years.
• Two circuit breaker switching operations per year.
• Annual IR scan on the main boards, 48 breakers total.

 This offers a total event profile would consist of 66 minor maintenance activities and 197 major maintenance or system state changes of state per year.  Over a ten year period, that equals over 2,600 activities or system changes.  If you reduce the system content to one module per side (from 4 to 2 on the  total module count) and change the UPS modules and generators to one major maintenance cycle per year, you drop the counts to 35 minor activities, 6 major activities for a total of 41 activities per year.

 If you simply make a minor topology change and modify your generator maintenance habits, you can reduce your exposure to human-induced incidents by 64.6%.  By having quantity two-less, UPS modules in a center would decrease critical load switching operations by ONE-HUNDRED AND TWELVE (112)  activities per year, or 1,120  switching operations in a 10 year period.  This is seriously good stuff!

 So, here’s what we see:

1.  Fewer components equal less maintenance activities – thereby reducing risk of downtime from human error and intervention.  Non-invasive scanning methods, like ExerTherm, will allow you to monitor your system without opening cabinets or exposing your staff to arc flash hazards.
2. To avoid killing a “side” when maintaining the UPS bypass, utilize cross tie between the systems at the UPS input or output in a 2N system.
3. There’s a reason that most of the more sophisticated data center wholesale operators use Trane Intellipaks.  While each company puts their own “pimp” on the units, they are simple, robust and, as we say about a 1970s GM car, anyone can fix them.
4. The more dynamic the system is, the more likely it is to drop a load.  Follow the suggestion of the ANSI/BISCI Data Center Best Practices Manual 002 – use only one step transfers or switching operations to arrive at the next steady state in any maintenance or failure mode of operation.  Do not have the automatic system response depend on something else happening.  In other words, avoid complexity.   For data center reliability, the KISS principle certainly applies to data center design.
5. The likelihood of new code failing is directly proportional to the degree of burn in and test/debug it underwent.  State machines are very important.  Unfortunately, unless coders are classically trained in how to develop a complete state machine, “putting things back to normal state” in any given failure scenario is not a common practice.  Most software developers focus only on the most likely outcomes.  The successful ones try to create the unexpected (see Netflix’s Chaos Monkey).  Don’t forget that any of those PLCs in your switchgear or paralleling gear require software development as well.  Buyer beware if design, test and commissioning  do not follow the best practices for software development.  Outages will occur.
6. Most of the truly epic data center outages have not been due to power or cooling.  It’s been the network or software.  Knight Trading, the Security Pacific ATM outage, the Bank America Securities outage and the “fail-to-fail over” at the central offices during 9/11 were all root caused to the network architecture or software systems.

Coming up in the next couple of weeks – The Death of the Snowflake.  How Digital Realty Trust and the Wholesale Colo providers coupled with the Open Compute Initative is about to make the modern data center over in a way you could have never expected.

Stay tuned and thanks for following me.

Cloud Computing

The greatest increase in data center, space and utilization was clearly in the cloud space last year.  When the 2011 prediction was made, the inference is that cloud, as a service or as a cornerstone to the technology plan for a business.  In review of where the money flowed last year, the kings of the hill were Amazon, Facebook, Google, Apple, Zynga, Linked In and other service systems.  In 2011, Siri, Apple Cloud, Facebook user growth, the Zynga and LinkedIn expansions simply offer irrefutable evidence that the big growth is in geographically independant, enterprise-level application delivery – the ALL/ANYWHERE/ANYTIME paradigm.  We also take this as a major and continuing shift to SaaS, IaaS and PaaS, depending on your needs.

Modular Data Centers

Don’t get me wrong, our modular business was very robust this last 12 months.  We have viewed the modularity of the data center speaks to the larger issue of outsourcing real estate solutions to the wholesale colocation providers.  This is related to scale, cost/SF or cost/kW and the generally efficiencies you see when you scale a business to, say 18M square feet of data center space.

Creativity from Manufacturers

Dead missed this one.  We expected breakthroughs on design or manufactured systems this year.  What we observed was the general failure of new designs when faced with a highly-efficient, capacitive IT load.  While not to count the high efficiency data center dead from an electrical standpoint, it certainly is handicapped by having to serve a heterogeneous IT equipment system.  Where Amazon, Facebook, Apple or Google (herein and now referred to in this blog as the CLOUD GIANTS) can vertically integrate their application, platform and facility solutions to come up with some very efficient models, not everyone can enjoy those savings unless they are willing to make a profound change in their hardware systems.  Everyone’s looking at this, but it may take a decade for our traditional enterprise-level users in the banking and manufacturing businesses to be able to churn out older hardware for newer systems current seen in the CLOUD GIANTS.

We also observed limited advancement in critical power and cooling systems outside of Facebook Open Compute.  While we applaud the use of new long-life battery systems, the power systems we have seen are merely derivative in their advancement.

And packaging into a modular building does NOT count!  That’s just a modification of a construction technique.

So this is what they mean when people mention the benefit from the wisdom gained from age.  It’s more like the movie “Groundhog Day”, where you are stuck in a tragic comedy.

On several occasions in the past six months, our team has run into significant power quality issues with 3rd order harmonics in a few of our data centers, both in the wholesale colo and enterprise markets.  Third order harmonics in data centers is like saying that it’s going to be hot in the desert – you know it’s true, but you hope it’s not.

We have come to discover that the commodity servers from Dell have been the culprit of some pretty ugly load-generated, third-order harmonics..

Power factor correction has been the boogeyman for servers and data center for the past three years.  Stemming from the need to run engine-generators at or below unity power factor, or a lagging power factor.  Power factor improvement was a byproduct of effciency improvements demanded by the marketplace.  This is accomplished by capacitors.  However, the amount of capacitance being injected into critical power system is starting to exhibit some darned strange things.  In solving one problem, the server manufacturers, as well as the OEM power supply vendors, are creating others.  It’s like the Will Smith movie, “I Am Legend”, where the vaccine for cancer ends up turning everyone into flesh-eating zombies.  Great.

Here’s what we are seeing.

• Rectifier walk-in setting up a harmonic feedback on a 10s interval in transformer-less UPS modules.  And this is happening on 100% transistorized IGBT modules, not old 6-pulse units.
• Harmonics in large multiples of the funamental, just like the 1990’s and early Dot Com years.  In fact, we dug 15 year old waveforms out of the files, and the new power supplies have a waveform exactly just like the crap that used to be supplied back then.
• Power factor correction in the server power supply is now “swtichable” by the customer.  If the power factor correction and power supply were engineered correctly (as so aptly pointed out by one of our Engineering Directors Clint Summers), why is it switchable in a passive critical power electrical system.

When you peel back all the hysterics back, the bottom line is that the THD and the amplification of the harmonic disturbance into large multiples of what’s expected and from the fundamental frequency are well outside of industry-stated standards.

Most data center electrical infrastructure systems can deal with relatively strong harmonics.  We have seen some issues with zig-zag neutral reference operations.  But we have also seen this pop up in PDU operations as well.  Our position here is that the Z-Z would not have an issue if the power supplies worked according to the industry tolerances in the first place.  It’s like arresting a home owner for murder when the mob dumped a dead body on the guy’s front lawn.  You are blaming the wrong party.

The best of all, your power professionals have sufficient power quality metering that allows complete diagnosis of these issues.  So, don’t let that server sales guy pull an ole’ on you.  The matador goes home and the bull ends up being steak dinners for everyone!

Make sure that you demand the power spectrum data for the server power supply, both in a waveform format as well as the summary of frequency values, similar to what we’re showing below.

And for you reading enjoyment, please see the redacted power survey from the site.  I thought I was back in 1999 when I read it.  Note the PU multiples!  Holy crap!

Sorry for the inverse order of the pages.  Enjoy!

And Merry Christmas everyone!

Enjoy.