Infrastructure 2.0

Huge Blackberry Outage Caused by a Faulty Back-Up Switch

Wed, 12 Oct 2011 13:42:41 PDT

For the past several days, Research in Motion (RIM) has experienced major issues across multiple continents. The response from RIM caught my attention:

“The messaging and browsing delays that some of you are still experiencing were caused by a core switch failure within RIM’s infrastructure. Although the system is designed to failover to a back-up switch, the failover did not function as previously tested.”

What caught my eye was how the planned back-up didn’t work as expected. For the past decade, enterprises have been focusing on “immediate disaster recovery” for critical services. Many of these initiatives were implemented based on major events such as 9/11, Hurricane Katrina, and the tsunamis, but for most organizations, it’s the day-to-day problems that are most common and dangerous—and RIM’s outage is a perfect example. It wasn’t a major natural disaster; it was a core switch failure that could happen any day to any organization.

The lack of network automation is one of the key drivers why avoidable problems like the Blackberry outage continue. For the past several months, I’ve spoken with hundreds of network infrastructure experts about problems like these. A common response is “Our management team expects an instantaneous disaster recovery plan, but in reality, I am scared to death of what happens if there is a major problem impacting our network infrastructure.”

The “wait until it breaks” approach is a self-fulfilling prophecy that will continue to grow because as long as you follow the same processes, you’ll likely get the same result. Actually, strike that, you’ll likely get worse results as things get more complex. For organizations like RIM, there are several steps where network infrastructure automation technology could have helped eliminate risk:

Define configuration standards, best practices and document them
Proactively and regularly, in an automated fashion, compare your desired configurations to current settings
Immediately identify discrepancies or violations and remediate
Track and document changes to eliminate unplanned changes or config drift

While these steps sound very straightforward, most organizations do not deal with these on a daily basis because it’s virtually impossible to achieve without the proper automated tools. Network automation tools, such as those from Infoblox, can help automate the repeatable tasks across complex networks and find potential issues faster.

For the RIM outage, maybe a non-standard configuration or unplanned changed caused the back-up failure. If RIM fixes today’s problem, but doesn’t start proactively approaching and testing network configuration changes, they will risk another outage and the question is not if it will happen again, it’s when. If they automate the configuration process and look for lurking problems 24x7, RIM may have prevented the configuration issue or change that caused the issue well before the core switch went down.

I recently did a webinar on the power of automation for network compliance and standardization that highlights these best processes. Be sure to check out the recorded version to see how automation can help you avoid potential problems like this.

The Infrastructure 2.0-Security Connection

Mon, 22 Aug 2011 03:44:23 PDT

If you take one thing away from the ability to programmatically control infrastructure components take this: it’s imperative to maintaining a positive security posture

You’ve heard it before, I’m sure. The biggest threat to organizational security is your own employees. Most of the time we associate that with end-users who may with purposeful intent to do harm carry corporate information offsite but just as frequently we cite employees who intended no harm – they simply wanted to work from home and then Murphy’s Law took over, resulting in the inadvertent loss of that sensitive (and often highly regulated) data. “The 2009 CSI Computer Crime survey, probably one of the most respected reports covering insider threats, says insiders are responsible for 43 percent of malicious attacks.” (The true extent of insider security threats, May 2010)

And yet one of the few respected reports concerning the “insider threat” indicates that the danger comes not just from end-users but from administrators/operators as well. Consider a very recent case carried out by a disgruntled (former) administrator and its impact on both operations and the costs to the organization, which anecdotally backup the claim “insider breaches are more costly than outsider breaches” (Interesting Insider Threat Statistics, October 2010) made by 67% of respondents to a survey on security incidents.

The Feb. 3 attack effectively froze Shionogi's operations for a number of days, leaving company employees unable to ship product, to cut checks, or even to communicate via e-mail," the U.S. Department of Justice said in court filings. Total cost to Shionogi: $800,000.

Cornish had resigned from the company in July 2010 after getting into a dispute with management, but he had been kept on as a consultant for two more months.

Then, in September 2010, the drug-maker laid off Cornish and other employees, but it did a bad job of revoking passwords to the network. (Fired techie created virtual chaos at pharma company, August 2011)

Let us pause for a moment and reflect upon that statement: it did a bad job of revoking passwords to the network.

Yeah. The network. See, a lot of folks picked up on the piece of this story that was directly related to virtualization because Mr. Malicious leveraged a virtualization management solution to more efficiently delete, one by one, critical operational systems. But what’s really important here is the abstraction of the root cause – failure to revoke access to the network – because it gets to the heart of a much deeper rooted and insidious security threat: the disconnected way in which we manage access to data center infrastructure.

INFRASTRUCTURE IDENTITY MANAGEMENT

Many years ago I spent an entire summer automating identity management from a security perspective using a variety of tools available at the time. These systems enabled IT to automate the process of both provisioning and revocation of access to just about any system in the data center – with the exception of the network. Now that wasn’t a failing on the part of the systems as much as it was the lack of the means to do so. Infrastructure 2.0 and its implied programmatic interfaces were just starting to pop up here and there throughout the industry so there were very few options for including infrastructure component access in the automated processes. For the most part these comprehensive identity management systems focused on end-user account management so that wasn’t as problematic as it might be today. But let’s consider not only where IT is headed but where we are today with virtualization and cloud computing and how access to resources are provisioned today and how they might be provisioned tomorrow.

Are you getting the sense that we might need something akin to identity management systems to automate the processes to provision and revoke access to infrastructure components? I thought you might.

The sheer volume of “services” that might be self-service provisioned and thus require management as well as eventual revocation are overwhelming^*.Couple that with the increasing concentration of “power” in several strategic points of control throughout the network from which an organization’s operational posture may be compromised with relative ease and it becomes fairly clear that this is not a job for an individual but for a systematic process that is consistent and adaptable.

What needs to happen when an employee leaves the organization – regardless of the circumstances – is their access footprint needs to be wiped away. For IT this can be highly problematic because it’s often the case that “shared” passwords are used to manage network components and thus all passwords must be changed at the same time. It’s also important to seek and destroy those accounts that were created “just in case” as backdoors that were not specifically authorized. These “orphan” accounts, as they are often referred to in the broader identity management paradigm, must be eradicated to ensure illegitimate access is not available to rogue or disgruntled operators and administrators.

(And let’s not forget cloud computing and the challenges that introduces. Incorporating management of remote resources will become critical as organizations deploy more important applications and services in “the cloud.” )

None of these processes – revocation, mass password changes, and orphan account discovery – are particularly sought after tasks. They are tedious and fraught with peril, for the potential to miss one account can be disastrous to systems. A systematic, programmatic, automated process is the best option; one that is integrated and thus able to not only manage credentials across the infrastructure but recognize those credentials that were not authorized to be created. The bonus in implementing such a system is that it, in turn, can aid in the evolution of the data center toward a more dynamic, self-service oriented set of systems.

THE INFRASTRUCTURE 2.0 CONNECTION

Thus we arrive at the means of integration with these identity management systems: infrastructure 2.0. APIs, service-enabled SDKs, service-oriented infrastructure. Whatever you prefer to call these components it is the ability to integrate and programmatically control infrastructure components from a more holistic identity management system that enables the automation of processes designed to provision, manage, and ultimately revoke access to critical infrastructure components. Without the ability to integrate these systems, it becomes necessary to rely on more traditional, old-skool methods of management involving secure shell access and remote scripts that may or may not themselves be a source of potential compromise.

The ability to manage identity and access rights to infrastructure components is critical to maintaining a positive security – and operational – posture. It’s not that we don’t have the means by which we can accomplish what is certainly a task of significant proportions given the currently entrenched almost laissez-faire methodology in data centers today toward access management, it’s that we haven’t stepped back and taken a clear picture of the ramifications of not undertaking such a gargantuan task. The existence of programmatic APIs means it is possible to incorporate into a larger automation the provisioning and revocation of credentials across the data center. What’s not perhaps so simple is implementation, which may require infrastructure developers or very development-oriented operators capable of programmatically integrating existing APIs or architecting new, organizational process-specific services that can be incorporated into the data center management framework.

More difficult will be the integration of operational process automation for credential management into HR and corporate-wide systems to enable the triggering of revocation processes. For a while, at least, these may need to be manually initiated. The important piece, however, is that they are initiated in the first place. Infrastructure 2.0 makes it possible to architect and implement the systems necessary to automate infrastructure credential management, but it will take a concerted effort on the part of IT – and perhaps a highly collaborative one at that – to fully integrate those systems into the broader context of IT and, ultimately, the “business.”

^* This is one of the reasons I advocate a stateless infrastructure, but given the absence of mechanisms through which such an architecture could be implemented, well, it’s not productive to wish for rainbows and unicorns when what you have is clouds and goats.

IPv6 is Coming…The Question is, "When, and Are You Ready?"

Wed, 20 Jul 2011 15:39:52 PDT

After spending four days at the Cisco Live conference in Las Vegas last week, the questions and interest in IPv6 were prevalent. Over the 20+ hours of booth duty I did, I was amazed at the number of questions and inquiries on IPv6 but was also intrigued by how almost every conversation started and ended the same way. I would estimate that at least 70% of people stopping by the Infoblox booth asked about IPv6 and the discussion went along these lines...

Enterprise IT staff: “Can you tell me how Infoblox helps with IPv6?”

Me: “Sure, but first can you tell me your plans and where you are in your implementation strategy?”

IT staff: “Um, we haven’t done much at all but heard it’s going to be a nightmare.”

Me: “I hear that a lot, do you have a general idea of your organization’s thinking?”

IT staff: “Well, we know it’s coming but hope it’s later rather than sooner.”

After the event, I was sitting around with some old colleagues talking about IPv6 (yes, I was in Vegas and was talking IPv6 at dinner—I do need to get a life). We were discussing the hype cycle and it reminded me a lot of VoIP when it first got attention from end-users. While the hype seemed about equal, there seems to be one big difference—the implementation process. In a previous position, my company provided solutions to help manage VoIP (and other applications) and I was surprised at the number of enterprises that implemented VoIP without a great deal of thought and planning and lived with pain for many years. I remember many companies deploying VoIP with a “trial and error” approach.

In my view, it appears IPv6 is quite different where IT organizations know it is coming but are secretly hoping it will be many years away. The good news is organizations could minimize the risk of “jumping in the deep end with both feet” but if they keep burying their head in the sand and hope it doesn’t come soon, they are wasting some valuable time in prepping for the inevitable—IPv6 is coming and you will have to eventually deal with it.

Carolyn Duffy Marsan of Network World wrote a good short post on “6 Steps Towards IPv6” and I recommend it as a quick read.

Her steps included some basic but critical points including:

Inventory hardware and software to see what is actually IPv6 capable
Create an upgrade plan
Earmark funds for training in 2012 budget
Consider an appliance approach to IPv6
Start playing with IPv6 now
Participate in the next sponsored IPv6 testing event

I agree with Carolyn’s steps and think two of her items are things you could do in the short-term to help you get ahead of the curve: plan on some training and start playing with IPv6 now. You could do the training online or invest a few hundred dollars to get up to speed and for most networking folks, hands on experience will teach you a lot.

If you want to get a jump start on learning a little about IPv6, visit our IPv6 Center of Excellence, which has some great IPv6 educational information and tools.

After spending decades dealing with IPv4 addresses like 110.12.12.34, the jump to IPv6 addressing will be interesting to say the least. Infoblox put together a short “Are You Ready for IPv6” challenge which will test your readiness and highlight one of the key challenges—as well as enter you in a drawing to win an iPad 2. Try the IPv6 challenge—and no cheating the first time you take it—give it a shot and see how you do.

Intercloud: Are You Moving Applications or Architectures?

Mon, 27 Jun 2011 03:39:38 PDT

he former is easy. The latter? Not so much.

In the many, many – really, many – posts I’ve penned regarding cloud computing, and in particular the notion of Intercloud, I’ve struggled to come up with a way to simply articulate the problem inherent in current migratory and, for that matter, interoperability models. Recently I found the word I had long been groping for: architecture.

Efforts from various working groups, standards bodies and even individual vendors still remain focused on an application; a packaged up application with a sprinkling of meta-data designed to make a migration from data center A to data center B less fraught with potential disaster. But therein lies the continuing problem – it is focused on the application as a discrete entity, with very little consideration for the architecture that enables it, delivers it and supports it.

The underlying difficulty is not just that most providers simply don’t offer the services necessary to replicate the infrastructure necessary, it’s that there’s an inherent reliance on network topology built into those services in the first place. That dependency makes it a non-trivial task to move even the simplest of architectures from one location to another, and introduces even more complexity when factoring in the dynamism inherent in cloud computing environments.

Virtualization will radically change how you secure and manage your computing environment," Gartner analyst Neil MacDonald said this week at the annual Gartner Security and Risk Management Summit. "Workloads are more mobile, and more difficult to secure. It breaks the security policies tied to physical location. We need security policies independent of network topology. [emphasis added]

-- Gartner: New security demands arising for virtualization, cloud computing, InfoWorld

We could – and probably should – expand that statement to say, “We need application delivery policies independent of network topology”, where “application delivery policies” include security but also encompass access management, load balancing, acceleration and optimization profiles. We desperately need to decouple architecture and services from network topology if we’re to ever evolve to a truly dynamic and mobile data center – to realize IT as a Service.

JIT DEPLOYMENT

The best solution we have, now, is scripting. Scripting usually involves devops who use agile development technologies to create scripts that essentially reconfigure the entire environment “just in time” for actual deployment. This includes things like reconfiguring IP address dependencies. Once the environment is “up”, scripts can be utilized to insert and update the appropriate policies that ultimately define the architecture’s topology. Scripting performs some other environment-dependent settings, as well, but most important perhaps is getting those IP addresses re-linked such that traffic flows in the expected topology from one end to the other.

But in many ways this can be as frustrating as waiting for a neural network to converge as dependencies can actually inhibit an image from achieving full operational status. If you’ve ever booted a web server that relies upon an NFS or SMB mount on another machine for its file system, you know that if the server upon which the file system resides is not booted that it can cause excessive delays in boot time for the web server as well as rendering it inoperable – a.k.a. unavailable. That’s a simple problem to fix – unlike some configurations and policies that rely heavily upon having the IP address of other interconnected systems.

These are well-known problems with not-so-best-practices solutions today. This complicates the process of moving an “application” from one location to another because you aren’t moving just an application, you’re moving an architecture.

Relying upon “the cloud” to provide the same infrastructure services is a gambling proposition. Even if the same services are available you still run the (very high) risk of a less-than-stellar migratory experience due to the differences in provisioning methods. The APIs used to provision ELB are not the same as those used to provision a load balancing service in another provider’s environment, and certainly aren’t the same as what you might be using in your own data center. That makes migration a several layer process, with logically moving images just the beginning of a long process that may take weeks or months to straighten out.

SNS (SERVICE NAME SYSTEM)

We really do need to break free from the IP-address chains that bind architectures today. The design of a stateless infrastructure is one way to achieve that, but certainly there are other means by which we can make this process a smoother one. DNS effectively provides that layer of abstraction for the network. One can query a domain name at any time and even if the domain moves from IP to IP, we are still able to find it. In fact we leverage that dynamism every day to provide services like Global Application Delivery in our quest for multi-site resilience. We rely upon that dynamism as the primary means by which disaster recovery processes actually work as expected in the event of a disaster. We need something similar to DNS for services; something that’s universal and ubiquitous and allows configurations to reference services by name and not IP address.

We need a service registry, a service name system if you will, in which we can define for each environment a set of services available and the means by which they can be integrated. Rather than relying upon scripts to reconfigure components and services, a component would need only learn the location of the SNS service and from there could determine – based on service names – the location of dependent services and components without requiring additional reconfiguration or deployment of policies.

A more dynamic, service-oriented system that decouples IP from services would enable greater mobility not only across environments but within environments, enabling higher levels of resiliency in the event of inevitable failure. Purpose-built cloud services often already take this into consideration, but many of the infrastructure components – regardless of form-factor – do not. This means moving an architecture from the data center to a cloud computing environment is a nearly Herculean task today, requiring sacrifice of operationally critical services in exchange for cheaper compute and faster provisioning times.

If we’re serious about moving toward IT as a Service – whether that leverages public or private cloud computing models – then we need to get serious about how to address the interdependencies inherent in enterprise infrastructure architecture that make such a goal more difficult to reach. Services will not empower migratory cloud computing behavior unless they are unchained from the network topology. That’s true for security, and it’s true for other application delivery concerns as well.

A core principle of a service-oriented anything is de –coupling interface from implementation. We need to apply that core principle to infrastructure architecture in order to move forward – and outward.

The Consumerization of IT: The OpsStore

Wed, 18 May 2011 03:01:03 PDT

Tablets, smart phones and emerging mobile devices with instant access to applications are impacting the way in which IT provides services and developers architect applications.

When pundits talk about the consumerization of IT they’re mostly referring to the ability of IT consumers, i.e. application developers and business stakeholders, to provision and manage, on demand, certain IT resources, most usually that of applications. There’s no doubt that the task of provisioning the hardware and software resources for an application is not only tedious but time-consuming and that it can easily – using virtualization and cloud computing technologies – be enabled with a self-service interface. Consumers, however, are demanding even more and some have begun to speculate on the existence of “app stores” within IT; a catalog of application resources available to consumers through a “so easy my five-year old can do it” interface.

Unfortunately, such systems always seem to lay upon the surface. It’s putting lipstick on a pig: the pig is still there and, like the eight-hundred pound gorilla, demands attention. The infrastructure responsible for delivering and securing the applications so readily available in such “enterprise app stores” are lagging far behind in terms of the ability to also be automatically and easily provisioned, configured and managed. What we need is an Ops Store.

IT as a SERVICE

Cloud computing environments, specifically IaaS, have gone about half-way toward creating the Ops Store necessary to complete the consumerization of IT and enable IT as a Service. Consider the relative ease with which one can provision load balancing services using most cloud computing environments today. Using third-party cloud computing provisioning and management frameworks, such processes are made even simpler, with many affording the point-and-click style of deployment required to be worthy of the moniker “on-demand” and “self-service.”

But in the enterprise, such systems still lag behind the application layer. Devops continues to focus primarily on the automation of configuration; on scripts and recipes that reduce the time to deploy an application and create a repeatable deployment experience that takes much of the guess-work and checkbox task management previously required to achieve a successful deployment. But in terms of providing an “ops store”, a simple, self-service point and click “so easy my five year old can do it” interface to such processes, we are still waiting.

But these automations are still primarily focused on topology and configuration of the basics, not on the means by which configuration and policies can be easily created, tested and deployed by the people responsible: developers and business stakeholders. Developers end up duplicating many infrastructure-related services – security, performance, etc… – not because they think they know better (although that is certainly sometimes the case) but because they have no means of integrating existing infrastructure services during the development process. It’s not that they don’t want to, they often aren’t even aware they exist and even if they are, they can’t easily integrate them with the application they are developing. And because ultimately the developer is responsible to the business stakeholder for the application, the developer is not about to abrogate that responsibility in favor of some unknown, untestable infrastructure service that will be “incorporated during deployment.” Anyone who’s sat through a user acceptance meeting for an application knows that the business stakeholders expect the application to work as expected when they test it, not later in production. It’s a Catch-22, actually, as the application can’t move to production from QA until it’s accepted by the business stakeholder who won’t accept it until it meets all requirements. If one of those requirements is, say, encryption of sensitive data then it had better be encrypted at the time the stakeholders test the application for acceptance. If it’s not, the application is not ready to move to production. The developer must provide all functionality and incorporate all services necessary to meet business requirements into the application before it’s accepted.

That means operational services provided by the infrastructure must be available to developers at the time the application is being developed, particularly for those services that impact the way in which data might be handled. Identity and access management services, for example, are critical during development to ensure that the application behavior respects and adheres to access policies.

In a DevOps world, the operations team provides infrastructure as a service to product teams, such as the ability to spin up production-like environments on demand for testing and release purposes, and manage them programmatically. [emphasis added]

-- Tired of Playing Ping Pong with Dev, QA and Ops? (CIO Update, May 2011)

Developers need a way to manage infrastructures services programmatically; an “Ops Store”, if you will, that enables them to take advantage of infrastructure services.

NEEDED: INFRASTRUCTURE DEVELOPERS

While it “would be nice” if an Ops Store was a simple to navigate and use as a existing consumer-oriented application stores. But that’s not reasonable. What is, reasonable, however is to expect that a catalog of services is provided such that not only can developers provision such services but that they can subsequently configure and invoke them during development. It seems logical that such services would be provided by means of some sort of operational API, whether SOAP or REST-based. But more important than how is that they are provided; made accessible to the developers who need them to incorporate such services as required into the broadening definition of an “application.”

It is not likely to be operational-minded folks that enable such an interface. Unfortunately, devops today is still more concerned with ops than it is development and continues to focus on providing operational automation without much concern for application integration – even though that remains a vital component to enabling IT as a Service and realizing the benefits of a truly dynamic data center. This concern will likely be left to a new role, one that has yet to truly emerge in the enterprise: infrastructure developer. One that understands how developers interface and integrate with services, in general, and can subsequently provide the operational services in a form more usable to developers, closer to an “ops store” than an installation script.

While scripting and pre-execution automated configuration systems are great for deployment, they’re not necessarily well-suited for on-demand modification and application of delivery and access policies. There are situations in which an application is aware that “something” needs to be done but it can’t do it because of its topological location. The delivery infrastructure, however, can. Consider that the dynamic nature of applications is such that it is often the case only the application, at execution time, knows the content and size of a particular response. Consider, too, that it may also recognize that the user is a “premium” member and therefore is guaranteed “higher performance.” The application developer should be able to put 2 and 2 together and instruct the infrastructure in such a way as to leverage whatever delivery policies might enable the fulfillment of that guarantee. But today there’s a disconnect. The developer, even if aware, can’t necessarily enable that collaboration because the operational automation today focuses on deployment, not execution. Developers need the means by which they can enable applications to be more contextually aware of their environment and provide actionable data to infrastructure regarding how any given response should be treated.

If we’re going to go down the path of consumerization and take advantage of the operational efficiencies afforded by cloud and service-oriented concepts, eventually the existence of Infrastructure 2.0 enabled components has to be recognized and then leveraged in the form of services that can be invoked from within the application. That will take developers, not operations, because of the nature of that integration.

Change of a Network Configuration Rains on Amazon’s Cloud Service - Company Issues Apology for Outage

Thu, 05 May 2011 07:10:18 PDT

As most of you are probably aware, Amazon and its cloud service Amazon Elastic Compute Cloud (EC2) suffered a major outage starting on April 21, 2011 that lingered on for several days for many major companies relying on the cloud service. Recently, Amazon issued an apology and an explanation for the outage.

As I was reading the announcement and apology, it struck me again how manual mistakes with network changes are the culprit behind 2/3 of all network issues, verified by the leading analyst firms and our own surveys. Even for the biggest providers and thought leaders in the space, a “simple” mistake with a configuration change can cause a multitude of issues, and in this case, crashing an entire cloud service for many leading companies throughout the world.

The detailed technical description has several interesting components. For the networking geek in most of us, the overview of EBS system and the description of primary outage, impact and recovering cover many of the technical components.

However, I was more interested in the section “Preventing the Event.” One thing I’ve learned from companies across the world is it’s bad enough to be blindsided by a major outage one time, the key is making sure you don’t make the same mistakes again. Amazon discussed the need for surviving failures at great length with different technologies and processes. One of the shortest sentences in the whole article highlights the risk many organizations face—whether it is in the “cloud” or with traditional networking. “We will audit our change process and increase automation to prevent this mistake from happening in the future.”

Audit and Automation.

As I speak to thousands of enterprises each year ranging from Fortune 500 to small and mid-sized companies, one common theme repeats itself over and over again—manual processes are the biggest cause of configuration and change problems. I call this the “Oops Factor.” Organizations spend huge sums of resources on back-up, disaster recovery and failover solutions but still rely on manual processes and documentation for critical network services. They hope the “over building” of the network and services will cover up any mistakes on the change and configuration front, but as Amazon just showed us, the best laid plans don’t guarantee success.

For the past several years, I’ve been in the Network Configuration and Change Management (NCCM) space and it’s interesting to talk to companies big and small about how they deal with change, configuration and compliance today. When I talk about how automation and intelligence can help them reduce the risk of a similar problem crippling their network, the vast majority of them nod and agree automation and intelligence makes sense. But when push comes to shove, they often keep their legacy practices of manual processes, basic tools or custom scripts. When I ask why they continue to do it this way, I get responses like:

“It’s not that bad yet,” “The network seems to be working fine” or “It will take more work to change the process than it’s worth.” It’s always an interesting discussion because many IT and networking groups think they have network change, configuration and compliance management covered and are doing it well today.

So if that is the case, why are over 2/3 of network related issues still caused by change? If we had a good handle on NCCM, wouldn’t that number be much smaller. I venture to bet the main reason that number is actually growing instead of shrinking is because we still do things manually and don’t automate many of the key processes.

Please don’t think I’m preaching that you should turn everything in your network over to a machine and you sit back and drink your coffee. When I’m discussing automation, I’m talking about the many actions that are repetitive and can be reviewed and implemented by the networking team. You need to reduce the amount of times human fingers log into a device or a configuration because every touch increases the risk of simple errors and major mis-configurations.

Organizations invest in NCCM for three key reasons:

Improve network availability - avoid mis-configured based outages like Amazon just had and knowing what’s on the network and how it’s configured so you can identify issues or potential problems before chaos ensues
Maintain standardization and compliance - consistent and standardized networks are inherently more robust and reliable and cause less “oops factors”
Improve efficiency - as networks get more complex and roll out new technologies and applications, it’s important the staff focus on what “they should be doing”, not what “they have to be doing

Want to know how you are doing with your automation maturity compared to your peers? Tally your network automation score here.

Isn’t It Ironic the Biggest Issues with Server Virtualization is the Network?

Fri, 29 Apr 2011 13:47:47 PDT

In the IT and networking world, there aren’t a lot of constants over time but one aspect tends to repeat itself time after time—if you’re on the bleeding edge of technology, the new aspects will cause new issues and someone in the organization will bleed whether it is the network manager spending the weekend troubleshooting or the executive explaining why a new application deployment just blew up. And with the rapid adoption of server virtualization, history is repeating itself. But this time, putting multiple band-aids on the problem won’t solve the issue because today’s biggest challenge with server virtualization isn’t the servers, it’s actually the network.

A few weeks ago, Jon Oltsik of Network World asked a roomful of IT professionals at the ESG Ahead of the Curve event the following question: With respect to server virtualization, which of the following is your organization’s biggest technical challenge?

With the question focused on server virtualization, I was expecting answers to deal with things like the complexity of diverse applications on a single physical server, disaster recovery, power/cooling or any other normal pain point for servers.

However the responses all dealt with the network:

33%: Adds, moves, and changes in the network
27%: Provisioning/configuring VLANs
20%: Managing virtual switches
13%: Managing increasing network traffic
7%: Managing an increasing number of physical devices

Jon’s article covers four of his thoughts on the question and answers. I recommended taking a look at the article, it’s a great short read.

Being in the IT world for a long time, I’ve heard for decades the desire for IT teams to become one cohesive unit, but for most enterprises, there is still a pretty big division between the server/application teams and the networking teams. Over the next few years, I’m predicting server virtualization will drive a bigger wedge between the groups as the finger-pointing escalates as the server world tries to move at machine speed while the networking world continues at human speed, unless the server and networking teams start to work much closer together to meet the needs of the enterprise which includes automation.

From the companies I speak with on a daily basis, I’m seeing the successful organizations are the ones who have bridged the gap between the server and networking teams and have focused on process and automation to help the network handle the new requirements. The ones that tend to struggle and fail are the ones who have tried to use the same old process and procedure from the past decade with the newest technology deployments. Here is a great white paper that illustrates how the network and server teams can come together and handle virtualization challenges via automation.

Cricket Liu LIVE!: Be Ready for IPv6 Migration and Beyond

Tue, 26 Apr 2011 13:19:00 PDT

2011 is the year that IPv6 really matters!

Even if you do not need to deploy it immediately, you should begin planning for IPv6, including making sure your infrastructure and your ISP can support it.

Join Infoblox online or in-person on June 7 as world IPv6 expert Cricket Liu, author of several books on IP Address Management, DNS, and IPv6, presents a variety of IPv6 topics, including:

Setting up forward- and reverse-mapping with AAAA and PTR records
Running name servers over IPv6
Registering and delegating to IPv6
and much more!

Click the banner below learn more and register:

How to Earn Your Data Center Merit Badge

Mon, 18 Apr 2011 03:33:35 PDT

Two words: be prepared.

Way back when, Don was the Scoutmaster for our local Boy Scout Troop. He’d been a Scout and earned his Eagle and, as we had a son entering scouting age, it was a great opportunity for Don to give back and for me to get involved. I helped out in many ways, not the least of which was to help the boys memorize the Scout promise and be able to repeat on-demand its Motto (Be Prepared) and its Slogan (Do a good turn daily).

Back then there was no Robotics Merit Badge (it was eerily introduced while I was writing this post, not kidding) but Scouts embracing the concept of being prepared were surely able to apply that principle to other aspects of their lives, covered by merit badges or not. I was excited reading this newest merit badge, of course, as our pre-schooler is an avid lover of robots and knowing he may be able to merge the two was, well, very cool for a #geek parent.

Now, the simple motto of the Boy Scouts is one that will always serve IT well, especially when it comes to operational efficiency and effectiveness in dealing with unanticipated challenges. It was just such a motto put forward in different terms by a director in the US Federal Government working on “emergency preparedness plans.” In a nutshell, he said, “Think about what you would do the day after and do it the day before.”

That was particularly good advice that expanded well on what it means to “Be Prepared.”

Now obviously IT has to be more responsive to potential outages or other issues in the data center than the next day. But the advice still holds if we simply reduce the advice to putting into place the policies and processes you would use to address a given challenge before it becomes a challenge. Or at least be prepared to implement such policies and processes should they become necessary. The deciding factor in when to implement pre-challenge policies is likely the time required. For example, If you lose your primary ISP connection, what would you do? Provision a secondary connection to provide connectivity until the primary is returned to service, most likely. Given the period of time it takes to provision such a resource, it’s probably best to provision before you need it. Similarly, the time to consider how you’ll respond to a flash-crowd is before it happens, not after. Ask yourself how would you maintain performance and availability, and then determine how best to go about ensuring that those pieces of the solution that cannot be provisioned or implemented on-demand are in place before they are needed.

EARNING the DATA CENTER MERIT BADGE

It is certainly the case that some policies, if pre-implemented as a mitigation technique to address future challenges, might interrupt the normal operations in the data center.As a means to alleviate this possibility it is advised that such policies be implemented in such a way as to trigger only in the event of an emergency. In other words, based on context and with a full understanding of the current conditions within and without the data center.

Because nothing says success

like an empty inbox

Contextually-aware policies implemented at a strategic point of control offer the means by which IT can “be prepared” to handle an emergency situation: suddenly constrained capacity, performance degradation and even attacks against the data center network or applications delivered from therein. Such policies and the processes by which they were deployed have traditionally been a manual operations’ task: push a new configuration, provision a new server or force an update to a routing table. But contextually aware solutions provide a mechanism for encapsulating much of the process and policy required to address challenges that arise occasionally in the data center.

You need infrastructure components that are capable of adapting the enforcement of policies with little to no manual intervention such that availability, security and performance levels are maintained at all times. That’s Infrastructure 2.0 for the uninitiated. These components must be aware of all factors that might degrade the operational posture of any one of the three, incurring operational risk that is unacceptable to the business. By leveraging strategic points of control to deploy contextually-aware policies you can automatically respond to the unexpected in many cases without disruption. This leads to consistent application performance, behavior and availability and ensures that IT is meeting the challenges of the business.

Similarly, when considering deploying an application in a public cloud computing environment, part of the process needs to be the asking of serious questions regarding the management and future integration needs of that application. Today it may not be business critical, but if/when it is – what then? How would you integrate that application’s data with your internal systems? How would you integrate processes that rely upon that application with business or operational processes inside the data center? How might you extend identity and application access management systems such that cloud-hosted applications can leverage them?

Being prepared in the data center means you need the strategic platforms in place before they’re necessary and then subsequently requires that you lay out a set of tactical plans that address specific challenges that may arise along the way, noting the specific conditions that “trigger” the need for such measures in order to codify the “day after” procedures in such a way as to make them automatically provisioned when necessary. Doing so improves the responsiveness of IT, a major driver toward IT as a Service for both IT and the business.

Fulfilling the requirements for a data center merit badge is a lot easier than you might think: consider the challenges you may need to address, formulate a plan, and then implement it. Then wear your badge proudly. You’ll have earned it.

An Aristotlean Approach to Devops and Infrastructure Integration

Wed, 16 Mar 2011 02:19:10 PDT

Aristotle’s famous four questions can be applied to infrastructure integration as a means to determine whether an API or SDK is the right tool for the job.

While bouncing back and forth last week with Patrick Debois on the role of devops , vendors and infrastructure integration he left a comment on the blog post that started the discussion that included the following assertion:

On a side note: vendors should treat their API's as first class citizens. Too often (and i personally feel iControl too) API's expose a thinking model based upon the internal implementation of the product and they are not focused on using it from a business perspective. Simplicity to understand Load balancer -> create_network, ... vs. understanding all the objects. There is real work to be done there!
Object Oriented languages are great, but sometimes a scripted language goes around easier.

Which was distilled down to: APIs need to be more than a service-enabled SDK. Nothing new there, I’ve made that assertion before (and so have many, many, many other pundits, experts, and architects). What Patrick is saying, I think, is that today it is often the case that an infrastructure developer needs not only understand the concept and relationship between a load balancer, the network, and the resources it is managing, but each individual object that comprises those entities within the SDK. In order to create a “load balancer”, for example, you have to understand not only what a “load balancer” is, but the difference between a pool and a member, monitoring objects, virtual servers, and a lengthy list of options that can be used to configure each of those objects. What’s needed is an operationally-focused API in addition to a component and object-focused SDK.

One of the failings of SOA was that it too often failed to move beyond service-enablement into true architecture. It failed to adequately represent business objects and too often simply wrapped up programmatic design components with cross-platform protocols like SOAP and HTTP. It made it easier to integrate, in some ways, and in others did very little to encourage the efficiency through re-use necessary for SOA to make the impact it was predicted to make.

Okay, enough of the lamentation for SOA. The point of an API – even in the infrastructure world – should be to abstract and ultimately encapsulate the business or operational tasks that comprise a process. Which is a fairly wordy way to say “an API call should do everything necessary to achieve a single operational task.” What we often have today in the infrastructure world is still a service-enabled SDK; every function you could ever want to perform is available. But they are not aggregated or collected into discrete, reusable task-oriented API calls. The former are methods, the latter are process integration and invocation points. Where SOA encapsulated business functions, APIs for infrastructure encapsulate operational tasks.

That said, the more I thought about it the more I realized we really do need both. Basically I think what we have here is a “right tool for the job” issue. The question is which tool is right for which job?

LET’S ASK ARISTOTLE

Illustration: Toothpaste for Dinner

Aristotle (384 – 322 BC) is in part known for his teleological philosophy. He more or less invented the rules of logic and was most certainly one of the most influential polymaths of his era (and likely beyond). In other words, he was really, really smart.

One of his most famous examples is his four causes, in which four questions are asked about a “thing” as a means to identify and understand it. These causes were directly related to his biological research and contributed greatly to our understanding for many eons about the nature of life and animals.

MATERIAL CAUSE: What is it made of?
FORMAL CAUSE: What sort of thing is it?
EFFICIENT CAUSE: What brought it into being?
FINAL CAUSE: What is it for?

These may, for a moment, seem more applicable to determining the nature of a table; a question most commonly debated by students of philosophy late at night in coffee shops and not something that broods on the mind of those who are more concerned with meeting deadlines, taking out the garbage or how to make it to the kids’ basketball game if that meeting runs late. But they, are in fact, more applicable to IT and in particular the emerging devops discipline than it might first appear; especially when we start discussing the methods by which infrastructure and systems are integrated and managed by such a discipline.

There’s a place, I think, for both interface mechanisms – API or service-enabled SDK – but in order to determine which one is best in any given situation, you’ll need to get Aristotlean and ask a few questions. Not about the integration method (API, SDK) but about the integration itself, i.e. what you’re trying to do and how that fits with the integration and invocation points provided by the infrastructure.

The reason such questions are necessary is because the SDK provides a very granular set of entry points into the infrastructure. The API is then (often) layered atop the SDK, aggregating and codifying the specific methods/functions needed to implement a specific operational task, which is what an infrastructure API should encapsulate. That means it’s abstracted and generalized by the implementers to represent a set of common operational tasks. The API should be more general than the SDK. So if your specific operational process has unique needs it may be necessary to leverage the SDK instead to achieve such a process integration. The reason this is important is that the SDK often comes first because inter-vendor and even intra-vendor infrastructure integration is often accomplished using the same SDK that is offered to devops. The granularity of an SDK is necessary to accomplish specific inter-vendor integration because it is highly specific to the vendors, the products, and the integration being implemented. So the SDK is necessary to promote the integration of infrastructure components as a means to collaborate and share context across data center architectures.

Similarly, the use-case of the integration needs to be considered. Run-time (dynamic policy enforcement) is different beast than configuration-time (provisioning) methods and may require the granular control offered by an SDK. Consider that dynamic policy enforcement may involve the tweaking of a specific “application” to for one response but not another or in response to the application of a downstream policy. An application or other infrastructure solution may deem a user/client/request to be malicious, for example, and need the means by which it can instruct the upstream infrastructure to deny the request, block the user, or redirect the client. Such “one time” actions are generally implemented through specific SDK calls because they are highly customized and unique to the implementation and or solutions’ integration.

CONCLUSION: Standardized (i.e. commoditized) operational process: API. Unique operational process: SDK.

FLEXIBILITY REQUIRES OPTIONS

Because the very nature of codifying processes and integrating infrastructure implies myriad use-cases, scenarios, and requirements, there is a need for flexibility. That means options for integration and remote management of infrastructure components.

We need both SDKs and APIs to ensure that the drive for simplicity does not eliminate the opportunity and the need for granularity in creating unique integrations supporting operational and business interests.

Many infrastructure solutions today are lacking an SDK (one of the reasons cloud, specifically IaaS, makes it difficult to replicate an established data center architecture), and those with an SDK are often lacking an API. Do we need service-enabled SDKs? Yes. Do we need operational APIs? Yes. An API is absolutely necessary for enterprise devops to fully realize its goals of operational efficiency and codification of common provisioning and deployment processes. They’re necessary to create repeatable deployments and architectures that reduce errors and time to deploy. Simply implementing an API as a RESTful or scripting-friendly version of the SDK, i.e. highly granular function calls encapsulated using ubiquitous protocols, is not enough.

What’s necessary is to recognize that there is a difference between an operational API and a service-enabled SDK. The API can then be used to integrate into “recipes” or what-have-you to enable devops tools such as Puppet and Chef that can be distributed and, ultimately, improved upon or modified to fit the specific needs of a given organization. But we do need both, because without the ability to get granular we may lose the flexibility and ultimately the control over the infrastructure necessary to continue to migrate from the traditional, static data centers of yesterday toward the dynamic and agile data centers of tomorrow. Without operationally commoditized APIs it is less likely that data centers will be able to leverage Infrastructure 2.0 as one of the means to bridge the growing gap between the cost of managing infrastructure components and the static budgets and resources that ultimately constrain data center innovation.

Healthcare Beware – The First Civil Monetary Fine for HIPAA Compliance Violations Just Happened

Thu, 03 Mar 2011 14:33:01 PST

Barely making it back to the office after a week in Orlando at the HIMMS conference, I just saw the headline in Government Info Security magazine that the U.S. Government slapped a healthcare institution with the first-ever civil monetary penalty—a whopping $4.3 million fine to Cignet Health.

After years of promising fines for failing to protect healthcare data, the government finally moved from idle threats to a strong action.

In January, I had a meeting with several healthcare IT executives and when the discussion of HIPAA came up, the topic generated a lot of buzz. One of my favorite comments was “In relation to IT and the network, the HIPAA mandate is so broad—it’s almost like saying don’t do anything wrong. But when we ask them how to define wrong, a common response is—we will tell you when you’re in violation.” While in truth it isn’t quite that broad, it’s still a “hard to define” standard.

I have spoken with many healthcare organization professionals who never thought a fine would happen anytime soon. In fact, a common response is something like “We all take security and privacy very seriously and we take every possible action to protect this information.” But when asked about how do they prove HIPAA compliance, many organizations hoped their strategy and approach would eliminate any risk and I’m betting many of them thought the big fines would never come very soon.

This first fine should be a wake-up call to IT and networking professionals in healthcare. While HIPAA extends far deeper than just the network infrastructure, the proliferation of IP-devices which all connect to the network and the very distributed environment of hospitals, clinics, staff and patients makes it extremely hard to prove success for compliance mandates like HIPAA. The days of placing your head in the sand and hoping the “compliance police” don’t come knocking at your door are over.

If you thought managing your complex healthcare network was difficult already, just wait until you have to prove you haven’t jeopardized any patient records. Just think, this proof doesn’t mean that everything has just been perfect today, you need to show the success over time.

I hope none of you ever have to go through an audit like Cignet Health did, but if you do—you might want to check out this Healthcare Solution Note with some helpful compliance tools, take 300 aspirin and call me in six months when the process is over.

How to Build a Silo Faster: Not Enough Ops in Your Devops

Wed, 02 Mar 2011 03:08:52 PST

We need to remember that operations isn’t just about deploying applications, it’s about deploying applications within a much larger, interdependent ecosystem.

One of the key focuses of devops – that hardy movement that seeks to bridge the gap between development and operations – is on deployment. Repeatable deployment of applications, in particular, as a means to reduce the time and effort that goes into the deployment of applications into a production environment.

But the focus is primarily on the automation of application deployment; on repeatable configuration of application infrastructure such that it reduces time, effort, and human error. Consider a recent edition of The Crossroads, in which CM Crossroads Editor-in-Chief Bob Aiello and Sasha Gilenson, CEO & Co-founder of Evolven Software, discuss the challenges of implementing and supporting automated application deployment.

So, as you have mentioned, the challenge is that you have so many technologies and have so many moving pieces that are inter-dependant and today - each of the pieces come with a lot of configuration. To give you a specific example, you know, the WebSphere application and service, which is frequently used in the financial industry, comes with something like, 16,000 configuration parameters. You know Oracle, has 100s and 100s, , about 1200 parameters, only at the level of database server configuration. So, what happens is that there is a lot of information that you still need to collect, you need to centralize it.

-- Sasha Gilenson, CEO and Co-founder of Evolven Software

The focus is overwhelmingly on automated application deployment. That’s a good thing, don’t get me wrong, but there is more to deploying an application. Today there is still little focus beyond the traditional application infrastructure components. If you peruse some of the blogs and articles written on the subject by forerunners of the devops movement, you’ll find that most of the focus remains on automating application deployment as it relates to the application tiers within a data center architecture. There’s little movement beyond that to include other data center infrastructure that must be integrated and configured to support the successful delivery of applications to its ultimate end-users.

That missing piece of the devops puzzle is an important one, as the operational efficiencies sought by enterprises by leveraging cloud computing , virtualization and dynamic infrastructure in general is, in part, the ability to automate and integrate that infrastructure into a more holistic operational strategy that addresses all three core components of operational risk: security, availability and performance.

It is at the network and application network infrastructure layers where we see a growing divide between supply and demand. On the demand side we see increases for network and application network resources such as IP addresses, delivery and optimization services, firewall and related security services. On the supply side we see a fairly static level of resources (people and budgets) that simply cannot keep up with the increasing demand for services and services management necessary to sustain the growth of application services.

INFRASTRUCTURE AUTOMATION

One of the key benefits that can be realized in a data center evolution from today to tomorrow’s dynamic models is operational efficiency. But that efficiency can only be achieved by incorporating all the pieces of the puzzle.

That means expanding the view of devops from the application deployment-centric view of today into the broader, supporting network and application network domain. It is in understanding the inter-dependencies and collaborative relationships of the delivery process that is necessary to fully realize on the efficiency gains proposed to be the real benefit of highly-virtualized and private cloud architectural models.

This is actually more key than you might think as automating the configuration of say, WebSphere, in an isolated application-tier-only operational model may be negatively impacted in later processes when infrastructure is configured to support the deployment. Understanding the production monitoring and routing/switching polices of delivery infrastructure such as load balancers, firewalls, identity and access management and application delivery controllers is critical to ensure that the proper resources and services are configured on the web and application servers. Operations-focused professionals aren’t off the hook, either, as understanding the application from a resource consumption and performance point of view will greatly forward the ability to create and subsequently implement the proper algorithms and policies in the infrastructure necessary to scale efficiently.

Consider the number of “touch points” in the network and application network infrastructure that must be updated and/or configured to support an application deployment into a production environment:

Firewalls
Load balancers / application delivery controller
- Health monitoring
- load balancing algorithm
- Failover
- Scheduled maintenance window rotations
- Application routing / switching
- Resource obfuscation
- Network routing
- Network layer security
- Application layer security
- Proxy-based policies
- Logging
Identity and access management
- Access to applications by
  - user
  - device
  - location
  - combinations of the above
Auditing and logging on all devices
Routing tables (where applicable) on all devices
VLAN configuration / security on all applicable devices

The list could go on much further, depending on the breadth and depth of infrastructure support in any given data center. It’s not a simple process at all, and the “checklist” for a deployment on the operational side of the table is as lengthy and complex as it is on the development side. That’s especially true in a dynamic or hybrid environment, where resources requiring integration may themselves be virtualized and/or dynamic. While the number of parameters needing configuration of a database, as mentioned by Sasha above is indeed staggering, so too are the parameters and policies needing configuration in the network and application network infrastructure.

Without a holistic view of applications as just one part of the entire infrastructure, configurations may need to be unnecessarily changed during infrastructure service provisioning and infrastructure policies may not be appropriate to support the business and operational goals specific to the application being deployed.

DEVOPS or OPSDEV

Early on Alistair Croll coined the concept of managing applications in conjunction with its supporting infrastructure “web ops.” That term and concept eventually morphed into devops and been adopted by many of the operational admins who must manage application deployments.

But it is becoming focused on supporting application lifecycles through ops with very little attention being paid to the other side of the coin, which is ops using dev to support infrastructure lifecycles.

In other words, the gap that drove the concept of automation and provisioning and integration across the infrastructure, across the network and application network infrastructure, still exists. What we’re doing, perhaps unconsciously, is simply enabling us to build the same silos that existed before a whole lot faster and more efficiently.

The application is still woefully ignorant of the network, and vice-versa. And yet a highly-virtualized, scalable architecture must necessarily include what are traditionally “network-hosted” services: load balancing, application switching, and even application access management. This is because at some point in the lifecycle both the ability to perform and economy of scale of integrating web and application services with its requisite delivery infrastructure becomes an impediment to the process if accomplished manually.

By 2015, tools and automation will eliminate 25 percent of labor hours associated with IT services.
As the IT services industry matures, it will increasingly mirror other industries, such as manufacturing, in transforming from a craftsmanship to a more industrialized model. Cloud computing will hasten the use of tools and automation in IT services as the new paradigm brings with it self-service, automated provisioning and metering, etc., to deliver industrialized services with the potential to transform the industry from a high-touch custom environment to one characterized by automated delivery of IT services. Productivity levels for service providers will increase, leading to reductions in their costs of delivery.

-- Gartner Reveals Top Predictions for IT Organizations and Users for 2011 and Beyond

Provisioning and metering must include more than just the applications and its immediate infrastructure; it must reach outside its traditional demesne and take hold of the network and application network infrastructure simply to sustain the savings achieved by automating much of the application lifecycle. The interdependence that exists between applications and “the network” must not only be recognized, but explored and better understood such that additional efficiencies in delivery can be achieved by applying devops to core data center infrastructure.

Other we risk building even taller silos in the data center, and what’s worse is we’ll be building them even faster and more efficiently than before.

Cloud Chemistry 101

Wed, 02 Feb 2011 02:55:17 PST

Cloud is about achieving a steady state where dynamism is the norm but actions and reactions are in perfect balance. It’s called “dynamic equilibrium” and you’ll need to pass Cloud Chemistry 101 to get there.

When you were a kid you might have had a goldfish. It lived in a bowl of water and you fed it and if you were lucky it lived for quite a while. You certainly didn’t concern yourself with things like water quality (unless the water started turning green, of course) or pH or alkalinity or gas exchange rates. Circulation and total dissolved solids (TDS) were not in your vocabulary and understanding the nitrogen cycle was something you might one day explore in high school biology or chemistry – but it wasn’t a concept you took home and applied to your goldfish bowl.

Even twenty years ago when marine reef keeping started to become popular these concepts were not something that were generally applied let alone understood. But like technology, our understanding of how all these factors interact on a daily basis to create a thriving ecosystem have come a long way. Today, it’s better understood how the dynamism of an aquarium impacts overall water quality (and thus the survivability of its inhabitants) but more importantly we’re learning quickly how to manage that dynamism such that we can achieve a state of dynamic equilibrium; a state in which a stable environment is created despite its underlying rapid rate of change.

Sound like the data center of today? Like cloud computing ? Like application delivery in general? It should, because just as the industry of reef keeping is advancing quickly such that we are learning to architect systems that achieve dynamic equilibrium, so too are we doing the same with cloud computing and application delivery.

WATER CLOUD CHEMISTRY

The technical definition of dynamic equilibrium is quite involved, requiring an understanding of chemistry and reactions and unfortunately for some of us a whole lot of math.

A dynamic equilibrium exists when a reversible reaction ceases to change its ratio of reactants/products, but substances move between the chemicals at an equal rate, meaning there is no net change. It is a particular example of a system in a steady state. In thermodynamics a closed system is in thermodynamic equilibrium when reactions occur at such rates that the composition of the mixture does not change with time. Reactions do in fact occur, sometimes vigorously, but to such an extent that changes in composition cannot be observed. [emphasis added]

-- Wikipedia, Dynamic Equilibrium

The basic principle here, however, is really quite simple: you want to create an environment, a system, in which reactions to change – regardless of frequency – are well-balanced. It’s almost Newton’s third law of motion which implies that the mutual forces of action and reaction between two bodies are equal, opposite and collinear. Newton’s law requires that action and its reaction are simultaneous; in aquariums and data centers the reaction is not necessarily simultaneous, although it is close enough to be considered applicable.

In an aquarium, as the bioload (waste production, oxygen and nutrient consumption) increases a reaction occurs that also increases the ability of the biological filtration system to manage the additional load. In some cases, such as when the rate of oxygen depletion exceeds the ability of the system to introduce oxygen to the water, additional mechanical or chemical components may be necessary to increase the overall capacity. If that’s beginning to sound like an application and cloud computing, it should.

For example, when a request for an application is received, the action is an increase in application demand. That increase in demand may evoke a reaction from the infrastructure if capacity is not available to meet that demand. In cloud computing and highly virtualized data centers, this is assumed to be the provisioning of additional capacity such that the request can be processed. Appropriately, as demand decreases so should capacity (what goes up must come down). As a result, a dynamic equilibrium is achieved; a steady state of change that makes it appear to the user that the system is stable while the reality is that the infrastructure is in a constant state of change based on the state of the data center at any given time.

COMPOSITION of an APPLICATION

Dynamic equilibrium maintains that a system is in equilibrium when reactions occur at such rates that the composition of the mixture does not change with time. In a data center, the composition associated with the data center and subsequently cloud computing is the application and comprises:

? security posture

? availability (capacity)

? performance levels

? costs

As demand, device and location diversity fluctuate it is the goal of application delivery to maintain the composition. In order to maintain the security posture, it may be necessary to apply policies. To maintain availability it may be necessary to provision or modify the compute, network, and storage resources. Maintaining performance levels may require the use of rate shaping or acceleration or optimization services. And costs may be controlled by leveraging resources based not just on function but cost.

Cloud computing is about process; it’s about devops and the ability of infrastructure to collaborate and automate its reaction to changing application and data center conditions. The ability to react within context to changes in the ecosystem with the appropriate reaction such that the overall state of the application is sustained. Whether through technology or process, resource management, or policy or some combination thereof, the goal is to sustain a steady state for an application. To maintain security and performance while managing costs and capacity. No single piece of the equation can be ignored or dispensed with, because that would throw the system out of balance.

This is impossible to achieve manually. Do not be fooled into thinking that such an environment can be achieved without technology. Doing so requires pre-positioning and deployment which results in increasing waste and “bioload” that unbalances the environment by creating too much cost and capacity overhead. It is precisely the ability to automate the processes that adjust the composition of the application based on current conditions – context – that make it possible to achieve technological dynamic equilibrium. But those adjustments need to happen in the right place within the system. Filtering out toxins produced by some corals in their efforts to secure their “space” in an aquarium require that certain chemical and mechanical filtration be placed in the flow at the right place. Similarly, in a data center, the application of security and performance-related policies must occur at the right place and time in the data flow to ensure efficiency and effectiveness of those policies in reacting appropriately to changes in the ecosystem.

This is the underlying driver for Infrastructure 2.0, for a dynamic control plane comprising the entire network, storage, and application network infrastructure: the ability to intercept, inspect, and instruct the components in such a way as to stabilize an application even as its composition is changing.

Dynamic equilibrium is the goal of cloud computing and IT as a Service and those who spend far too many hours toying with a reef aquarium. The reef aquarist knows they’ve achieved dynamic equilibrium when the animals and life in that environment are thriving and growing without interference. When applications are delivered securely and are always available and perform up to business and end-user satisfaction and are able to scale seamlessly - without manual interference -then we’ll know we’ve achieved dynamic equilibrium in the data center. You’ll have earned an “A+” in Cloud Chemistry 101.

The dreaded "Ooops Factor" bites again—human error causes London Stock Exchange trading bug

Thu, 13 Jan 2011 14:04:14 PST

After an extensive two-month probe into a crash of its European trading platform, this week the London Stock Exchange (LSE) admitted the culprit was an unidentified human error. Initially, the prospect of a high-profile sabotage attack gained momentum when first reports cited the cryptic and alluring “suspicious circumstances” as a cause. But after the exhaustive investigation, this week a LSE representative stated “The incident was the result of human error and has been closed.” No further explanation or details of the human error were reported.

The risk of downtime has never been greater for virtually every organization, including the LSE. It’s easy to see the potential risk and losses of millions of Pounds, Euros and other currencies when real-time trading of a leading stock exchange was compromised for a two-hour period. However an article in the financial website www.thisismoney.co.uk brought a very interesting angle to my attention. While most everyday citizens probably don’t think of stock exchanges as highly competitive, they have some of the most fierce competition from enormous, worldwide organizations. For example, the LSE is trying to fight back from upstart rivals such as Bats Europe and Chi-X Europe taking a sizeable chuck of trading.

All of the sudden, the risk of outages goes from a pure focus on loss of transactions from a two-hour period to a much wider risk of losing future trades and revenue when customers and publically traded organizations consider listing or trading shares on a completely different platform. This just shows how the danger of network issues quickly can spread out of control.

This announcement coincided perfectly with a recent Infoblox study on Network Automation. As a sneak peek at one of the results, 44% of the responding enterprises surveyed stated at least three human errors occur every month within their IT organization. While each human error might not cause a catastrophic outage like the LSE experienced, organizations really can’t be sure which error might be a nuisance compared to a major risk.

Over the past decades, organizations have built more resilient, redundant networks and vendors have claimed to provide “self healing networks”, but the biggest headaches for most organizations today are still tied to manual processes and human error. While it is impossible to eliminate every potential human error from an IT perspective, there are ways to greatly reduce the number and severity of the “Ooops factor”—and the good news is they are tools and processes you can implement today. Some ways to reduce human errors include:

Reduce human touches to network devices—Every time a human interacts with a keyboard and a network device, the risk of fat fingers, bad copy and paste or transposition of characters occurs. Automation of repetitive tasks can greatly reduce the number of human errors. Too often, our IT teams are swamped and overworked and often try to work as fast as possible, and this leads to enhanced risk of inadvertent mistakes.
Keep up-to-date records of network components—One of the biggest causes of human errors is making decisions on incorrect and out-of-date information. For example, if your IP address management information is out-of-date or your basing configuration decisions without knowing the current settings and topology, the risk of human error is multiplied. There aren’t many guarantees in networking, but out-of-date information leads to more mistakes.
Maintain consistency and standardization—Virtually every organization has best practices and gold standards on how they want their network devices to look, but don’t maintain proactive monitoring and configuration drift occurs. If devices and settings are standard and consistent, you can normally predict results much better. On the flip side, if things are different (and you don’t know they are different), you run the risk of assuming everything will work fine.
Don’t just focus on individual devices—Too often, IT experts focus on individual devices and internal expertise. Things like device A looks good, device B is good and so is device C and I know how they are related in the service path, so we are good to go. From a configuration point of view, the biggest risk lies in this approach because things change all of the time and you must have a better understanding of how the devices interrelate today—but more importantly, what changes could impact the service path tomorrow and going forward as things evolve.
Track and document changes—Every good organization has a formal change management process that should be followed for every network change. However, even with the processes in place, most organizations have times where unplanned changes are made. It’s imperative to always track and document changes so you have the ability to know exactly what your current network looks like and not be surprised because a coworker made another change you didn’t know about.

In a time when the network and applications become more critical, it is a little scary how often the “ooops factor” of human error or unintended consequences comes up to bite an organization. The good news is some of the risk can be averted with the right process, visibility and tools.

Unleashing the power of your Microsoft DNS/DHCP servers with IPAM

Wed, 10 Nov 2010 14:11:53 PST

Infoblox's Cricket Liu, author of DNS and BIND, will speak to audiences across the globe on December 7th in a complementary, one-hour live session on how to increase network uptime and efficiency.

Cricket will cover a variety of topics related to IP Address Management for Microsoft environments including:

Enterprise requirements for IP address management
Sample architectures for introducing IP address management to your network
A walkthrough of Infoblox’s approach to Microsoft IP address management

You will be able to watch the presentation live over the internet, or if you are near one of eleven select cities, you may attend a sponsored event party, where, in addition to the live broadcast, a breakfast, luncheon, or evening reception will be held.

Click the image below to learn more and to register.