T2P Business Contingency, Continuity & Disaster Recovery News

Creating a pandemic response for your disaster recovery plan

2009-11-09T14:07:21Z

As more and more people are affected by the H1N1 flu, companies need to include a pandemic response into their DR plans.

Cybercriminals down five British police forces in a year

2009-11-09T23:16:32Z

InfoSec News: Cybercriminals down five British police forces in a year: http://www.theregister.co.uk/2009/11/05/police_breaches/
By Chris Williams The Register 5th November 2009
In the last year five British police forces have suffered major computer failures lasting three days or more as a result of malicious internet attacks. [...]

Hackers blacked out Brazil: Report

2009-11-09T23:16:07Z

InfoSec News: Hackers blacked out Brazil: Report: Forwarded from: Simon Taplin <simon.taplin (at) gmail.com>
http://www.timeslive.co.za/scitech/article184124.ece
Nov 7, 2009 11:02 AM | By AFP
Massive power outages in Brazil in 2005 and 2007 that impacted millions were caused by cyber hackers attacking control systems, the US [...]

Website Provides Data on Cellular ReceptionThroughout U.S.

2009-11-08T05:04:18Z

"GotReception was created in response to a major issue plaguing millions of cellular consumers, unsatisfactory service. This issue is compounded...

Detailing Contingency Planning

2009-11-09T05:00:00Z

Despite the inclusion of "for Federal Information Systems" in the title, SP 800-34 Rev 1 has a great deal of value for all information assurance and business continuity specialists.

Business interruptus: Prep now to avoid H1N1 flu outages later

2009-11-06T11:00:00Z

Even a mild H1N1 outbreak could disable IT departments on short notice. Are you ready to assist employees before, during and after the swine flu hits?

Rackspace cloud suffers third outage since June

2009-11-03T16:00:00Z

Rackspace suffered another data center failure Tuesday that caused outages for its cloud computing customers, the third such incident since late June.

Considerations for High Availability Designs Used for Disaster Recovery

2009-11-03T13:39:11Z

With more focus being placed on rapid recovery times for disaster recovery (DR) operations, much of the design, strategy, and practice work done for DR in the past has shifted more toward the high availability (HA) concept. For many businesses, an “always on, 24/7/365″ concept is key, so a recovery time of 48 hours is simply too long, and a data loss of an entire week would be catastrophic and considered a definite disaster in its own right. So, availability is now king–how do we achieve it? See my article on Virtualization, Replication, Storage and High Availability for introductory concepts on replication and how storage requirements increase, and on the general ideas behind clusters and replication.

Many of you here are from a Microsoft Exchange and therefore a Windows Server environment. While much has changed in the capabilities for Windows server clustering, especially in the Exchange area, many of the core concepts are the same regardless of what the latest features and options are. For example, block-level replication across drives on a SAN solution such as EMC’s SRDF/CE option is specifically designed to assist in replication of Windows databases such as SQL and Exchange, but the block-level replication works in essentially the same manner as DRBD does on Linux.

Clustering conceptually is the same regardless of the platform or systems as well. Although that might seem to be heresy to those that are irrationally tied to one platform or the other, it’s true. It’s even more true for dealing with the considerations for multi-site clusters or geo-clusters. Round trip times and network latency limits tied to the speed of light for geographically distant systems can’t be ignored, regardless of the platform or application. Also, clustering solutions have to deal with defining fail-over and fail-back procedures, and the theory behind most of these solutions is the same. Nodes in a cluster communicate via a heartbeat, and there is often a tie-breaker or “witness” node present to assist in validating that the primary node in the cluster has failed. For multi-site or geo-clusters, this is especially important both in the design stage and in understanding the possible failure modes. If network communication is down between sites, but not to and from clients at a site, multi-site clusters may fail-over and present a “split brain” situation where each site’s believes it is the active one, that the other is down.

Does the likelihood of a network outage mean that we must change our expected recovery time to be greater than the acceptable down-time for the network listed in our network SLA? Probably? This is a key question. How long must communication between sites be down before the secondary site decides that the primary site is really down and takes over as active? Do you believe that having alternate paths for the heartbeat connection will solve this? Could that create an even greater problem? Let’s look at it:

Multi-path Communication for Multi-site Clusters
The servers will likely have a subnet spanning (cross-site VLAN) solution where their heartbeat network interfaces communicate. This network path therefore includes distinct network adapters (NICs), cabling, possibly separate switching, and may take a different path to and from the remote site. If the sites communicate via a traditional WAN link, but clients connect between sites or to each site via separate Internet facing routers or VPN concentrators, the client path to the remote site and its server(s) in the cluster may be very different. Consider already that client communication on the primary site with the active node(s) may fail, but the different network path for the hearbeat and quorum info may have the cluster in a state where it is healthy, but unreachable.

If the cluster fails over due to heartbeat communications failing, but when clients can still reach the primary site’s active servers, very strange problems can arise. Depending on how DNS is configured, and on how the cluster’s IP address is managed, clients might be directed to the secondary site based on the interruption of communications on the heartbeat network. In fact, the primary site is still active. Depending on the SAN or replication solution, one or the other of the sites will be writable with the data, while the other is just being replicated to. The load-balancing or DNS management needs to align with which cluster site is active. If the heartbeat network goes down and the cluster fails over to the secondary site, but clients are still directed to the primary site by a load balancer or DNS, that site likely won’t have access to the disk volumes since the SAN will have failed over to the secondary. If the replication solution still allows write access, the data between sites will be inconsistent. The cluster will think the secondary site is active, yet data has been written to the primary. Granted, if things are set up correctly this should not happen. But it can. Be warned.

Liked this post? Get more email management and administration related news from TheEmailAdmin.com!

Considerations for High Availability Designs Used for Disaster Recovery

Computer failure paralyses Swiss ministries

2009-11-05T18:36:30Z

InfoSec News: Computer failure paralyses Swiss ministries: http://timesofindia.indiatimes.com/world/europe/Computer-failure-paralyses-Swiss-ministries/articleshow/5162275.cms
The Times of India 26 October 2009
GENEVA: Complicated computer problems at key Swiss government ministries have paralysed work at several offices since Friday, a government [...]

Disaster recovery plan basics: Updating and reviewing DR plans

2009-11-02T13:45:26Z

Each organization must decide when it is right to review and update their disaster recovery (DR) plan, but how do you know when it's right for your company?

What's the difference between disaster recovery and business continuity planning?

2009-11-03T16:02:12Z

Kevin Beaver describes the difference between disaster recovery and business continuity planning in this ask the expert response.

Ex-CEO charged with attack on former company

2009-10-30T18:04:00Z

YouSendIt battered by DoS attacks

Khalid Shaikh, former CEO of YouSendIt, has been indicted by a grand jury on four counts of mail fraud after been accused of launching four denial-of-service (DoS) attacks against the company's servers, the US Department of Justice said.

Swine flu national emergency should spur businesses to action

2009-10-26T18:48:00Z

President Obama's declaration of a national swine flu emergency should send up a red flag to businesses that are still unprepared for a pandemic.

Flu Myths and Facts on flu.gov

2009-10-22T05:59:02Z

"The new Flu Myths and Facts section on flu.gov debunks some of the myths about the H1N1 virus and vaccine,...

Five Lessons from a Data Center's Crisis of Capacity

2009-10-15T16:00:00Z

When the data center neared its failure point, Pacific Northwest National Laboratory embarked on a project to revamp facilities without breaking the budget. Consider these practical lessons from the edge of failure.

Sidekick's lesson: Back up your data

2009-10-15T20:32:00Z

After the Sidekick near-disaster, mobile phone users might be wondering if they should ever trust any cloud services to protect their vital data.

DIY: Defending Against A DDoS Attack

2009-10-14T21:41:00Z

Proactive self-defense can make DDoS attacks less painful and damaging

Virtualization, Replication, Storage and High Availability

2009-10-13T15:43:22Z

One of the great benefits for us in IT is that as hardware and storage prices have come down, and performance has increased, we are more able to offer services that in the past was prohibitively expensive to deliver. Rapid deployment and expansion of service, redundancy, and very high availability are all possible now for a fraction of the cost of a few years ago. Granted, it still costs more to provide such high quality service. Let’s take a look at how virtualization, replication and high availability, impact storage requirements and costs.

Virtualization allows us to deploy servers without tying resources to a single specific hardware system. The images can be moved from one system to another, cloned, made redundant and thereby easily allow expansion of particular applications and services. Virtual servers are a foundation for simple, rapid, consistent scalability. Having several or many identical instances allows us to deliver high availability far more easily. Virtual images do take space, and must run on a base platform, so clearly a single VM takes more space and resources that the same service running on dedicated hardware.

High Availability (HA) is the IT goal of having continuously available service for a particular application, connection or resource. Sometimes this is done via fail-over from a primary to a secondary connection or resource. It is also possible via load balancing. The load balancing can be accomplished at the application layer, at a gateway layer, or via an appliance. Load balancing is also possible at the name lookup level. For the purposes of this discussion we are considering application, gateway, and appliance types of load balancing and fail-over. Application layer mail gateway routing is often built-in to the system, whereby the gateway has alternate choices to try if its primary gateway is unavailable. This may be implemented in different ways depending on the vendor and the service. For SMTP there are underlying standards and requirements for gateway and routing behavior.

Replication of data that changes is key for us to have consistent service in the event of a failure of one of the data storage servers. So virtual images aren’t enough–we need to have the changing data replicated from the primary location to one or more redundant locations, ideally in real time.
Storage requirements obviously go up linearly for every replicated server. If you have a series of servers with the same OS, configuration, applications and local data replicated, you should then have for every n servers you have n times the storage requirements. For three servers, you have three times the base storage requirements. For 10 servers, 10 times the storage needed. Fortunately performance and reliability scale far better than the required storage. Another important factor is that the front end application layer or Web layer doesn’t hold all of the data presented. It should be clear that not every server hosts the directory of email addresses and user identities. And the Web interface doesn’t host the mail messages or the directory–it’s just a front end. The mail messages themselves are in a database, data store, or file store (depending on the mail server, platform and configuration you have picked) and that database can be highly available and replicated, but there isn’t a message store duplicating all messages on every server and replicating them to each one. Instead, the design is usually a central store, perhaps with one replica in a cluster. Similarly there are few directories, often replicated between sites or across long distances to improve performance for lookups by local users.

A very straightforward HA server layout might look something like this: two (or more) servers on the Web tier, two for each of the apps on the application tier, two directory servers and two message store servers for each site. So what might be possible to run on even one server or two in total, we now have eight servers defined, such that we have redundancy at every tier.

HA Cluster Architecture

We then need to consider base storage requirements for each type of server along with the number of servers we are going to have of each type, in order to determine how much virtual drive space and/or SAN space our servers will consume. As we discover, we always want to budget toward the high end of space calculations, and then put in even more in our estimate for future unexpected situations. For example, on some of the servers we may want or need to take a snapshot of the entire message store to work with, but need to create it locally; so however big in gigabytes our message store is, we’ll need at least that much more room locally to copy or restore such an image. Repair and optimization tools for data stores and databases also may need similarly large amounts of space to work creating temp files or new copies of the data. So, a 15 GB virtual drive might seem big to begin with, but if you build up an 8 GB data store on it, you’re “out of space” if you need to make a copy locally. Consider also if you need to restore from a backup and don’t want to delete the in-place store. Of course, often such work can be done on network drives, but again be warned that disk performance will be much higher locally.

Many of the clustering and distribution of services concepts are available within Microsoft Exchange and are integrated into the application suite, but it’s worth it to understand how these ideas work independent of just a single messaging platform like Exchange.

Liked this post? Get more email management and administration related news from TheEmailAdmin.com!

Virtualization, Replication, Storage and High Availability

IT disaster recovery and business continuity planning for non-catastrophic disasters

2009-10-05T13:33:47Z

It's equally important for companies to have a DR plan in place for both catastrophic and non-catastrophic events.

Are virtual servers a viable disaster recovery strategy?

2009-10-05T23:23:47Z

Want to use virtualization as a disaster recovery tool? Read on to learn if this application will work for you.