AstroArch Consulting, Inc.

vSphere Upgrade Saga: Move the Data Center: Step 1

Edward Haletky — Thu, 21 Mar 2013 22:00:02 +0000

We recently moved from Boston to Austin, in doing so, we had to move our data center but we still need to maintain uptime while we physically move the data center. The obvious answer was a Cloud presence to help us with the move. While this move was planned for a bit, finding an inexpensive but good provider was difficult.

We choose to use an Infrastructure as a Service (IaaS) based cloud. Into it we planned on moving 4 carefully selected workloads. Which workloads to move to the cloud was a relatively easy decision based on how our virtual environment was already split. Our customer facing systems were located together and where very well understood due to the level of monitoring we had in our virtual environment.

We moved to the chosen cloud the following systems:

External Web services
Email
Database to support external services
A Jump machine to access customer systems (a static IP was required) as well as our other cloud systems

We chose just these 4 workloads based on which of our 100s of virtual machines were required to keep the business running while the datacenter moved. The only truly required systems were those that were externally facing used for communication with customers.

Moving these workloads to the cloud could have been accomplished by direct copy of the workloads, but only if the cloud I chose had the support and if I was already running the workloads conveniently within a similar cloud structure, but that was not the case. So instead, I had to set up everything from scratch. The steps where straightforward:

Within the cloud service deploy the virtual machines from a template, my four templates were 2 minimal CentOS, 1 minimal CentOS + MySQL, and 1 LAMP Stack.
Deploy the applications from within the cloud virtual machines, this was a simple as setting up MailScanner + Postfix + Dovecot (which I wrote about previously in my Email Upgrade Saga) within a minimal CentOS, Setup WordPress within the LAMP Stack, Setup MySQL inside the minimal CentOS + MySQL, install the necessary communication tools within the last minimal CentOS.
Copy my Data Over from the production systems. This stage presented the most technical issues.
Transfer DNS from my existing DNS services to the Cloud Service, which required the Cloud to put in a reverse name look-up.
Wait 24-48 hours until all internet DNS servers were updated before performing final data migration
Shutdown old workload virtual machines

These steps seem fairly straightforward and they were. The big choices where to use the existing WordPress installation or start from scratch, I choose to use the existing WordPress stack. While deploying everything I created a script to handle the deployment in the future, just in case I have to do it again or I migrate my workloads to another cloud provider or back to the main site. There are some key considerations that I found when using a cloud service provider that could be helpful to others. Items that are hypervisor dependent as each hypervisor handles memory over-commit differently. The cloud I chose runs Citrix Xen Server while my workloads generally run on VMware vSphere.

Shutdown any unnecessary processes, this includes Cron jobs. The problem one for me was cron jobs, there are a bunch that chew through all available memory to create the Whatis directory. Disable makewhatis and their ilk.
Tune your LAMP stack to use only what is necessary. No need to run 1000s of web processes when the server will work just fine with a lower number but be ready to add more as necessary.
Use an Application Performance Management tool such as New Relic to track everything your web application does and re-tune as necessary. Such tools can also email you when the site is down due to the cloud provider availability issues (if any) or system crashes from using too much memory or other reasons.
Be a good public cloud citizen, you are sharing resources, so tune your application for those shared resources, what may work within your datacenter may not work as expected within a cloud.

The key is to plan for all aspects of a migration to the cloud, even if it is only for a short time. In this case, migrating the application code was the easy part with data migration being the more difficult part. We had to migrate the following data:

WordPress database records (twice for some aspects of the data). The second data migration was a limited subset based on knowledge of the data. The goal was to ensure we had the most up to date statistics data after the DNS switch over, which can take 48 hours in some places.
Email accounts. Email account data, such as folders, and unread email, etc. had to be migrated.

A well orchestrated plan, scripting, and knowledge of your data will improve any cloud migration. But also understand not only how the application works but how the underlying operating system works. The makewhatis cron job caused our cloud services to crash with out of memory issues. It was not the application running that caused the problem, but a relatively useless cron daemon that is available on all systems.

Would we have caught that with a scripted install? Not really, each cloud is different. Now it is part of our scripted install of the OS and applications!

RHEV upgrade saga: Creating VMs on Open vSwitch

Edward Haletky — Fri, 18 Jan 2013 11:51:08 +0000

Previously we create our network by integrating Open vSwitch into RHEL KVM. Now we need to create some virtual machines to run the workloads required.

RHEV upgrade saga: RHEL KVM and the Open vSwitch

Edward Haletky — Fri, 18 Jan 2013 11:49:37 +0000

A customer recently switched from VMware Server to KVM, but we wanted better networking, which required us to install and operate the Open vSwitch.

vSphere upgrade saga: HP Insight Management integration

Edward Haletky — Wed, 16 Jan 2013 16:26:54 +0000

I have previously written about upgrading HPSIM and installing HP specific offline bundles within my vSphere environment. This was to increase the integration between vCenter, vCloud Director, and my physical hardware. Still, everything was not integrated well enough. Here’s what I did to make this work.

vSphere upgrade saga: Adding a D2200SB w/P4000 VSA

Edward Haletky — Wed, 16 Jan 2013 16:25:20 +0000

In order to increase the size of my virtual environment, I need to increase the size of my primary storage. The choices were to increase the capacity of disks for slower disks, add another disk tray to my SAN, or to use a virtual storage appliance (VSA) within vSphere. I chose to go the route of a VSA. Here’s how I did it:

vSphere upgrade saga: Veeam upgrade

Edward Haletky — Thu, 10 Jan 2013 14:53:44 +0000

Upgrading to the latest Veeam Backup and Replication v6.5 went smoothly. However, I got errors on all but one of my backup jobs. A simple patch fixed that.

Email Upgrade Saga: Replacing Axigen mail with MailScanner

Edward Haletky — Tue, 08 Jan 2013 19:23:16 +0000

I’m tired of moving spam mail to spam folders from which the system never learns. So it’s time for me to try a new approach.

vSphere upgrade saga: Upgrading the storage on your Iomega ix2-200

Edward Haletky — Wed, 02 Jan 2013 16:57:19 +0000

While technically not part of my vCloud environment, I use an Iomega ix2-200 as my tertiary storage for ISOs used within my vCloud environment. I also use an 8TB iSCSI server that also acts as my RHEV server. I will be upgrading this later as well with a 10G network card and the latest RHEV. However, this is about my Iomega ix2-200.

1TB is just not enough storage.

vSphere upgrade saga: Upgrading vCenter Operations Manager

Edward Haletky — Wed, 26 Dec 2012 12:18:05 +0000

With both VIN and vCops broken, I had to fix both before moving on to the next phase of my migration.

vSphere upgrade saga: Fixing backup and other virtual appliances

Edward Haletky — Wed, 19 Dec 2012 16:58:19 +0000

Ever since I upgraded vCenter Server to v5.1, my backup tool has been failing, unable to connect to vCenter. This is not unexpected as my SSL certificate for vCenter changed, and with that change comes the need to fix my backup tool. How to do this is fairly straight-forward actually.