Why the war started?

Traditionally, the responsibility to update web farms was of the IT Operations team. They are the owners of the server infrastructure and they know best how to handle all sorts of problems that may pop up (partial update of a web server, taking servers off and on a load-balancer, proper backup, etc.). In the good old days this was part of the standard IT Change Management process – something that you plan for 2 weeks in advance. Now days changes are more frequent and timely and don’t fit the Change Management process anymore. Furthermore, IT organizations that decided to allocate permanent sys-admin resources to handle the growing number of web site update requests found out that the surge is permanent and more and more dedicated resources are needed.

The option of opening the systems so that end users (even power end-users) will deploy on their own is never seriously considered because chances that anarchy will prevail in no time. End users are not disciplined to handle all the technical aspects of web content deployment, and the basic tools they use (such as FTP, Rsync, RoboCopy) don’t provide any proper support either.

The flip side of the coin are the frustrated Business people who feel the IT are unresponsive/unfriendly/careless and as usual “don’t understand who are they working for“… And its difficult not to understand them; they have important and timely tasks to complete and IT happens to be the gate keepers (and unfortunately the weakest link) at the very end of them. The boss usually does not take “it was not MY fault” so well…

War broke off! How can you put out the fire?

Modern web content deployment tools provide one excellent solution to the conundrum – they allow IT Operations to eat the cake (i.e. empower end users to deploy on their own) but keep it too (i.e. no room for anarchy whatsoever). How it is done? Divide and concur!

The concept is that Administrator is the one to define the Replication Jobs and decide on all aspects of replication (what, how, from where, to where, etc.), than Execution Rights are delegated to specific Users and Groups in the system. The End Users will log into the web content deployment system via a web portal and will be able to see only those jobs that have been specifically assigned to them. The only operation left for the End User is to launch the job as needed. The third leg of the table is the web content deployment system itself. A good quality system will provide all the automation features needed: logging, persistency, recovery, surgical precise file selection, etc.

Simple, isn’t it?!

Happy deployments

RepliWeb’s R-1 has 2 built-in Transport Engines designed for high bandwidth/low error rate (usually LAN) and low bandwidth/high error rate (typically WAN) networks. Both include a special Per-File Data Integrity Assurance feature that guarantees that no broken/half-baked files will ever reach the Target.

The way it works is very simple yet ultra robust: each file is first replicated to an administrator-defined temporary (hidden) directory on the Target host. Only after the file has been verified to be completely and successfully replicated it is than moved from the temporary location to its final Target destination via native OS rename command and than R-1 replicates the next file.

This powerful one little gadget actually kills two birds!! It absolutely guarantees that no broken/half-baked files will ever reach the Target destination (note that the original file is not touched until the final rename action). It also protects the file from being accessed during replication because the replication is performed into a temporary (hidden) directory where no User or Application will look for the file. The virtue of the rename command is that the whole file is being linked at once to the Target directory so there is zero exposure there.

Risks #1 and #2 – 100% mitigated!

Another mechanism built into RepliWeb’s R-1 is the Transactional Deployment which is a Quantum-Updated Integrity Assurance mechanism. ‘Quantum-update’ refers to the inclusive group of files that need to be deployed on specific host. The mode of operation resembles Two Phase Commit or in layman terms: “All or None”; ALL files are first replicated into an administrator-defined temporary (hidden) directory on the Target host, and are moved to the final Target destination only after two conditions are fulfilled. The first condition is ‘hardcoded’ – all files must have been completely and successfully replication (no point in moving forward if at least one file is damaged/missing). The second condition is selectable by the administrator and can be one of the following:

• As soon as all files made it successfully to the temporary directory
• At specific time of day
• When a designated Trigger File is created by an external application/process
• Upon named User approval

Risks #1, #2 and #3 – knocked out!!

RepliWeb’s R-1 includes another powerful mechanism – inter deployment synchronization points when running a Distribution (1-to-Many) deployments. Simply put, R-1 can make sure that concurrent deployments to multiple Target hosts progress through the same steps simultaneously. When set to synchronize just before the Transaction Commit point an administrator can extend the Transactional Deployment boundaries across multiple hosts, thus unless ALL hosts have received their Quantum-update in whole, NONE of them will be updated!

Risks #1, #2, #3 and #4 – busted!!!

Last, but certainly not least, is RepliWeb’s R-1 Rollback mechanism. This serves in conjunction to the Per-File Data Integrity Assurance and Transactional Deployment mechanisms, providing an insurance policy against wide variety of harmful affects of deployment including successful deployment of the wrong content. When used the Rollback engine will set aside every original file that has been touched by R-1 (i.e. overwritten or deleted) and will record any new file that has been introduced by R-1. At any time the Administrator can instruct R-1 to rollback in time and reverse the affects of all deployments as of specific point-in-time.

Risks #1, #2, #3, #4 and #5 – eradicated from the dictionary!!!!

Happy Deployments.

Here are some additional ideas, I welcome your thoughts (add a comment just below this post):

Risk #1 (Broken/Half-baked file) – I don’t think there is much you can do except for holding your fingers crossed and/or pray. One thing I saw people do is to run all deployments interactively so that any error message is immediately reviewed and handled accordingly. I don’t think this is too practical in real-life enterprise environments.

Risk #2 (potential access to file while it is being copied) – Again, this is typically out of your control. Perhaps the only thing you can do is to eliminate User and Application access to the Target host(s) while replication is running. For example, disengage a webserver from a load-balancer before kicking off replication to make sure it’s ’empty’. I’m not sure this is too practical in large scale environments.

Risk #3 (partial update of single target) – Nothing much you can do, except carefully sniffing the logfiles (and I do hope your deployment solution generates good enough logs) after each and every deployment.

Risk #4 (partial update of some targets) – same as Risk #3

Risk #5 (successful deployment of the wrong content) – stay close to your blackberry/cellphone/beeper!

Happy Deployments!!

Risk #1: broken or half-bake file at the Target.
This simple problem is actually one of the deadliest there is. The reason: it is close to impossible for IT to spot such issues, thus they are left to be discovered by the business and the users… not a good position to be in, you will surly agree.

The primer cause for such problem is the old-fashioned replication technology used. If you’re using FTP, copy, cp, Robocopy, Rsync, Xcopy and a handful of commercial products you are betting each and every file replication will be successful – but in reality you have no guarantee. All of these tools will replicate each file DIRECTLY into the Target location so the first thing they’ll do is to KILL the ORIGINAL file and only than replicate the new file to that location.

So what will happen if for whatever reason replication breaks halfway (network jitter, no more room on Target, etc. etc.) or if some of the packages came through corrupted? You’ve guessed it – the original file is lost and instead got KABOOM. It’ll be extremely difficult for you to recognize the problem because from the outside the file seem to be there so dir/ls check will mislead you. The problem is internal to the file thus will be noticed only by end users (either see partial data or corrupted data or may get an error message).

Risk #2: potential access attempt to the file while it is being replicated.
This is not really an IT issue but the user’s ‘fault’. File replication is never a zero-duration operation. Depending on the size of the file and the speed of the network/host/storage it will take from a fraction of a second to minutes and even hours in some cases. When a file is replicated directly into the Target directory (just like FTP, copy, cp, Robocopy, Rsync, Xcopy and many others do) it seems to be available for use from the very first moment (try to run a dir/ls command and see for yourself) thus humans and applications may attempt to access it WHILE IT IS BEING COPIED.

The result is either access to partial data (only the portion that made-it-through can be read) or more seriously an access-violation error on either the accessing application side or the replication tool.

Risk #3: partial update of single Target
This perhaps is the classic problem that everyone thinks about when considering replication. In simple terms: “not all the files made it” – the update was for 57 files but only 55 made it – 2 files are MIA!!

The risks are two: (1) the Target is not identical to the Source as expected, and (2) since the Target is not fully updated there may be functional problems (such as broken URLs) or compliance problems (with regulations such as SOX and HIPPA, or regulatory bodies such as SEC and FINRA).

Risk #4: partial update to several Targets
Well, if running singe Webserver may be difficult, think what happens when out of 5 web servers in a farm only 3 are completely and successfully updated. There is now an imbalance in your farm hence some users will hit the updated servers and some will hit the partially updated ones.

Like with most of the problems, this one is more quickly noticed by the business and the end users and not by IT team!! Special difficulty is to pinpoint the problematic host(s).

Risk #5: successful deployment of the wrong content
To put it plain and simple: Who said life’s fair? Everything ‘IT’ went well and its really not your fault that stupid Joe decided to deploy the wrong content. So what, it is your a__ that is on the hot grill, and you need to correct the situation ASAP!!

In my next post I’ll discuss techniques to mitigate those 5 deadly risks, but before we get there one comment about statistics. Yes, some of you will say “What is he talking about??!! None of this can/will happen in my environment!”, and the answer is pretty simple – it is all numbers’ game. Today’s hosts, storage and networks (even WAN links) are pretty good and fairly reliable, but on the other hand the amount of web content deployed these days is ever increasing and in some cases reaches imaginary (aka astronomical) proportions. Furthermore, the criticality of web environments to day to day business operations reaches new heights from quarter to quarter. So with so much traffic the error is just a matter of time – statistically it WILL happen (even to YOU!). The question you should ask yourself is What does it mean to me? What will be the cost to my Organization (and will I be the one to pay the price)?

Little counter intuitive, right? People have been coping files day in and day out for years and years, what’s the fuss? Well, the difference in our case is that web farm updating is highly automated (unattended) and frequently executed (typically several times a day) which increases the statistical probability of something-going-wrong.

To make things more interesting a typical WWWROOT is comprised of 15,000-30,000 to 120,000-300,000 individual files. Indeed not ALL files change frequently but even a minor change to 1%-2% of the site translates to quite a bunch of files to be replicated.

Finally, one has to keep in mind couple of factors that further amplify any glitch: the first is the immense business criticality of Web environments today (one can not say enough about the cost of down-time), and the second is the fact that Web glitches are usually noticed BY END USERS – and that is the least desired option of them all…

The 5 risks you take every time you synchronize a Web farm are:

1. Half-baked/Corrupted files in Production
2. Potential access to files while they’re being copied
3. Incomplete update of specific Web Server
4. Incomplete update of the whole Web Farm
5. Successful deployment of wrong content

How does your Content Deployment solution handles such situations? Are your operational procedures adequate?

(1) Is a local DMZ Web Farm. In this case we have 3 Web Servers each with its local DAS, behind a load-balancer.

(2) Is a Co-located Web Farm. Connection to such Web Farms is typically done over the open Internet, but in some cases a VPN or a dedicated WAN Circuit is available.

(3) The core of this Web infrastructure is the Staging Server. This is an internal server on which the entire website is assembled and usually tested (although in some cases an independent host is used for testing).

The Staging Server has 2 main purposes: (a) serve as the assembly area for all parts of the website (application and all content), and (b) serve as a master backup for all Web Servers.

This server is used to update all local (DMZ) or remote (Co-located) Web Farms.

(4) Web Developers and Content Authors will work on their individual workstations (a PC or MAC) and at some point will upload their finished products to the Staging Server.

Big question is how do they do it? Trivial methods include FTP and Shared Folders, however both impose major security/auditing/logging and other ‘change management’ issues.

(5) The Administrator bares the ‘monkey’ of how to replicate the Master website from the Staging Server to each and every Web Server. This is where a Content Deployment solution comes into play. The Administrator will use its Management Console to setup, execute and monitor Content Deployment operations.

(6) As Web operations become more ubiquitous and intense the basic model in which the Administrator is responsible for all Web Farm updates no longer holds and a growing need arises to empower non-administrator users to perform some or all Web Farm updates. Such non-administrator users are usually select developers and ‘business’ users such as Release Managers, QA Managers, and Business Line Managers.

Some Content Deployment solutions enable those power-users to transact via dedicated WebUI.

The definition I like the most is as follows:

“File-based content that is created in a Directory-tree on one host has to be repeatedly replicated (i.e. deployed) to other host(s) while preserving the Directory-tree structure.”

Well, not perfected but describes the idea pretty well…

The definition to Application Deployment is:

“The action of distributing an installed Application from one host to other host(s), while making the target copy ready-for-use.“

Been wanting to start a blog about Content Deployment for quite some time so I guess that now I’m in business…

Stay tuned – I’ll try to post something every couple of weeks. Probably once a month I’ll post something personal which is not necessarily related to the topic of this Blog.

Your feedback is ALWAYS welcome!!

Enjoy,
Motti.

Some things you should know about this blog:

• This blog is not affiliated or sponsored by any company
• I am affiliated with with one vendor in the field, however the purpose of this blog is not to promote specific product or service, nor the purpose is to throw mud at anyone (person or company)
• All trademarks mentioned are the property of their respected holders