MUSA Technology Partners - Technology Blog

Developing a Disaster Recovery and Business Continuity Plan

noreply@blogger.com (MUSA Technology Partners) — Mon, 01 Feb 2010 17:51:00 +0000

By Douglas Lantigua, MUSA Technology Partners

There are several techniques available to provide reliable Disaster Recovery and Business Continuance for companies, but before you can chart a path you must recognize that it is the nature of your business that will dictate your DR/BC requirements. Typically the business will tell you that your application or service can never go offline. That is, until you are presented with the cost of a 100% uptime scenario. Enterprise environments should be armed with a cheat sheet for business units to understand what they are asking for and how much it will impact the budget. If your IT department is in a chargeback model, the argument becomes simpler. Companies whose business requests and requirements become IT problems and budget busters need to be better armed with information.

Most companies do not start planning DR/BC initiatives early enough. It is far easier to implement a plan early (and then build upon that original plan) than it is to implement a new plan after you have hundreds of servers over multiple geographic locations. There are several techniques to determine which applications and services require special attention, and most are dictated by the business. However, since IT provides the core services that enable access to these business essentials, there’s a cascade of dependencies which also need to be accounted for. For example, it really wouldn’t matter if the ERP system was up and running if the end users couldn’t authenticate or locate the ERP system on the network. When mapping out the business-critical applications and services, make sure to design a highly resilient supporting infrastructure that is easy to maintain and keep in alignment with business and DR/BC goals.

Notice I’ve been writing about ‘applications’ and ‘services’ and not servers. This is because businesses care very little about a server, but instead care very much if an application or service is available. The server culture is under constant attack from clustering, virtualization and cloud computing deployments. As long as applications and services are available, the business will not care if half of the servers supporting this functionality crashed. Appliances are a different story. In the last decade, the trend was to deploy appliances to lessen the burden on IT departments for deployments. Today, most vendors are moving toward virtual appliances and you should take advantage of this trend.

Tier your applications and services in a list broken down into:
Uninterruptible Goal of 99.999% or better uptime (roughly 5 minutes of downtime per year)
Essential Goal of 99.995% or better uptime (roughly 30 minutes of downtime per year)
Critical Goal of 99.99% or better uptime (roughly 1 hour of downtime per year)
Important Goal of 99.95% or better uptime (roughly 4 1/2 hours of downtime per year)
Enhanced Goal of 99.9% or better uptime (roughly 9 hours of downtime per year)
Standard Goal of 99% or better uptime (roughly 3 1/2 days of downtime per year)
Other – Can be down for multiple concurrent days

Companies that use a chargeback model should include DR/BC costs against this chart or a modified version of this chart. Calculate the overall cost and amortize for the life cycle of the implementation.You may find you don’t want all of these levels of uptime and would be better suited to only offer a couple of levels. It is important to differentiate between Disaster Recovery and Business Continuity. You can argue the definitions for years, but to make it simple, I’ll break it down this way: Disaster means that your datacenter or facility is completely offline for an unacceptable amount of time where people/applications/services need to be relocated to continue normal operation of the business. Everything else falls under Continuity. It is far easier to recover from an issue without moving people/applications/services. The Business Continuity is mostly a site-specific action plan with a couple of modest exceptions relating mostly to infrastructure.

Your typical SunGard type facility works with tape restoration. This can take several days to get fully back on line before people can work and your core business is functioning again. Unless your facility is completely unusable for a period of time, this option is perfect for a pinpoint Disaster like a fire, but fails the local community when multiple businesses are in need of the same facility.

Only the business can determine the criticality of each deployment. Infrastructure will need to respond by making the core services available at the same service level or better. For example, if infrastructure is responsible for ensuring a database-driven application deployment needs to be available at 99.999% uptime, this requires every dependency to have even better availability. Consider the database, network, servers and connectivity to the system all have 99.999% uptime; if each of these dependencies has a 5-minute outage, the overall deployment has a 99.995% uptime. The business should understand the cost of requiring multiple 9’s right of the decimal point. The cost incline isn’t as steep as it was a couple years ago, but the cost can be significant and multiplied if the business and IT organization doesn’t take a holistic view of DR/BC.

For help developing a disaster recovery and business continuance plan, contact MUSA Technology Partners, experts at DR/BC solutions.

Digg This!

The DISASTER in Disaster Recovery

noreply@blogger.com (MUSA Technology Partners) — Thu, 21 Jan 2010 18:02:00 +0000

by Warner Jones, MUSA Technology Partners, Cambridge, MA

Most people today agree that business data and applications (email, financials, supply chain, etc.) are key components of their business operations. Many will state unequivocally that their businesses simply could not operate without them. Yet, people address this dependency with varying levels of commitment to protecting it. Some people operate under the assumption that nothing bad will ever happen to their data and applications - those people are often surprised in the worst way at the worst time. Others recognize that something bad might happen, but they figure that they can create a work-around (using paper files and manual processes) if that ever occurs – those people often underestimate what they can cobble together in a crisis. Some people do understand the full extent to which a loss of business data can affect their company and recognize that some companies will never recover from such a loss, but not nearly as many people put serious thought into this issue as should.

So, what could possibly happen to your business data and applications?

A disgruntled employee deletes data from your server.
A virus corrupts your data.
A hard disk crash occurs in your file server or mail server.
A fire breaks out within your datacenter or server room.
A loss of power occurs due to storm damage.

Any and all of these will interrupt communications within your business as well as between your business and your clients. In fact, there are myriad reasons why you need to develop a plan to first back up your data and, second, have an alternative plan to run your business if your primary datacenter is down for an extended period of time.

Business Continuity

Disaster recovery is actually part of the larger topic of Business Continuity that also includes planning for personnel, facilities and communications in case of disaster. For this discussion, we are focusing specifically on IT infrastructure and planning for the eventuality of a disaster that affects your data and applications.

Time is Money

In order to adequately assess your options, you need to answer some tough questions:

How much data can you lose in the event of a failure?

Could you survive without the archive of your emails from the last 12 months?
How would losing the Excel, Word and data files you’ve created in the last 30 days affect your business?

How long can you be without your data, business applications and email?

How would you run your business without email for a day? What about a week?
Could you get by without access to your financials for a few days?

Possible Courses of Action

Each of the following are possible courses of action - progressing down the list increases the security and reliability of the course of action to turn it into a solution for your business:

Do nothing and accept the inherent risks of data loss and downtime.
Back up your data to portable media and store it off-site.
Back up your data off-site to a third-party online service.
Back up your data to another company location.

Certainly, solutions for disaster recovery will vary depending on the size of your company and often on your budget. Larger companies have more data, but they also usually have more money to implement complete solutions to ensure their continuity of service.

Assessment and Planning

The first step towards understanding an appropriate solution for your business is to conduct an assessment of your internal systems and your risk tolerance. MUSA Technology Partners can assist with this evaluation and provide recommendations for solutions that are relevant to your feature and budget requirements.

For a more technical discussion of how a large company can leverage its Virtual Private LAN Service (VPLS), visit the article by Doug Lantigua on Networking for Disaster Recovery and Business Continuance. This article also discusses the Run Book, Recovery Point Objectives and Recovery Time Objectives.

Project Management in Life Sciences - Notes from the field.

noreply@blogger.com (MUSA Technology Partners) — Thu, 21 Jan 2010 18:00:00 +0000

By Richard Wood, Director of Program Management, MUSA Technology Partners

PART 3 - Change Management is My Friend

Many years ago at MUSA I was taught something by the man who built the electronic payment process for Visa.

He said. “[Rich] Wood-Any IT system or project you build are only as good as the change management process and the backup plan.”

This man knew what he was talking about. In Life Sciences, change management for projects is absolutely critical. MUSA’s MPMA methodology has significant Change Management controls. The design is a delicate balance between rapid deployment and a steady and cautious process.

We can do both.

Let’s go back with Doc Brown and Marty McFly to see a perfect example of both.

August 2008

“This is Richard,” I said when the phone rang.

“Uh…Hello? Is this the technology guy at MUSA?” said a voice on the other end of the line.

“How can I help you?”

“So, for our new warehouse we have two closed systems….”

There were two major systems installed for the warehouse to monitor building functions and control systems. With biologics, the complexities of live material handling are immense. These new systems were absolutely critical to the operation of the new plant and IT was told unequivocally that it would not be involved with either environment.

The request now changed to allow firewalled integration with these complex systems we’d never seen before into the corporate network. This was to allow access for remote monitoring, log capture and disaster recovery. I guess the millions spent to install these systems didn’t include a few things. And it needed to be done by the end of the year.

To give you an example on how difficult this was to be, let me use an incident from the old Apollo Moon program.

When the oxygen tank on the Apollo 13 command module exploded, the three astronauts (James A. Lovell, Jack Swigert, and Fred Haise) needed to use the Lunar Lander as a lifeboat in order to survive the return to Earth. However, the CO2 scrubbers in the LEM were built to handle only 2 people.

To solve this problem, NASA and the astronauts devised a plan to use the filter from the damaged Command Module in the Lander; however, the actual filters from each space craft were completely different shapes. A lot of duct tape later, the new scrubber was installed and the three men made it back to Earth.

We were going to need a lot of duct tape to make our clients systems integration work.

Using MUSA’s Change Management procedures, we simultaneously put in for a budget increase, a request for networking and architectural personnel and set up a sub project to manage the process. I took this one on myself as our other project manager on the manufacturing project was already working at capacity - and you should never over burden a former Army Ranger.

The sub-project was spun up within a couple of days, and we identified a budgeting source for the work by the end of that week.

Normally, an integration project of this scope and complexity would have run in parallel with the construction of the new facility. In other words, from design to ‘go live’ should have taken eighteen to twenty-four months. We had ninety days.

The change management process is taken care of exclusively by the project manager, in this case, me. This allowed engineers from each of the diverse systems and from the IT group to focus on the design, architecture and security elements of the sub project. MUSA was in charge of these aspects of development and deployment while the internal IT group for our client covered security and run-state operations.

The sub-project was successful, delivered on time and under budget, because of the dedication of the engineers and the change management process that put hard parameters around the unforeseen requirement without interfering with the work that needed to get done.

The Life Sciences industry requires a great deal of controls and process management, not to mention documentation. MUSA not only complies with these requirements, but it’s the well developed, streamlined process developed specifically for technology consulting in this industry that ‘gets the job done’ while being able to stand up to validation requirements and quality controls.

Technology consulting is a service, after all.

Next up: PART 4 - Deployment

The Life Sciences Industry: 2009 and Beyond

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:14:00 +0000

By Puneet Lakhi, MUSA Technology Partners

Looking back on 2009 as it comes to a close, it has been a tough year for most, but there appears to be hope for the future, especially for the Life Sciences industry. The recession has hit many companies in numerous industries very hard and continues to offer challenges to many, but economic indicators show that the economy is improving. The national unemployment rate is still in the double digits, but according to a December 4th report from the Bureau of Labor Statistics, the number has dipped from 10.2% in October, to 10% in November. According to another survey from the bureau, there were 11,000 jobs lost last month, a “vast improvement over the 111,000 jobs lost in October and the 597,000 jobs lost last November.” (Boston.com)

The Biotechnology and Life Sciences industry has shown itself to be an anomaly in the economy, and appears to be in better shape than most. According to Mark McClellan, director of the Brookings Institution's health-care reform unit, “the health-care industry in general is growing much more rapidly than the rest of our economy.” This trend has been confirmed by many, including MarketWatch, which says that “through the decade, the industry has proven it’s not only recession-proof. It’s also remarkably lucrative by any measure. Along the way, those riches have turned the sector’s stocks into Wall Street Darlings… From drug makers to biotech firms to insurers, all of the subsectors within the industry showed healthy gains in one realm or another. And every one of them is expected to turn a profit in what has been a difficult year for most other companies.

This is great news for Massachusetts, which relies heavily on the nucleus of high-tech and biotechnology companies in Boston and Cambridge. According to WickedLocal.com, in 2008, the Massachusetts Legislature approved the Life Sciences Act, a “$1 billion initiative that promised to bring biotechnology companies and jobs to the state.”This act was spearheaded by Governor Deval Patrick and Senate President Therese Murray, who have been quoted as saying that “"Every new job created in the life sciences results in two additional jobs in support services for suppliers, vendors and construction. What's good for the life sciences and biotech is good for Massachusetts." With government funding, biotechnology firms continue to grow and expand, including Bristol-Myers Squibb, which will open a $1 billion manufacturing plant in Devens, MA in 2011, further proving that the Life Sciences industry indeed is recession-proof, largely due to government funding and the relatively stable demand for healthcare products, regardless of the state of the rest of the economy.

Preparing for the VMWare Certification Test

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:13:00 +0000

By Jared Falcon, Solutions Consultant, MUSA Technology Partners

Preparing for and taking the VCP-410 exam is not for the faint of heart. VMware recommends the following path for those that are new to VMware:
1) Take the required course (VMware vShpere: Install, Configure, Manage)
2) Gain the required experience
3) Take and pass the VCP-410 exam.

From a “boots on the ground” perspective you will definitely need expierence on the product itself, especially with troubleshooting and best practices in a multi-host DRS and HA environment. Most people have at least a year or two of intense VMware experience before they even take the exam. It is wise to prepare the best you can beforehand since the required course is upwards of $4000 and taking the exam is $175 a shot.

As far a preparation goes, there is an exam “blueprint” on VMware site. The blueprint lists all the topics that are covered in the exam and is the official guide. Some topics can be reviewed using the product documentation, but it is very important to have the hands on experience for the bulk of the questions. Also, knowing the best practices and maximum and minimum parameters of vSphere is also very important.

Here is a quick checklist for those who are preparing for the exam:

1) Know the maximum and minimum configuration parameters (there are a few max and min practice exams online)
2) Know the best practices for vSphere
3) Know the new features in vSphere as compared to ESX 3.5
4) Know the new vSwitch architecture down pat
5) Review all the mock exams online (a quick google search will help you find them)
6) Know where common menu items are and what you can do on particular screens and within the vClient.
7) Know how to deal with and troubleshoot vSwitch network issues, storage issues and resource contention (this is where the hands on expierence comes in)

There are also a few sites on the internet that can help guide you further through the process. A word of advice from someone who has gone through this process: do not take studying for the exam lightly, but if you are comfortable with the product and spend a great deal of time studying, you will find it to be a very rewarding certification to have on your resume.

Why become VMWare Certified?

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:12:00 +0000

By Jared Falcon, Solutions Consultant, MUSA Technology Partners

The VCP is one of the most coveted certifications in today’s technology market, and refers to a vSphere Certified Professional. A quick look around the IT industry will tell you that many organizations or moving towards virtualization and will be for the foreseeable future.
While there are many reasons companies opt for virtualization, the primary ones are the relative ease of management, lower cost of capital, increase in ROI and reduced energy consumption (going green). At the forefront of the virtualization market is VMware, specifically its ESX, ESXi and vSphere products. That being said, seeking a VCP certification is both advantageous for an engineer’s personal career and for the firm he works for. It is one of the few certificates next to the CCIE from Cisco that can actually help you demand a higher salary and attract more clients to your business. Due to the experience and dedication required, there are currently not many VCP’s out there, hence making it a more desirable certification.
The VCP certification allows one to demonstrate that he has the experience and knowledge to work VMware’s flagship products. Thankfully, unlike some other technical certifications, the VCP requires prerequisite knowledge and experience of the products to even qualify to sit for the exam. This helps reduce the problem of “paper” experts, where one can pass the exam simply by reading a book or manual. The new VCP-410 exam is more in depth than even the VCP-310 exam, which is the ESX 3.5 products certification. This test requires a great deal of background knowledge and experience, and can be quite grueling. However, all the preparation and studying will pay off when you pass the exam and become recognized as a vSphere Certified Professional.

Is Server Virtualization Here to Stay?

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:11:00 +0000

by Puneet Lakhi, MUSA Technology Partners
Server virtualization has been touted through BusinessWire as “one of the biggest trends in recent IT history.” Is this hype justified or is this just another passing fad? Virtualization, for those who are not familiar with the concept, refers to dividing one actual physical server into multiple unique virtual environments. This consolidation of servers is undertaken through products by vendors such as VMWare. By allowing administrators to make more use out of one server, virtualization has become so appealing that according to CMI Corp. research, “80% of companies [are now] considering the [cloud] computing model at some level.” Even though the concept of collocation and cloud computing is literally and metaphorically hard to grasp, it appears that there is justification for at least most of the hype about virtualization, because of its savings of money, hardware, and physical space.
One of the reasons why virtualization is increasingly being seen as a viable solution for businesses of all sizes is its ability to control server sprawl. If you’ve ever seen a server room, you’ll know that space and crowding often becomes an issue. What many people don’t know is that most servers typically only use about 10% of their capacity. Through virtualizaton, this one physical server can run numerous different environments to process multiple tasks and processes at once.
Further, virtualization has been heralded by many as the latest way for a company to go green. Since as mentioned above, virtualization allows a company to get greater capacity out of one physical server, it leads to the company needing fewer machines. Thus, the environment is spared of the electricity usage of multiple servers, as well as the burden of producing and ultimately, disposing of the raw materials that an additional physical server would be composed of. So it is apparent that through virtualization, not only can a business further its progress in environmental sustainability, but it can at the same time help cut its bottom line, allowing for the proverbial killing of two birds with one stone. This stone, dubbed virtualization, appears to be here to stay for the long haul.

p.s. For a very comprehensive guide to how virtualization actually works, check out this very informational article on HowStuffWorks.com, http://communication.howstuffworks.com/server-virtualization.htm.

Part 1: Networking for Disaster Recovery and Business Continuance

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:08:00 +0000

Part 1: Networking for Disaster Recovery and Business Continuance

By Douglas Lantigua, Principal at MUSA Technology Partners

Effectively solving the problem of corporate Disaster Recovery and Business Continuity (DR/BC) starts with proper planning and networking. A company that owns only a few servers or a complete datacenter will need a failover location and plan known as the Run Book. The other location can be a collocation facility, another business location, other service providers or some sort of hybrid. The business needs to address key questions such as: “How much data can we afford to lose in the event of a system failure?” The amount of possible data loss, measured by either data or time, will help direct the DR/BC solution. The business then needs to identify the Recovery Point Objective (RPO). The RPO is the acceptable level of data loss measured in time (i.e. 5 minutes or 4 hours). The RPO is married to the Recovery Time Objective (RTO) which is the amount of time it takes to get critical systems back into a functional state.

How does networking assist in the RTO and RPO objectives? Larger companies can take advantage of Virtual Private LAN Service (VPLS) to extend the datacenter network to another physical location. Unlike its cousin Multiprotocol Label Switching (MPLS), VPLS acts on a lower level of network activity. Users and computers connect to systems by name (a friendly translation of an IP address). MPLS works at the IP level. Geographically dispersed locations connected by MPLS need to have different networks (or IP address ranges). Even if you fail a server over to another location using the latest application technology, you will still need to change the IP address of the server, and you will probably need to change dozens of other attributes in the network modified to bring the system back into an operational state. The failover procedure and plan are compiled and updated in the Run Book.

VPLS’s key advantage is that it works at a lower network level than the IP address; it works at the machine address (MAC address). This makes the IP address transferrable anywhere in the network. So a failover of a system can move geographical locations and still maintain its’ IP address and remain reachable by users and computers alike. VPLS can be expensive, but there are alternatives for companies on a budget and those who do not need the large bandwidth requirements most VPLS providers mandate. Companies with sub-VPLS requirements can use IP tunneling and/or channeling to achieve the same goals. By extending the network across a geographically dispersed location(s) at the machine address level (layer 2), you allow the IP addresses of the servers to move freely. The latest in virtualization technology and storage replication makes an aggressive RTO and RPO very inexpensive.

The Run Book is the instruction and procedure plan on how to handle DR/BC scenarios. Given in previous scenarios of failovers where the IP needs to change of the system, the dependency on the IP address can be far reaching. Not only would the server need to change its IP address, but the name to IP (DNS) relationship, connections to data sources, internal application settings and finally, the end user network path to the server/service– which could include dozens of pieces of network gear, will all need to be updated. These types of systems are set up over days/weeks or months when originally deployed, an emergency change under tight deadline for a single system could be difficult even under perfect preparation. Then assume people get busy and the Run Book doesn’t get updated when changes occur. The Run Book then becomes a massive paperweight and budget nightmare to maintain effectively.

Leveraging a geographically dispersed layer 2 network either by VPLS or IP tunneling/channeling shrinks the DR/BC run book, allows the staff to fix the original problem and frees engineers to solving unforeseen issues. Any failover involving IP address changes is fraught with time consuming issues in order to bring missing critical systems back online. Those industries with heavy compliance requirements are in need of simple solutions to meet regulation standards. The networking base does include an upfront investment for setup and enough bandwidth for failover. Managers must maintain routine checks that enough bandwidth is available for a catastrophic failover event of critical systems. Secondary access points should be considered to the failover location if key users will need to perform their job function from outside your network for a prolonged period of time. Routine testing of failovers should be part of the standard operating procedure (SOP) of the IT/IS department. The network is only one part of the overall picture. With a flexible, geographically dispersed network the ground is fertile for system and application failover tools to work their magic with the least complications to achieve success.

PART 2 - SOP Hell

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:06:00 +0000

Project Management in Life Sciences - Notes from the field.
By Richard Wood, Director of Program Management, MUSA Technology Partners

The Life Sciences’ vertical is tough for IT folks, whether they are ‘in house’ resources or consultants--the job is more difficult then in other fields.

There is an obvious set of reasons for this, of course.

The Life Sciences industry is dedicated to the betterment or saving of lives. Which means the field is highly regulated, as it should be.

This means that the systems that are used to aid in the discovery, production, approval and distribution of biologics and pharmaceuticals have strict regulation as well. Not only from a validated perspective, but from our own controls outside the vertical’s needs as well. Sarbanes-Oxley comes to mind. But that’s a topic for another blog post.

Take, for example, the clinical trials system at our client that we built the IT portion of a new manufacturing facility for.

It’s not ONE system. But THREE. There is the test or ‘Sandbox’ version where versions of software, new patches of the OS and many other things are….well…tested. The next version of the system is the ‘Validated’ environment. When all the testing is down in the Sandbox, patches, upgrades etc., are uploaded and documented for the quality team. There are Installation Qualifications (IQs), Operational Qualifications (OQs) and Production Qualifications (PQs) to name just a few things that need to be scripted, tested, documented and signed-off on. MUSA over the years has developed a series of templates to greatly speed up this process, but there are still a lot of people involved to ‘validate’ the changes. The templates we’ve developed fit right into the MPMA project methodology and give extremely accurate and detailed timelines and task lists. This all helps lead to proper documentation and delivery.

The third system is of course the Production environment. Once the documentation is finished, the validation has been run through and Quality has been signed off on, only then can production commence. And again the validation teams must check and sign-off for every step of the process.

So you can imagine what the chore of adding monthly Microsoft patches to existing validated systems can be like. Think about that next time you find an old NT 4 system running an application for a scientist somewhere.

I spoke in my previous post about a brand new manufacturing facility for one of our clients in the Life Sciences business. More specifically, this client specializes in biologics, which means instead of making powders or liquids for pills (for example) they are dealing with live cultures.

Taking into account my little speech about three tiered validated environments above, and the fact that the new manufacturing plant will be dealing with biological material, we once again get in Doc Brown’s Delorean to visit the main topic of today’s post.

March 2008

I love coffee. It’s been my friend since I discovered I’d spent my entire year’s spending money budget in the book store on my first day of college and realized I needed three jobs to make it.

The problem with coffee (for me anyway) is that when I don't have it first thing in the morning, things can get ugly.

I got to our client’s office on a rainy Monday morning after having spilled my mug of caffeinated goodness on the ride in after getting cut off on the highway by a Prius. Our client provides free coffee, so I was about to make a cup, when my ex-army ranger project manager walked up to my desk and dropped an entire national park’s worth of paper on my desk.

“We got the first draft of the SOP’s for the new building. There’s about a hundred of them.”

“That’s a lot of paper,” I responded wittily.

“That’s just the SOP for the PCS system. There’s ninety-nine more. They take up three DVD’s”

And then he walked away.

Well, needless to say we didn’t have to review them all, and we had a large team from MUSA to pour through the technical details of the ones we did need to review. Between collaborative meetings and review changes we finally got what we needed done a few months later.

Keep in mind that while we were reviewing the SOPs, steel was going up at the building site. No moss grows under the feet of this client!

By summer of ’08 we were pretty happy with the SOP approvals and our submissions. The actual planning and vendor coordination was next and I moved from direct involvement in the project to program reporting.

After all I had a PM for the project and our SOP’s were all set. What could possibly go wrong?

Next up: Part 3 - Change Management is my Friend

Setting the Bar for Lab Environment Technology Support

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:06:00 +0000

by George Wilgram, Solutions Consultant at MUSA Technology Partners, Cambridge, MA

As analytical platforms in lab environments become increasingly complex, IT professionals are all the more pressured to stay up to date and support these systems. Life Sciences labs in particular call for an atypical approach to instrument support. Because lab processes - especially in the biotechnology and pharmaceutical industries - are often the core business function, it is critical that support for these activities affords the highest level of instrument integrity and performance.

Supporting this type of environment is not simply about troubleshooting and fixing problems. It involves a more in-depth, specialized approach and methodology, tailored to the unique demands of each instrument, the processes involved, and the people who use it.

MUSA’s approach to this problem is to use consistent project management principals to resolve these complex situations. We combine problem discovery and identification, and project planning, scheduling and testing into a comprehensive service approach.

For example, a large biotech firm recently ran into a problem migrating their AKTA instruments and data onto their corporate network. They had developed a plan to configure the instruments, and when they went live with the transition, they discovered that they could not save data to the network as expected. This was a considerable disruption that caused a great deal of confusion and delay because of the large number of instruments affected.

Given the scope of the problem, it was obvious that a solution had to be found quickly. At the same time, however, it was very clear that the solution absolutely had to be the right one, and that further setbacks and delays were not an option. MUSA’s approach to this task was to break it up into two phases: Immediately identify the root cause of the problem and develop an effective solution, and then, as soon as possible, develop a new strategy to get the AKTA’s and the project back on track.

The most important component of MUSA’s project management approach is the discovery process. This goes much further than that simply identifying the problem. Just to start, it means delving in and understanding the instrument as a whole – how it works, how it’s used, and what the end user requirements are. We work with the vendors, identify who the instrument experts are, and obtain as much information as we can about the problem at hand, as well as all possible solutions and options. It’s an exhaustive process, but it can be done quickly and effectively if you know the right questions to ask, and have right people on your team. In the case of the AKTA’s, this process allowed us to quickly identify and present the most effective solution, one that we may not have found otherwise.

Once this step was completed, we moved onto the second critical phase, the implementation strategy. Essentially, this became a “micro-project”, and we developed a schedule, identified tasks and activities, key users, as well as individual data backup and configuration strategies.

Next, we needed to be sure that the proposed solution was stable, reliable, and performed to expectation. To do this, we “piloted” the solution with a small number of instruments and users. Any unforeseen issues or complications were identified, a solution proposed, and subsequently re-tested. At the end of testing, this new solution was confirmed and finally, applied to all instruments.

We scale the above process and tailor it according to the size and nature of the problem. The AKTA scenario described above is fairly large in scale due to the number of instruments, departments and users involved, as well as the stringent project requirements. However, we make sure to apply these same basic principles to singular instruments and smaller projects to insure the integrity of the instruments and data at hand. Our primary goal is to meet or exceed service expectations, and we have found that sticking to this tried and true process helps us deliver the consistent results that keep our clients in the life sciences industry coming back to us for consultation of their technical issues of their lab environments.

Is Domestic Outsourcing Here to Stay?

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:04:00 +0000

By Douglas Lantigua, Principal of MUSA Technology Partners

During the 90’s, the migration of IT support jobs overseas was a natural move. Flocks of highly educated and motivated individuals in the United States found themselves without a rich localized technology sector looking to them for help. The United States and the rest of the developed world looked overseas for the highly qualified teams of individuals who would work from their respective developing nations for the price of one local resource. This was the proverbial “no brainer” even in spite of the cultural misunderstandings or the overall trouble of making such a transition.

For a chance at these newly created jobs, the overseas workers were more than willing to work overnight to be in line with client needs. Pioneers who jumped on board early received skilled and motivated resources at a reasonable price. This made the cost of transition and the language and cultural differences with workers in the developing world bearable. Outsourcing overseas was a risk many made, and it paid off early.

However, a decade later, the story has changed drastically. With an influx of immigration into the United States, along with the unmatched technological education available here, the local technical environment is richer than ever not only with multinational opportunities, but also with local ventures that do not require overnight work and still pay a good salary. The laws of supply and demand have spoken, and the huge influx of outsourcing over the last decade has drained the remaining talent pool. These positions are no longer coveted and the salary can be beat. The resources overseas are coming right out of college without the proper experience and communication, and are hastily thrust into roles supporting critical systems.

The move back to a support model that is based in the developed world is the first of several back and forth waves yet to come over the coming decades. The four or five foreign bodies to one local body ideal no longer holds true today. A local resource can now be of service for the price of two or three foreign resources, without the cost of cultural differences and drastic time-zone differences.

These individual domestic IT workers are on average more reliable and better equipped for success than the resources currently available overseas. Eventually, there will be a global multi-vendor support model, broken down by regional centers of excellence and accessibility. For now, those willing to bring the work back to the developed world will reap the benefits.

PART - 1 “Oh, by the way…”

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 20:01:00 +0000

Project Management in Life Sciences - Notes from the field.
By Richard Wood, Director of Program Management, MUSA Technology Partners

PART - 1 “Oh, by the way…”

It’s done.
One hundred and fifty million dollars spent. Twenty-one months of planning and construction. Nine separate IT vendors’ managed. One hundred and fifty staff now working in a brand new 200,000 square foot state-of-the art bio-tech manufacturing facility. And MUSA ran the IT portion of this effort utilizing our MPMA methodology specifically designed for rapid deployment of IT projects for life sciences. Although geared toward application and infrastructure projects for our chosen vertical, we were able to quickly adapt our methodology for a validated manufacturing facilities project. Yay team.

So how did we get there?

I’m writing up my notes from the Post Mortem we just completed. It’s 11 PM and I’d be lying if I told you I didn’t have a glass of celebratory wine in hand while tidying up the paperwork loose ends. ‘How did we get there’ is a great question. At this particular client, projects seem to start the same way:

“Hey Rich,” asks the Director of Infrastructure for our Bio-tech client

“Yes?” I ask, knowing the tone of voice.

“So I just found out we’re building a new manufacturing facility at the new campus. Got a minute?” Says the same Director.

Project Initiation had begun.

Pack up a lunch, and strap yourself into the Delorean. With the Flux Capacitor using 1.21 giga-watts, we approach 88 miles per hour…

January, 2008

Our client just put the finishing touches on their second building at a new campus designed to bring the entire Massachusetts business unit into the same location. MUSA is working through the details of a new Roller Bottle system (if you don’t know, Roller Bottle is a manufacturing method for cell culture growth).

My job at this particular client is running the global infrastructure PMO. At MUSA, we firmly believe in practicing what we preach, so everybody in management also tend to have gigs as consultants. It’s how we refine ourselves as a consulting organization and keep in step with the Life Science’s industry as a whole.

Anyway, the Roller Bottle project is humming along when I get the call into the Director’s office.

He’s got a set of architectural site plans for the new campus and I notice a big rectangle taped on it in between two existing buildings. He points at it.

“We’re building a new Manufacturing facility. And we need to ramp up a project right now as they are in the design phase. Talk with Facilities Planning.”

So I did.

MUSA works with many partners. We have a simple philosophy that the client needs work done and, especially in Life Sciences where outsourcing is the norm, there tends to be a lot of consulting companies working in an organization at any given time.

Two years beforehand, I met a seasoned Project Manager (Not with MUSA, but with a company we’ve worked with many times in the past) that I knew would be perfect for this new manufacturing facility project. He’s an ex Army Ranger, and currently in the reserves. Which means for two weeks out of every year this man willingly jumps out of airplanes. His credentials are amazing, he has military discipline when it comes to running a project, he knew that our client had adopted MUSA’s MPMA methodology and was very familiar with it (having trained him myself). And he’s a little crazy.

Perfect.

He and I met with Facilities Planning and proceeded to setup the project. Within two hours, we had the charter written, the Project Initiation documentation to the IT group, a kick-off meeting scheduled and were pouring over the plans for the new building.

We were working in silence over those plans for an hour, occasionally making comments.

“We’ll need a second IDF closet on the second floor. The cable runs are out of spec.”

“Yep.”

“They’re putting in a completely separate manufacturing system. The Network guys are gonna have to put in a new firewall.”

“Uh-huh.”

“Those clean rooms near the bio-reactors have high ceilings. We’ll have to run a wireless survey simulation”

“I know.”

“The building is going to be one big Faraday Cage. Did they budget for a cell repeater system so the push-to-talk will work?”

“Yep.”

And so it went. The move-in date was scheduled for end of Q4 2009.

Our Army Ranger partner Project Manager left the conference room with plans and I was alone to think about the big picture. We’d developed the MUSA MPMA for Life Science system deployments. Would it work for an entire validated facilities project as well?

When MUSA originally used a PMI model (and a Prince2 model in Europe) we found very quickly that we PMs were getting in the way of real work. In bio-tech, where applications used in production can sometimes be only a step or two removed from the ‘basement development model’ or where scientists throw an application on a Linux PC under their desks for Mass Spectrometer analysis, there isn’t a lot of tolerance for what was perceived as project management ‘bureaucratic overhead.’

But we all know, especially in a highly regulated environment, that the PM process is critical for success. So how to eliminate even the perception of ‘overhead?’

By using a sophisticated blend of collaboration, task management streamlining and a unique dash-boarding system for online real-time reporting while maintaining the relationship touch necessary to get the job done. Our PM’s manage the process and let the engineers’ just get on with it.

The process was developed for system-based projects. Would it flex enough for a large-scale building initiative?

The short answer was yes; with some minor tweaking to our standard communication plan templates and our vendor management plan. But more on that later.

To be continued... PART 2 - SOP Hell

Closing The Gaps in Open Source

noreply@blogger.com (MUSA Technology Partners) — Thu, 17 Dec 2009 19:57:00 +0000

By Josh Campbell, Senior Solutions Engineer, MUSA Technology Partners

Recently at MUSA Technology Partners, we've been upgrading; we've upgraded our office, increased our staff, and decided to redesign our infrastructure to better support the pace at which we've been expanding over the last few years. In the past we had made a conscious decision to avoid building out too much of an infrastructure, we were an extremely mobile group and tended to live on our Blackberries, so we didn't have a need for it. Time goes on. We've doubled and doubled again (and then some) and we now need internal support for a much larger group of people, including non-technical office staff, which means we need IT infrastructure.

Since we're a consulting shop with a major focus on Open Source software, we decided to build our entire infrastructure using Open Source products. The goal was to take some of the available products, get them running, and then make them all work together in harmony. The first two steps were completed relatively easily. I'm actually quite impressed with the strides the community has made with regards to documentation and product offerings since the last time I undertook an Open Source project of this size. While there's been great individual advances, it seems that a familiar problem arose when I tried to knock step three off my list, weaving together all the individual projects into a coherent infrastructure. Don't get me wrong, I didn't expect this to be the easy part, I was just surprised at how little had changed since my last trip down the road less traveled.

There are a number of extremely valuable products in the community that would only benefit from improved interaction with other projects, why this hasn't really happened yet I'm not sure. The Open Source community has done an incredible job creating polished products to serve just about every niche, however it's time to focus some resources on bringing the individual products together in a more meaningful manner. While I'm not suggesting the forward motion of feature development should stop all together, it would be nice to see some of the more refined projects move some of their development effort into increasing compatibility and building a framework to allow more meaningful interaction with each other.

There are many excellent projects out there for building intranets and web sites (CMS), managing client data (CRM), and managing the resources of an enterprise (ERP), but these products would be exponentially more useful if they could be meshed together by non-developers. By improving compatibility and focusing on closing the functionality gaps inherent in each individual offering, CMS projects could serve as an accessible and easy to use entry point for CRM and ERP systems. Using the strength of CMS systems to serve content to users, CRM/ERP developers could stop focusing development resources on client applications, leaving them free to strengthen the core of their offering and add additional features, further increasing the value of their project. Leveraging a web based CMS system also provides the added benefit of being compatible across a number of platforms with minimal development overhead. This is just one example of how better collaboration between projects can help to improve the experience for the user base as well as provide developers an opportunity to free themselves from spending development time on the more peripheral parts of their software.

With so many high quality individual projects in the community the future of Open Source may well be determined by the ability of many projects to work together to close the gaps, producing a cohesive environment, and providing a real challenge to some of the commercial suites.