TCP header, Lego (tm) style

The older I get, the more lessons I seem to learn (or, not learn) over and over.  Have you ever seen TCP offload work correctly?  Of course not!  I’ve been bitten by a TCP offload (aka TCP Offload Engine or TOE) problem in just about every environment I’ve touched in the last 20 years, and sadly this week was no exception.

To make a long story short, we have a production vmware ESXi 4.1 host with both Linux (CentOS) and Windows Server 2008 guests.  No problems were reported (or measured) with the Linux guests, but the Win 2008 guests suffered from extremely choppy network connections, for common services like Remote Desktop and backups (including lost connections).  As you probably know, I’m big into actually investigating the underlying cause of a problem rather than randomly throwing darts at it, and as such I grabbed some packet traces with wireshark.  Check this out:

wireshark analysis of poor TCP connection

Ouch! That is super ugly (this is across a LAN, btw)!  How can you screw up a single TCP connection so badly in 6 feet of cable?  Probably not the cable (or network), sherlock.  It appears this is a “known problem.”    While this problem (described in the vmware article as “Network performance is very slow and connections drop intermittently”) seems contrained in the article to vmware guests running on a Windows host, I can attest to this occuring on both ESXi 4.0 Update 1 and ESXi 4.1 hosts with Windows guests.  After following the instructions in this vmware article to remedy the situation by disabling TCP offload on the Win 2008 guests, they exhibit downright snappy network performance.  Check out the improved trace results:

wireshark analysis after disabling TCP offload

Moral of the story:  TCP offload always sucks.  Turn it off on Windows Server 2008 vmware guests.

Writing a book of this magnitude is an intense process that I learned all about. The steps to produce the book, from inception to dead trees, include:

• Brainstorm and agree on full topic list
• Brainstorm and agree on contributing authors
• Assign chapters to authors and contributors
• Write chapter, distribute for review
• Integrate reviewed comments from all other authors, distribute to external reviewers
• Integrate external review comments
• Repeat for all 32 chapters
• Edit chapters
• Index chapters individually
• Engage artist (Lisa Haney) for new chapter cartoons, dividers and cover art
• Engage outside organizations (IBM, Sun, HP) for test equipment
• Regular (semi-weekly) meetings with authors, occasional meetings with publisher
• Read and revise page proofs, searching for any obvious errors or inconsistencies
• Deliver final manuscript to publisher and wait patiently

One of the biggest challenges in producing this edition was the distributed collaboration effort. We Skyped regularly to stay in sync. Evi was around for much of the development, but we also corresponded with her while she was sailing in the Caribbean and the Pacific. We used a subversion repository for the Adobe FrameMaker source files to avoid stomping on each other’s work. I’d say this was met with mixed success; Frame’s binary files are hard to merge, despite Garth’s valiant efforts at a scripted solution.

Special thanks to our named and unnamed contributors whose efforts are highly appreciated and certainly worthy of recognition. This is the best edition yet!

Goodie #1: Learn why version control is important for all businesses across the board.

Goodie #2: Get some assistance in deciding “Git or Subversion? Git or Subversion? Git…?”

Goodie #3 (otherwise known as the cherry on top): Meet Jim Turpin, one of our fabulous network engineers, who embodies the concept of multi-discipline to a T both inside and outside of the office.

Click here to read Q3 2010, and, as always, enjoy the treat!

We’d love to hear from you, so please post your comments and questions here.

Twitter is well known for great application-layer monitoring and instrumentation, so this gap in monitoring is a surprise. It exposes a common misconception among social software companies – that their server and network infrastructure is “covered” by their hosting provider.  As web applications scale to even 1/1000 the size of Twitter, software becomes critically interdependent on the underlying network. Infrastructure should be instrumented and monitored at least as closely as the software that depends on it.

For more The Barking Seal articles on monitoring and troubleshooting, see:

I should emphasize that this was our experience, and your mileage may vary. AppliedTrust is not a web design firm – we pride ourselves on infrastructure (servers, networks, security, performance, etc.). Still, I am tremendously impressed with Drupal and you are probably making a mistake if you are building a complex web site and haven’t considered it.

Looking back, since 2001 we have transitioned from static HTML (managed with GoLive), to Joomla, to WordPress (which we continue to use for this blog), to Drupal. Each transition has been a marked improvement, and today, I can’t imagine using anything except Drupal (for complex sites) or WordPress (for simple ones). In closing, here is a visual history of AppliedTrust’s web platform “evolution”:

1) Set goals: Both in exercise and in information security, it is good to set goals. For example, before I can write up a training plan for myself, I need to know what race I’m training for, and what my target pace is. Similarly, before I can write up my information security plan, I need to know what information I need to protect and how much protection I need (is this credit card data, or is it records of what color paint my store sold last year?)

2) Stick to the plan: Sometimes on Saturday morning at 6 a.m. I don’t really want to go for a run, but I know that the only way to reach my goal is to throw my legs over the side of the bed and stand up. I also know that the best infosec plan in the world does no good if it doesn’t get followed. It may not be fun to conduct that periodic audit again, and it may be frustrating to have to patch those darn servers in the middle of the night so you don’t impact the production systems, but you’ve got to do it. A plan in a drawer is no plan at all!

3) Make the plan doable: I could probably run my race a lot faster if I was willing to quit my job and train full time, but that’s just not practical. I need to keep perspective on the rest of my life and make the plan something that I can accomplish. The same is true for the security plan. It’s not reasonable to expect your team to install each and every patch within 24 hours of release. Save that extreme stuff for the really critical items that only come up once in a while. Be reasonable, and you’ll make everyone’s life easier. No one wants a security approach that flies in the face of usability. But there are some things that just can’t slip – and make sure you know what those are. Passwords, for example: I know it’s easier to remember five-letter passwords with no complexity requirements, but if you let that happen, you may as well forget the rest of the plan.

4) Celebrate your successes: I run with a group twice a week, and the coach keeps track of our times throughout the season(s). When I hit a personal record, she knows it and we celebrate the accomplishment. Do the same thing with security! Did your annual assessment just come back with 20% fewer recommendations? Did you just pass a penetration test with flying colors? Great! Celebrate! There’s always another mitigation recommendation you can implement, but don’t forget you’ve done many already, and congratulate yourself on a job well done.

The Deepwater Horizon rig was equipped with a vessel management system (VMS), which records dozens of different metrics about the conditions on the rig and in the well. These VMS logs would contain valuable details about the blowout, much like an airplane “black box” is essential in understanding a plane crash.

Steven Newman, the CEO of Transocean, said during a recent senate hearing, “There is some delay in the replication of our data, so our operational data, our sequence of events ends at 3 o’clock in the afternoon on the 20th. And so the VMS system, along with the logs of the VMS system, would have gone down with the vessel.”  The blowout and massive explosion happened at 10, taking eleven lives and seven hours of VMS data to the bottom of the ocean. Representative Bruce Braley from Iowa followed up with “So you have no mirrored backup data device so that that information is recorded at some other location than on the rig itself?”.  Newman replied, “We do not have real-time off-rig monitoring of what’s going on on the vessel”.

The costs to synchronize this data back to shore closer to “real-time” are nothing compared to the catastrophe at hand.  If an IT Disaster Recovery risk analysis had been performed, this replication delay would have stood out like a sore thumb.  We can be certain that new congressional regulations will be established to ensure that VMS data is replicated back to shore in a timely manner, but what about the rest of us?  Now is a perfect time to take a look at your business and make sure that critical data is being backed up appropriately to an off-site location.

Above, AppliedTrust’s  banner is displayed along with other “Laps for Learning” supporters at the corner of 76th and Baseline in Boulder.  Although the Douglass PTO had obtained advance permission to hang banners for the event along an Open Space-managed fence across the street from the school on 75th Avenue, it was later determined that such use for elementary school fund raising events constitutes prohibited “commercial use” of the open space lands and must be removed. Brady Van Matre volunteered to host the banners on his nearby fence as an alternative, and wrote a letter to the editor of The Daily Camera titled Lend Schools A Helping Hand.

We’re delighted to be an active supporter of our community, wherever our banner hangs!

As our regular readers know, AppliedTrust is looking for a great infrastructure engineer who wants to work in Boulder, Colorado. This role is a “Jack of all trades” within the broad field of Information Technology – they get to play with networks, servers, software, and security.  One ideal candidate for this job would be a graduating Computer Science or Engineering major who has experience with Windows and Linux system administration and doesn’t want to spend all day programming.  We would definitely also consider someone with more work experience.  If you are interested, or know of a good candidate, please check out our jobs page: http://www.appliedtrust.com/jobs

As discussed in detail by the Apache infrastructure team, a cross-site scripting vulnerability in Atlassian’s JIRA led to a full root account compromise on the ASF’s issue and request tracking server. If you don’t care to read the full story from the infrastructure team, the following sequence of events led to the compromise:

1. Attackers opened a new JIRA issue with a malicious tinyurl.com link that led to the JIRA page with an XSS vulnerability
2. Simultaneously, attackers launched a brute force attack on the JIRA login form
3. Several administrators clicked the tinyurl link, which compromised their cookies (giving the attackers JIRA admin access)
4. Attackers uploaded malicious a JAR file that collected JIRA passwords at login. One of the compromised passwords had also been used for a local account with full sudo privileges.

There’s more to the story, but those points capture the bulk of the attack.

This compromise interests me because it’s an explicit, targeted, successful attack against a security conscious and capable next-generation web technology team. Several techniques were used in this attack:

• Social engineering. The attackers opened an issue as if they were a trusted source posting a legitimate link. The Apache administrators trusted them.
• Web application security flaw. XSS is #2 on the OWASP top 10 list.
• Lack of vigilance. As the infrastructure team points out, the same password was used in a number of cases, and the JIRA user was overly privileged.

I hear a lot of grumbling when I highlight XSS vulnerabilities in a penetration testing report. “Is this really a serious problem?” and “we’re not a target” and “it doesn’t matter if they steal the cookie” are common complaints. Let’s face it – if the Apache team can be powned, we should all be wary.