BreakingPoint Labs Blog

TCP Portals: The Handshake's a Lie!

Tod Beardsley — Tue, 10 Nov 2009 16:00:00 PST

Whenever I interview someone for an Application Engineer or Security Research position, my favorite introductory question is, "Can you describe for me the TCP three-way handshake?". It is a fine baseline question to understand a candidate's knowledge of modern networking. Answers range from "SYN, SYN/ACK, ACK,", to a full description of ARP, to initial sequence number generation. It's a good springboard question, because then you can start talking about spoofing addresses, port scanning, the significance of IPIDs, and more.

We are hiring a lot here at BreakingPoint, which means I'm asking this question a lot. After the fourth or fifth interview, I decided one morning to look over RFC 793 to make sure that I really did know everything there is to know about the handshake. That is when I found out that we've all been living a lie.

If you've spent any reasonable amount of time around network protocols, you're probably familiar with some version of this diagram:

Here, we see the client on the left starting up a conversation with the server on the right. All pretty normal and familar, right? Well, when I was reviewing the RFC again, I noticed something very, very, odd. Disturbing, even. Allow me to quote at some length:

  The synchronization requires each side to send its own initial
  sequence number and to receive a confirmation of it in acknowledgment
  from the other side.  Each side must also receive the other side's
  initial sequence number and send a confirming acknowledgment.

    1) A --> B  SYN my sequence number is X
    2) A <-- B  ACK your sequence number is X
    3) A <-- B  SYN my sequence number is Y
    4) A --> B  ACK your sequence number is Y

  Because steps 2 and 3 can be combined in a single message this is
  called the three-way (or three message) handshake.

Do you see what I see? Because I'm thinking, "this is not a three-way handshake. This is a four-way handshake." The handshake is a lie, born of coalescing steps 2 and 3.

Now, surely, if I just decided to ACK a SYN, then send my own SYN, that couldn't possibly work, right? Enter PacketFu, my little Ruby library for crafting packets. Turns out, 28 years or so after this RFC was written, clients behave rather strangely when you decide to actually honor ol' RFC 793. After some experimentation, I have a pretty decent proof-of-concept stack that behaves like so:

This is the point where things get a little weird. What's happening here is:

    1) A --> B  SYN my sequence number is X
    2) That's nice. I'm not going to bother to ack that, because...
    3) A <-- B  SYN my sequence number is Y.
    4) A --> B  ACK your sequence number is Y, and my sequence number is X.
    5) A <-- B ACK your sequence number is X

Does this work? You betcha! Take a look at the packet captures, collected from Linux (stock Ubuntu), Apple (stock OSX), and Microsoft (stock Windows XP). These three desktop operating systems are all totally cool with this crazy backwards TCP portal.

But what does it mean? Is this simply a parlor trick, where you can reverse the roles of client and server? How does this affect stateful firewalls? How about inspection devices like IPSes, which often need to have an idea of who the "real" client and server are? How about NAT devices, where the idea of "relatedness" is absolutely tied up with where SYN packets come from.

Clearly, there is a ton of testing work to be done here. Lucky for me, I happen to work at a really advanced testing equipment manufacturer, so I've dropped this nugget in the next StrikePack. Now, strikes can employ the "SneakAckHandshake" TCP override option, and all servers simulated will behave in accordance with this crazy backwards handshake. We'll see how well network inspection gear detects clientside attacks when the client is tricked into behaving like a server.

At the very least, now I have better interview questions and I should at least be able to detect if the next candidate is reading this blog. :)

60 Minutes on Cyber War: The True Threat of Cyber Attack

Kyle Flaherty — Tue, 10 Nov 2009 05:52:22 PST

Ten years ago, 60 Minutes reported on the new threat of "cyber war." At the time, the story introduced the American public to a developing danger that would come to realization in the future. According to Sunday's program, that day has arrived. The television show dived back into the topic of cyber warfare and it was evident that not only has the threat arrived, but the United States is not prepared to face the attacks.

Retired Admiral Mike McConnell, the former chief of national intelligence and director of the National Security Agency, outlined the gravest threat to the U.S.'s critical infrastructure: our power grid.

"If I were an attacker and I wanted to do strategic damage to the United States I would take the cold of winter or the heat of summer and sack electric power on the East Coast...and hope for a cascading effect. All of this is in the art of the possible for a sophisticated attacker and the United States is not prepared for such an attack."

A few months ago President Barack Obama admitted that the U.S. electrical grid had already been probed by cyber intruders and acknowledged that another country had seen entire cities plunging into darkness due to cyber attacks. The president failed to mention which country had seen its electric grid shut down due to cyber attacks, but 60 Minutes reported that Brazil had experienced the attacks in 2005 and 2007. The actual perpetrators are unkown, but 60 Minutes acknowledged that there are now highly trained cyber warriors throughout the world poised to lead such attacks. (UDPATE: Raphael Mandarino Jr., director of Brazil's Homeland Security Information and Communication Directorate, told the newspaper Folha de S. Paulo that he’s investigated the claims and found no evidence of hacker attacks.)

Electronic Pearl Harbor

Awareness of the threat of cyber war started to take hold in 2007 according to Jim Lewis, a director at the Center for Strategic and International Studies. It was at that point the United States witnessed what Lewis called an "electronic Pearl Harbor," when unknown foreign entities conducted online espionage and broke into the Department of Defense, Department of State, Department of Energy, NASA and others, walking away with terabytes of information. Lewis also acknowleged that the intrusion into CENTCOM last December was the second major "wake-up call" for the government, as foreign entities penetrated the highly secure military system and remained inside the digital infrastructure for several days "listening" to all traffic and activities.

The full segment is embedded below and I encourage everyone to watch. Overall, the piece reiterates much of the information we have blogged about several times, but as I watched the piece I was reminded of comments made by BreakingPoint's Dennis Cox during the recent "Preparing for DDoS and Botnet Attacks" webcast:

Dennis warned about the threat to the electrical grid as a whole but expanded that threat to individual households as we see the continuing adoption of "eHome" technology such as online energy monitoring. This all being part of the cascading effect mentioned by Admiral McConnell.

Also during the DDoS/botnet webcast, I asked our experts why more wasn't being done to stop these attacks by the government or crime-fighting organizations. One of the answers was simple yet powerful. The perception remains that nobody is truly being hurt by these attacks, thus there is never a groundswell to take action. Jim Lewis called it "death by a thousand cuts" during the 60 Minutes segment. Unless we actually see the damage inflicted by cyber war we often will stand idly by and let the threat remain.

Reality Must Defeat Perception

In 2009 more than $100M was stolen from U.S. banks according to Sean Henry, assistant director in charge of the FBI Cyber Crime Division. Yet how many of those bank robberies have you heard of during this year? Henry continued:

"I've seen attacks where $10 million has been lost in a 24-hour period. If that had happened in a bank robbery where people had walked in with guns blazing it would have been headline news all over the world."

And this is where the problem lies today. We are faced with an imminent threat of cyber war and cyber crime, yet the perception remains that nothing is seriously wrong. Today however, due to the 60 Minutes piece, people are seemingly waking up to the threat and the serious damage cyber warfare and cyber crime have already inflicted. Will last night's show prove to be the tipping point for solidifying the cyber war battle lines or forgotten again as the news cycle churn us towards another topic?

As someone who is involved in this topic every day, I'm certainly rooting for the former and I think the piece will actually help in five distinct ways:

Put pressure on the Obama administration to begin taking action on securing our critical infrastructure including solidifying the cyber coordinator position.
Educate Congress on this threat so that more members become involved with pending and new legislation.
Create support for the emergence and success of such programs as USCYBERCOM and National Cyber Range.
Force everyday Americans to realize that the threat of cyber war is real and that we are all responsible in some way for defending against this invisible enemy.
Place some pressure on service providers and network equipment manufacturers to test equipment and services under true cyber war replication.

As promised, here is the full 60 Minutes segment:

DDoS and Botnet Test Methodology Released

Kyle Flaherty — Thu, 05 Nov 2009 14:05:49 PST

Distributed denial of service (DDoS) attacks have become an enormous risk, shutting down businesses, halting bank transactions and disrupting government communications. Yesterday, BreakingPoint hosted a webcast diving into many of the issues surrounding DDoS and botnet attacks and what people can do to prepare. The main lesson? Test, test and test! I've included the webcast below, slides and audio, the discussion was highly informative, so take a look.

Additionally, we have just published the BreakingPoint DDoS and Botnet Test Methodology, which details, step-by-step, how to replicate a variety of DDoS and botnet attacks to help you find weaknesses before others do. We encourage you to download the test methodology today and certainly let us know your thoughts.

Preparing For DDoS And Botnet Attacks Webcast

Live Video Webcast: Preparing for DDoS and Botnet Attacks

Kyle Flaherty — Tue, 03 Nov 2009 16:00:00 PST

UPDATE: Today's WebCast has concluded, but you can watch it in its entriety by clicking on the viewer below. You can also download the BreakingPoint DDoS and Botnet Test Methodology.

Distributed denial of service (DDoS) and botnet attacks are an imminent threat to your network and the only way to test network equipment and application servers is by hitting them with actual DDoS and botnet attacks. Join BreakingPoint security experts right here at 2pm CT to learn the best ways of replicating DDoS and botnet attacks to find vulnerabilities before someone else does. This discussion will include BreakingPoint Chief Technology Officer, Dennis Cox and BreakingPoint Labs security researchers Tod Beardsley and Dustin Trammell.

At 2pm CT today you will be able to watch the webcast in its entirety below or on our USTREAM Channel. If you would also like to recieve the "BreakingPoint DDoS and Botnet Test Methodology" after the webcast be sure to register here.

MILCOM Cyber Security Takeaways: Lessons from the Physical Battlefield

Pam O'Neal — Mon, 02 Nov 2009 11:07:04 PST

If the recent barrage of government security events is any indication, civilian and military personnel are energized about hardening cyber security. The largest event in recent months was MILCOM 2009, where I had three days to talk cyber simulation and cyber warfare with some of the best and brightest. The experience was both educational and a bit surreal, as BreakingPoint showcased resiliency testing products on an exhibit floor where networking products were intermingled closely (almost perilously, in my case) with military hardware like satellite receivers and armored vehicles.

These products once formed a strange combination, but now their mingling illustrates how the lines are blurring between the physical battleground and cyber warfare. Experts from academia and government have warned for years that new battles will be fought online, where traditionally weaker parties have a level playing field with the militaries of larger nations. Just this month the United Nations Telecommunications Agency chief warned that “the next world war could take place in cyberspace.”

Each day, thousands of individuals attack networks around the world for reasons ranging from personal amusement to organized cyber crime. As this graph taken from Infonetics Research data shows, cyber intruders and enemies of the state are becoming more sophisticated and aggressive in network attacks.

While government-operated networks are the targets of more than a million attacks each week, privately owned infrastructures are also increasingly vulnerable to attack. And more than data and communications are at risk. Here in the U.S., much of our critical infrastructure, such as energy, transportation, and financial networks, is private. A single attack taking out one of these networks could threaten the U.S. market and personal safety.

Both the public and private sectors are investing heavily in experimental and proven techniques for hardening the networks and data centers that form our critical information infrastructure. They are also scrambling to recruit and train security researchers in an effort to launch a proactive defense. This is particularly difficult in the world of information or digital warfare, where the terrain is complex and virtually invisible, conditions are ever changing, and attackers are widely distributed. To fight on this virtual battlefield, the U.S. is gearing up to hire some 4,000 specialists, and they are going to need actual hands-on experience to identify and block attacks that are ever morphing.

For the U.S. military, training was—and still is—one of the greatest challenges facing leaders in achieving the goal of Information Superiority, defined as “the capability to collect, process, analyze and disseminate information while denying an adversary’s ability to do the same.” It strikes me that there are a few lessons from the physical battlefront we can apply to the virtual world of cyber warfare, such as the use of simulation to prepare soldiers for battle.

Unlike training for the battlefield by using war games and battle simulations, our cyber warriors will conduct their missions online, and these cyber warriors must have Internet-scale cyber simulation capabilities to replicate commercial, military, and government network conditions. When it comes to cyber war, network mayhem is the enemy. Cyber warriors must know how to navigate the mayhem and spot the most deadly attacks in some of the most complex and confusing terrain imaginable—an ever-morphing online world filled with invisible enemies.

With an increasingly sophisticated army of guerrilla hackers operating in a highly distributed cyber battlefront, the ability to detect an attack and respond to prevent damage instantly is paramount. The enemy who strikes first can take out an entire network through coordinated distributed denial of service (DDoS) attacks. Before most products or experts even know anything is happening, the attack is over and critical systems are compromised.

Intelligence and the ability to spot and block attacks instantly are paramount to protect against devastating DDoS assaults. However, network-traffic-generation products or emulators cannot generate the evolving application and attack traffic or simulate the user load necessary to replicate realistic attacks and prepare our defenses to block them. In fact, from the reactions we received at MILCOM, it is evident that cyber simulation of the scale and sophistication created by BreakingPoint products has never before been possible. While that is exciting to hear, it is also frightening.

The U.S. government’s answer is to invest billions in building the U.S. Defense Advanced Research Projects Agency’s (DARPA) National Cyber Range (NCR). It’s a great idea and an ambitious undertaking, but it will take years to fully realize. Meanwhile, our military and intelligence communities do not have the tools they need now to properly prepare for the dangers of cyber warfare, and that is simply not acceptable when the technology exists in such a small form factor.

It is encouraging to speak with so many in the military who “get it” and are actively looking for solutions to train cyber soldiers to defend the network. With any hope, we can take yet another lesson learned from the physical battlefront and arm our soldiers with the tools they need to keep our infrastructure safe from growing cyber threats.

Preparing for DDoS and Botnet Attacks: Webcast and Test Methodology

Kyle Flaherty — Thu, 29 Oct 2009 06:57:31 PDT

DDoS and botnet attacks are an imminent threat to every network. Just look at the damage we saw over the summer. In July, cyber attacks targeted a number of government, news media, and financial websites in both South Korea and the United States. Shortly after, we witnessed DDoS attacks, aided by botnets, bringing down social networking services such as Twitter, Facebook and Google. In the face of an imminent threat how do you prepare?

If I can channel my inner Sun Tzu for a second; it is critical to know your enemy in order to protect your network assets. This means testing your network elements and application servers with genuine DDoS and botnet attacks to determine your weak points, before someone else does.

Next Wednesday, three BreakingPoint experts are getting together to discuss how to prepare for DDoS and botnets through replicating these attacks during testing. Dennis Cox, the CTO of BreakingPoint, will be joined by Tod Beardsley and Dustin Trammell for what plans to be a lively conversation. The webcast will also highlight some of the information in the soon to be published "BreakingPoint DDoS and Botnet Test Methodology". The methodology will be sent to all registrants of the webcast and details, step-by-step, how to replicate these scenarios in your testing environment.

Demonstrating Cyber Range Simulation During MILCOM

Kyle Flaherty — Sun, 18 Oct 2009 17:00:00 PDT

This week we have a team in Boston at MILCOM 2009, demonstrating cyber range simulation . MILCOM, for those not familiar, is a "conference for military communications, and attracts the best and brightest with high-level attendance from government, military, industry and academia. MILCOM 2009 gives industry the opportunity to promote communications technologies and services to commanders from all branches of the armed forces, Department of Defense, federal government, and the heads of multi-national forces from around the globe."

BreakingPoint will be demonstrating how our products deliver cyber simulation on a scale never before possible without building and maintaining an enormous environment similar to the vision for the U.S. Defense Advanced Research Projects Agency’s National Cyber Range. During the show we will be showing off the three-slot BreakingPoint chassis in booth number 453. Show attendees will be able to see how BreakingPoint replicates massive Internet-scale network traffic in order to find and address critical performance issues and security vulnerabilities, all in order to assure network resiliency.

If you are attending MILCOM let us know or stop by the booth. I will also be blogging a bit from the show during the week for those of you who are not attending.

Realistic Testing of Server Load Balancers

Kyle Flaherty — Tue, 13 Oct 2009 13:14:06 PDT

Realistic server load balancer testing is now critical, as load balancers have become a key network device. Lack of proper testing has unfortunately resulted in serious performance and security issues. One primary example would be the recent problems for Google's Gmail. Testing load balancers means hitting them with realistic and actual network scenarios, rather than the "ideal scenario", in order to assure their resiliency. BreakingPoint has published an updated Server Load Balancer Test Methodology and I took this opportunity to sit down with BreakingPoint CTO Dennis Cox to talk server load balancer testing.

You can download the BreakingPoint Server Load Balancer Test Methodology, which includes multiple test scenarios including:

Testing the number of TCP connections per second the load balancer is able to handle to provide a baseline test of the device’s performance capabilities.
Determining the overall bandwidth the load balancer can support through testing the number of HTTP/HTTPs connections per second the device can handle.
Emulating blended application traffic to validate the load balancer can handle a true network scenario.
Simulating dynamic pages and image files to validate HTTP Caching performance.
Confirming the load balancer can handle malformed packets or errors with the packet through application fuzzing.
And more...

Open Letter to Twitter: Can We Help?

Kyle Flaherty — Fri, 09 Oct 2009 09:44:09 PDT

Dear Twitter,

In December 2006 I jumped onto Twitter, not realizing the impact it would have on my life. Twitter helped introduce me to Pam O’Neal (@poneal) at BreakingPoint (@breakingpoint) and subsequently move my family from Boston to Austin to work here. Now, each day at BreakingPoint, we use Twitter (30%+ of our employees are registered) to communicate with our community of server load testing professionals and network and security engineers, as well as communicate amongst one another.

As a company, BreakingPoint recognized the potential of Twitter early and in 2008, even added a product feature to test the ability of network devices' and application servers' to handle the load of Twitter traffic. Our team has worked for decades developing the latest in networking and data center technology and they know first-hand how hard it can be to sustain a reliable network and service. It is why they created a better network and load testing solution.

Yesterday, when Twitter was down due to a "bug triggered by an edge case in one of the core services", I thought about how important Twitter had become to our business and me. I watched the predictable posts complaining about the fail whale and it hit me; rather than throwing criticism, I would be best served getting my hands dirty and helping with the problem. An idea surfaced, which I talked through with our CTO and co-founder Dennis Cox (@denniscox), and the green flag was waved.

BreakingPoint wants to help Twitter by providing the use of its server load testing product and wicked smart folks (sorry, the Boston still in me) to help assure the resiliency of your company's network devices, servers and overall data center infrastructure. We want to provide the use of our BreakingPoint Elite, combined with a team of experts, many of them industry-recognized resources in security, Ruby, software engineering and load testing. As your company will see, we can provide server load testing on the scale a service like Twitter would need. I can have someone from our San Francisco office meet with your team today if needed (and I can be on the next flight).

BreakingPoint is a huge fan of Twitter and hopes that it can help! I've included the Twitter profiles for some of our team below. We are all looking forward to working with your company.

		denniscox - Dennis Cox CTO and founder of @breakingpoint			tmanning - Todd Manning Author of @breakingpoint Twitter testing protocol
		todb - Tod Beardsley @breakingpoint "protocol monkey" (his words, not mine)			poneal - Pam Oneal Vice President of Marketing @breakingpoint

		mikehamilton - Michael Hamilton Product Marketing @breakingpoint			kkuehl -Kirby Kuehl Writes protocol (de\|en)coders in C/C++ @breakingpoint
		jgstroud - Jonathan Stroud Hardware design @breakingpoint			druidian - I)ruid @breakingpoint Labs
		chrisfenton - Chris Fenton @breakingpoint--West Region			KyleFlaherty - Kyle Flaherty Communications for @breakingpoint

Many thanks for everything your company has provided our community,

@kyleflaherty

UPDATE: I'd be remiss if I didn't also link to @busterbcook and @rhythmx, two more great BreakingPoint Tweeters!

Cisco Security Agent Exits the Market?

Dennis Cox — Mon, 05 Oct 2009 12:29:14 PDT

This morning, Ron Gula tweeted a link regarding the possible discontinuation of Cisco Security Agent (CSA). Gula, the CEO/CTO of Tenable Security, pondered whether this was the first of many Cisco security products to be discontinued. While I think he may be right in that regard, I was hoping CSA remained alive. Patrick Ogenstad wrote the actual blog post in Network Lore to which Gula referred in his tweet. It's a wonderful article and I agree with most of what Ogenstad writes, with the exception of a sentence in the last paragraph:

"Perhaps Cisco is the wrong vendor to have this specific product in its portfolio, and perhaps someone else will buy it."

While I do hope someone picks up the product, I actually think Cisco is the best company to own CSA. This was its trojan horse into the desktop to disseminate a whole host of other products and services. Perhaps even TelePresence?

Cisco has the manpower, a technical sales force and a strong technical support organization. Those are key factors, in my opinion, to make CSA successful. CSA reminds me a lot of Network Flight Recorder (NFR), acquired by Check Point in 2006. The products are (were?) both extremely powerful. You could do most anything you wanted and neither product required constant upgrading. The general feedback on both, however, was that they were complicated and required knowledgeable people to set them up and get the most out of them. I really dislike that as an argument for their demise: "I'm too lazy to read the manual and do a few Bing searches, Mr. Vendor. Just make it all auto-magically happen for me.".

Sorry, buddy, but networking doesn't work that way and network security definitely doesn't work that way. It's a detail-oriented profession and if you are not detailed enough to understand the difference between UDP and TCP, get out of networking. You are not doing anybody any favors by judging everything on presets and defaults. You sir, are the type of person being mocked in beer commercials.

We see this all the time in testing. Vendor A has built a new content-aware firewall, and its QA team tests the product using a bit blaster to see how many UDP packets can go through at any given packet size and any second. When does that happen on a real network? Never. The QA team is doing what they did in the past and is now simply being lazy. They are not helping the product succeed in the real world. Here is a suggestion to anyone with a content-aware firewall, test with some actual content, and you'll be surprised by the results.

As I noted in my last blog post, network administrators are once again failing to secure their networks properly, whether it's failing to update their routers and switches with the latest Cisco patches or not deploying solid security solutions such as CSA. It leads me to a couple of important questions for the peanut gallery: Why is CSA leaving the market (or is it)? And what could Cisco have done to save it?

Oh, one last thing, what are you doing to save your product? God knows I hope you're testing.