The 2011 World IPv6 Day event was an opportunity for website and networks operators to trial IPv6 functionality for a day and see what worked and what needed to be understood better. It was an opportunity to face the fear, uncertainty and doubt which has plagued and haunted IPv6 almost since the IP version’s introduction in 1994.

For the most part, the 2011 World IPv6 Day event went very smoothly. A primary challenge was for organizations to make their primary web sites accessible via both IPv6 and IPv4. The Internet Society (organizer of World IPv6 Day) also asked network operators to inform their customers about issues they could encounter on World IPv6 Day. Some organizational web sites also attempted to identify what they thought were “broken stacks” and warned users with these potentially broken stacks that they may have problems accessing the organization’s services during World IPv6 Day. Some World IPv6 Day participants ran “war rooms,” where they monitored their networks and customer call centers for problems related to IPv6. For the most parts these war rooms were quiet.

One of the surprises of World IPv6 Day was that even though there were a large number of web sites participating in World IPv6 Day, there wasn’t a proportionate jump in the quantity of IPv6 traffic observed on the Internet. In hindsight, this shouldn’t have been a surprise. On June 8, 2011, the number of IPv6-enabled web sites may have increased considerably, but the quantity of users who could access the IPv6-enabled web sites via IPv6 had not increased.

This is perhaps one of the motivations behind why network operators are being challenged in World IPv6 Launch to configure their networks such that 1% of the web accesses from their customers are via IPv6 transport. Without users with the ability to reach IPv6 enabled web sites, IPv6 enabled web sites will not be able to acknowledge that they can support any significant volume of IPv6-enabled users.

Leslie Daigle, Chief Internet Technology Officer at the Internet Society, confirmed that this was a motivation. Said Daigle, “Indeed, it’s been clear for some time that for real IPv6 deployment (which is what is happening with the World IPv6 Launch), there needs to be content for people to access over IPv6, and people that can access content on IPv6-enabled websites. So, we had both content providers and operators in the room when discussing this year’s challenge. 1% is enough traffic to demonstrate that access providers are well advanced in their actual deployment plans. With that, and the fact that content providers are turning on IPv6 and leaving it on for this year’s challenge, we have the basis for our statement that: this time it’s for real; after June 6 2012, IPv6 is the new normal for Internetworking.”

It is easy to underestimate the complexity of ensuring that a web site can be reached via IPv6, that an network’s customers can reach IPv6-enabled Internet destinations or that a home router is able to interoperate with broadband providers and end-user operating systems to allow the end-user to reach an IPv6-enable web site.

To help illustrate this point, for an organization’s web site to be able to serve up IPv6 web pages, many pieces must be in place and working correctly:
• If the organization’s web site utilizes load balancers, the load balancer(s) must support IPv6.
• The organization’s DNS infrastructure must correctly support IPv6 (AAAA) record types.
• The organization must peer with other networks via IPv6.
• Security infrastructure must support IPv6 as well as it does IPv4 traffic and the infrastructure must be aware of uniquely IPv6 challenges. This includes knowing how to handle “packet-too-big” ICMPv6 messages and extension headers. As Arbor Networks’ 7th annual Worldwide Infrastructure Report has illustrated, IPv6 has begun the transformation from novelty to an intrastructure of value which must be protected from attacks.
• Routers must provide the functionality that the organization requires to transit IPv6 traffic, and these routers must be able to handle IPv6 traffic at acceptable rates.
• Web analytics applications must support IPv6.
• Web applications should support IPv6, as should links off the IPv6-enabled web site. There were multiple examples of web sites registered as World IPv6 Day participants, for which the primary web page was IPv6-enabled, but all links and services off the primary web page led to IPv4-only pages and services.
• The “IP address geolocation” capabilities must be acceptable for IPv6 (and in fact, this was one of the challenges with IPv6 cited after World IPv6 Day). Many web services rely extensively on reliable IP address geolocation data.
• The organization’s primary web site may be hosted by an outside provider which doesn’t support IPv6. This is slowly changing; one of the bigger announcements of IPv6 support recently has been by Akamai At least one university participant in World IPv6 Launch was able to announce an IPv6 address for the campus’ primary web site because of Akamai’s newly announced capabilities.

For the network operator challenge to have 1% of its users access web sites via IPv6, similar complexity exists, while additionally making fulfillment of the challenge based at least partially on the inherent IPv6 capabilities of the end-users’ home network or computer operating system. For a large network of perhaps 10 million customers, not only would the IPv6 connectivity need to be in place in the ISP’s network from the users’ home network to the global Internet, 100,000 users need to have computer operating systems supporting IPv6. As a presentation by Lee Howard of Time Warner Cable pointed out in a mathematically interesting presentation at the US IPv6 Summit in Denver this year, there are several overlapping aspects of their network which must support IPv6 in the right manner before Time Warner Cable’s network will be recognized for having 1% of their traffic being via IPv6 transport.

World IPv6 Launch will be upon us in a very short time. While it has the immediacy of World IPv6 Day, its planners hope IPv6 will become business as usual. Future Internet “challenges” will occur regardless of whether the majority of the world stays with IPv4 or the world migrates to IPv6. Sure, there will be unforeseen problems in implementing IPv6. However, IPv6 is the only sustainable long-term direction for the Internet and Internet Protocol. The time spent solving IPv6 challenges will be more strategically valuable than time spent patching IPv4.

In the past few years, the popular kit we saw in our zoo was Black Energy, an easy to use, powerful DDoS bot distributed in Russian-language spheres. You could find it on the net for anywhere from $40 to free, depending on how hard you looked. Starting in 2009, Black Energy version 2 was available. It boasted a modular architecture and included banking theft Trojan functionality to augment the DDoS functionality. Another competitor that appeared in this timeframe was Optima or Darkness. It then becomes interesting to look in one’s zoo to see which families are popular at present.

The graphic below is from some quick analysis of our zoo, looking at our automated classification system that tags samples with the appropriate family name. Optima includes the group of Darkness, Optima and Votwup samples we have seen; Black Energy focuses on the BEv1 samples we have seen in the wild, and omits any BEv2 samples we have; Dirt Jumper includes all Russkill variants and Dirt Jumper variants we have seen. What we saw was a bit surprising, namely the rate at which Dirt Jumper samples appear in our zoo and quickly overtake the other families in terms of popularity. This matches anecdotally what we had seen, but the magnitude itself is surprising.

Some ideas as to what is going on:

• With BEv2, the Black Energy author (back in 2009 which it was being developed and tested) appears to have tried to piggy back on the Zeus and SpyEye craze that was really gathering momentum at the time. Modules to steal from banks would have been a great complement, in theory, but in reality BE targeted DDoS actors who hang out in different forums than the financial thieves. With the notable exception of the Gameover series of attacks, these two groups don’t spent a lot of time together from my own observations.
• Optima and Darkness make a decent product. I didn’t keep track of pricing or advertising, but their usability, reliability and features all come together to make a great follow-on to the Black Energy model (kit which includes an easy to use web UI and a builder to configure the feature-rich DDoS bot). Why it didn’t take off is really something I can’t explain.
• Finally, Dirt Jumper’s meteoric rise in popularity in this time frame suggests that author (and any promotors they have working for them) is doing something right. Features are getting incorporated well, new versions are released, and the bot’s got traction in the community. An alternative explanation is that the leaks we see leading to “unofficial versions” are also classified as DJ and explain the rise.

In this competitive underground world, it’s fascinating to see market forces at work so clearly. Bear in mind that all this popularity leads to attention, both in terms of CnC tracking (and shutdown) and AV detection, which is counter-productive. We’ll see how these guys react to larger responses.

It’s like trying to plan your life by picking a college major as a high school student. You think you’re picking a destination; however, you’re just picking a trailhead that’s marked with a destination you think you’d like to get to. You are in fact picking a path through unknown territory to a place you’ve only read about.

To make this more concrete with an example, I remember sitting down with Farnam at a coffee shop and interviewing our first VP of Sales – Arbor’s first VP hire. Not only did we not know what makes a good VP of Sales, we didn’t really even know what a VP of sales did day-to-day. Should they run Marketing? What is Marketing? Advertising? Hell if we knew… did we like this guy? We did – and that turned out to be the actual right question for that hire.

Well, suffice to say, it’s been a long time since 2000. There have been a lot of executive positions that have been filled and refilled since then – I’ll leave it as an exercise to the reader to count them via google and the way-back machine.

However, it turns out that building teams is only part of a founder’s job. Over the course of the last twelve years I’ve constantly shifted my focus as Arbor matured: I used to love to code – still do – I wrote the first version of the Peakflow system; I turned into a salesman – I was the pre-sales engineer that sold all of our early customer wins (paired with our VP Sales); I opened up EMEA sales, hired our early team there and sold the first customers; I stepped in as VP Product Management for three years – most thankless job in the company btw; I led our strategic entry into the Enterprise market with countless trips to NYC and DC; I was tasked with making APAC a meaningful contributor to revenue; I was tasked with M&A – find something to buy – pretty hard problem for VC-backed stock-only private-to-private currency; and I was tasked with M&A again – find someone to buy us; serve as a board member – until the acquisition; and most recently, come up with a next-gen fixed-line and mobile strategy. I’m not trying to write a resume here, but just to point out that wow… that’s a lot of stuff that was certainly not disclosed on the sign at the beginning of the trailhead that said, “Start a Tech Company and be its Chief Technology Officer!” You can see why I always groan on the inside when someone asks me what a CTO does.

The most unexpected part of starting the company, that was both humbling and weighed the most on me though, was the responsibility to the employees and customers that I felt. As a researcher, I was used to doing all the lifting on my own. I could be as risky and crazy with my ideas and projects as I wanted – the way to innovation. However, you quickly learn in a startup, that you need to share the load in order to carry anything meaningful. The problem is, once you’ve convinced friends (and friends to be) to relocate their families and join you on your crazy trip, the joint weight of that responsibility (kids’ shoes, medical care, food, mortgage payments) comes to bear on you.

We started raising our second round of funding Sept 1, 2001 – not good timing. We came to really appreciate our capital for the oxygen that it is. We closed it in February of 2002, the last dollar we took in fundraising. Needless to say, that a day didn’t go by when I didn’t worry about all of the families that depended on our decisions. Don’t screw it up, Rob!

Another unanticipated aspect of founding the company was the travel. For the past twelve years my home has been Northwest Airlines (now Delta). I spent more days on the road than at home. I’ve been {platinum, diamond} at {Northwest, Delta} since 2000 – at least Detroit’s (DTW’s) only one-hop from anywhere on the planet. I’m past the million-mile mark and closing in on two. I’ve used up all of my visa pages (and extensions) in two passports. As a kid, the idea of traveling to the South Seas was intoxicating. Last month I finally got there – spending a week in APAC attending Arbor’s customer summit in Bali. That trip along with some follow-on meetings with customers in Singapore gives a good example of my flight itineraries: DTW->AMS->SIN->DPS->SIN->NRT->DTW. I spent 48 hours travel time for about 72 combined hours in Indonesia and Singapore – multiply this by twelve years to approximate my travel. It was sitting on the beach in Bali under a tree watching the surf – and wishing I were only home – that it finally hit me. Time to take a break.

There are reasons why it won’t be so bad on Arbor for me to leave now. Arbor’s been on a hiring tear these last twelve months (and it’s still going strong). We’ve filled out the management team with great players, and have been building onto a fantastic engineering team as fast as we can hire in all three development centers: Ann Arbor, Boston and Atlanta. We’ve hired new product managers for the enterprise product lines that are full of energy and come from battle tested security companies. We just hired a new leader for our corporate development team and another for our security research team. I won’t spoil any of the surprises, but there are lots of great new products and features coming down the roadmap that are just what the customers need, want, and in some cases, don’t even know they need or want yet. The sales teams continue to do what they do best: overachieve in every region.

I remember meeting with a VC back in October or November of 2001 on Sand Hill Road with Farnam. Due to the terrible funding climate post Sept-11 attacks, we had scaled back our sights to just trying to raise a mezzanine round. He sneered at us, told us that denial-of-service was a fad that had passed, and that we’d be out of business within six months. It turns out that his firm is the one that no longer exists. Denial-of-service attacks – unfortunately for society – were not a fad. They continue to escalate in sophistication and magnitude. Arbor’s core market is only growing. Arbor’s acquisition by Danaher has gone smoothly. Two years into the merger, life is pretty much the same for most of the company. The promise of the acquisition has been playing out: stand alone Arbor backed with significant financial firepower. This combination of company and market stability allows me to be able to pick now as an opportune time to step off – without any worries.

We’re a long way from the winter of 2000, when our small band of crazies broke onto our building’s rooftop to raise the pirate flag; or the winter of 2001 when we drilled a hole through the roof to get a live Olympics feed for our “SOC”. These days people bang on the glass and feed the hackers. There are so many people that have made Arbor great, that I can’t call them all out here. You know who you are: Employees (what a horrible way to say friends), Customers, Partners, even Competitors – you’re all there! Thank you all, from the heart.

Your friend always,
-Rob

PS. If you get confused, just listen to the music play.

What do we mean when we talk about measuring botnet populations? We are trying to measure the number of infected devices to figure out how many people are affected, the number of accounts or customers, and the like. Because of the way the Internet is structured, we can only measure the number of infected PCs or IP addresses received in a time period. We then have to use this information to estimate how large the botnet infected population is.

We count botnet populations for several reasons. First, we want prevalence measurements in order to understand which threats to focus our limited efforts on. We want to understand the prevalence of a botnet by geographic region, for example, to understand to whom we need to reach out. We also want to understand how we should prioritize our efforts, focusing on botnets that will yield a significant impact if they are addressed. Finally, we want to understand the scale of the resources we need to gather as we tackle the botnet. Continuous measurement is vital in order to understand what mechanisms are effective at reducing the botnet’s population. Also, if the numbers ever drop to zero, we can call it a victory. Finally, we also want to understand the size of the possible attacks and any expected financial impact, in order to prepare defenses.

Measurement Methodologies

Counting methodologies are broken into several different methodologies. Measurements using sinkholes are the most popular mechanism right now to count. In this method, we take the botnet command control server and redirected either by DNS or IP redirection to servers that the good guys operate so it’s now outside of the botnet operator’s hands. Then we’re able to count the number of unique IP addresses connecting every day to this server, and we know that these belong to a particular botnet. We can also fingerprint the traffic coming in and are able to distinguish one botnet from another, giving us a prevalence count.

Sinkholes are the most common mechanism right now to count botnets, and are widely done by many groups. All we have the number of IP address is a connect to us here, but sometimes there is a piece of information and the communications from the botnet to the server that we can use to uniquely identify the client and identify when there are more than one PCs source IP address. This might include for example the MAC address for from the botnet, the hostname from the PC, or in the case of the recent Flashback malware the UDID from the device itself. This can help give us some better numbers about the population size.

Shown above is the number of Conficker infected systems that we counted over a two week period. This was gathered using the “q” value from each individual communication and then summed per source IP every day, yielding a decent estimate of the size of the botnet. In this period we estimated the botnet grew from 200,000 zombies to much more than 700,000 zombies.

Another method for counting botnets and estimating their size we call dark IP monitoring. This method takes large unused IP address blocks and then listens for traffic. The collection system is able to fingerprint bots based on specific signs. This could include the exploit traffic or traffic to a specific TCP/IP service used. This then gives you some passive mechanism to watch the botnet and try to spread. Arbor used this method to measure the size of the 2003 Blaster worm, watching a /8 network and counting worm sources.

This graphic is from of paper that we wrote called The Blaster Worm: Then and Now covering the Blaster worm’s propagation over time. Shown here are the various stages of the worm’s specific traffic from our dark IP monitors, showing the worm’s initial burst onto the Internet, followed by the decay phase as networks shut down those hosts and the TCP/IP services the worm used to propagate. The final phase in the graphic shows the diurnal rise and fall of the worm’s populations as PCs are turned on and off each day. The counts are the number of unique source IP addresses every hour.

A direct method for measuring botnets is actually counting on infected hosts. Microsoft has the best option here because they’re able to count reports from their Windows antivirus software, the MSRT executable pushed down during Windows Update, and other host-based antivirus solutions. Distributing this tool globally has enabled them to measure how many infected PCs hit each individual signature. While this is the most direct measurement possible, this is not accessible to many people outside of Microsoft.

Another direct methodology is to crawl a peer-to-peer botnet, gathering the peer list from every node and recursively walking the botnet. This enumeration of the botnet is possible if you know the P2P protocol, but is easily thwarted by strong cryptography. Kaspersky Labs has used this to track the Storm worm, the Miner.h botnet and others. Shown below is a graphic from a Kaspersky Labs blog post on the Miner.h P2P botnet, showing how the nodes are connected.

Limitations

Clearly with botnet measurements you have possible visibility issues. If, for example, ISPs are blocking ports or are blocking collection addresses and instead directing clients to go to their own sinkholes on their own servers, identifying customers, this will lead to under-counting. Similarly, if the domain names for the botnet, which now point to sinkholes, are used in DNS blacklists, clients will never be recorded at the sinkhole, again leading to undercounting. Also, if hosts are offline – not connected or just powered off – they wont be counted. Finally, if the bot’s self-reporting mechanism is to be trusted to count the botnet population, you are possibly the victim of inaccurate reporting by the bot, either being actively deceived or through errors in the bot’s counters. All of these can lead to inaccurate values.

Complications

There are also problems in estimating populations from the source IP counts we gather. DHCP, for example, can lead to over counting. We know that one IP address does not mean equal one device, as DHCP churn can lead to the same device getting multiple IP addresses in a given day. NAT is another issue that can lead to reductions in the numbers. We see ratios about 10 to even 100 to 1 in the wild, meaning we believe that 100 PCs exist for every IP address in some parts of the network. The Blaster worm example from 2003 that I showed earlier is a striking example. Our estimate we present in the IEEE paper was about 800,000 hosts infected with the worm, while Microsoft’s direct measurements showed about 8 million hosts in the same timeframe.

Conclusions

Botnet infection data is widely available now from groups such as Arbor, Shadowserver, Team Cymru, and others. Data feeds from sinkholes and other measurements can be used by network administrators to identify infected hosts and remediate their problems. A number of these are covered in a recent report from ENISA entitled Proactive detection of network security incidents.

Obviously robust measurements are a crucial element to addressing the botnet problem. In the measurement community, we have identified gaps and inconsistencies in our available methods. Where we are going with this now is trying to standardize methodologies so we can measure consistently. Furthermore, we’re trying to identify the causes for the gaps in the methodologies (e.g. network vs host measurements) and provide stronger data by closing those gaps. Based on this data, we also work globally to identify working strategies that effectively shut down botnets and drop infection rates. We then want to coordinate these efforts globally to lead to lower infections in each region.

Over the last two years in particular, Arbor has grown as a company both in the solutions we provide and the problems those solutions enable our customers to solve. Most people, among our customers and across the industry as a whole, understand the thought leadership and expertise ASERT delivers through our blog, and the technical expertise we provide through our security reports and frequent media appearances.

I wanted to take a few minutes and explain the importance of ASERT, ATLAS, and the Security Intelligence they provide. I would also like to introduce a strategic addition to the security research team who will drive new capabilities and areas of focus for Arbor Networks.

ASERT is a world-renowned group of security engineers and researchers dedicated to monitoring Internet threats at all times. With ASERT, service providers and enterprises gain the expertise needed to reinforce their overworked security response groups and optimize the defense of their entire network infrastructure. ASERT lets our customers and products detect and mitigate DDoS attacks, worms and other security threats long before they impact business service availability and integrity.

In partnership with our service provider customers, Arbor Networks launched ATLAS in February 2007, creating the world’s largest distributed darknet sensor network. Today, ATLAS sees 24Tbps of Internet traffic. For Arbor customers, ATLAS delivers a globally scoped view of malicious traffic traversing the backbone networks that form the Internet’s core. Additionally, the ATLAS Intelligence Feed (AIF) provides built-in, automated protection from virtually all known botnets plus a real-time update service that protects customers from new botnets as they emerge. No other vendor can deliver the combination of micro- and macro-level visibility like Arbor does.

In concert with our investments in ASERT and ATLAS, we have focused on increasing the capabilities of our solutions via new products, such as the enterprise and data center focused Pravail Availability Protection System. Pravail APS is a purpose-built platform to identify and block application-layer DDoS attacks that threaten enterprise and data center availability.

With that backdrop, I am excited to announce an important addition to the leadership team at Arbor Networks. Dan Holden has joined Arbor Networks, reporting directly to me as Director of Security Research, responsible for overseeing and leading the strategy and execution of Arbor’s ASERT, ATLAS, and overall Security Research mission.

There are a few reasons why it was the right time to bring on an executive level leader for Security Research:

• To add to Arbor, someone who has successful built out world class, marketplace differentiated research functions that span both Service Providers and Enterprise
• To provide complimentary, new DNA into the team as we expand and grow our security research function and evolve our products to take on new types of security problems
• To work with Dr. Jose Nazario, Senior Manager of Security Research, and provide an unprecedented “one-two punch” in the expertise and capabilities we can provide our customers.

I am excited for Arbor Networks but more importantly, I am excited for our customers as they get to benefit most directly from Dan, Jose, and the rest of the security research and intelligence team we have assembled here at Arbor. Welcome aboard, Dan, we are fortunate and excited to have you join the Arbor team!