Computer scientists from the University of California, San Diego broke “the terabyte barrier” — and a world record — when they sorted more than one terabyte of data (1,000 gigabytes or 1 million megabytes) in just 60 seconds. During this 2010 “Sort Benchmark” competition — the “World Cup of data sorting” — the computer scientists from the UC San Diego Jacobs School of Engineering also tied a world record for fastest data sorting rate. They sorted one trillion data records in 172 minutes — and did so using just a quarter of the computing resources of the other record holder.

Companies looking for trends, efficiencies and other competitive advantages have turned to the kind of heavy duty data sorting that requires the hardware muscle typical of data centers. The Internet has also created many scenarios where data sorting is critical. Advertisements on Facebook pages, custom recommendations on Amazon, and up-to-the-second search results on Google all result from sorting data sets as large as multiple petabytes. A petabyte is 1,000 terabytes.

“If a major corporation wants to run a query across all of their page views or products sold, that can require a sort across a multi-petabyte dataset and one that is growing by many gigabytes every day,” said UC San Diego computer science professor Amin Vahdat, who led the project. “Companies are pushing the limit on how much data they can sort, and how fast. This is data analytics in real time,” explained Vahdat. Better sort technologies are needed, however. In data centers, sorting is often the most pressing bottleneck in many higher-level activities, noted Vahdat who directs the Center for Networked Systems (CNS) at UC San Diego.

The two new world records from UC San Diego are among the 2010 results released recently on http://sortbenchmark.org — a site run by the volunteer computer scientists from industry and academia who manage the competitions. The competitions provide benchmarks for data sorting and an interactive forum for researchers working to improve data sorting techniques.

World Records

The Indy Minute Sort and the Indy Gray Sort are the two data sorting world records the UC San Diego computer scientists won in 2010, the first year they entered the Sort Benchmark competition.

In the Indy Minute Sort, the researchers sorted 1.014 terabytes in one minute — thus breaking the minute barrier for this terabyte sort for the first time.

“We’ve set our research agenda around how to make this better…and also on how to make it more general,” said UC San Diego computer science PhD student Alex Rasmussen, the lead graduate student on the team.

The team also tied the world record for the Indy Gray Sort which measures sort rate per minute per 100 terabytes of data.

“We used one forth the number of computers as the previous record holder to achieve that same sort rate performance — and thus one fourth the energy, and one fourth the cooling and data center real estate,” said George Porter, a Research Scientist at the Center for Networked Systems at UC San Diego. The Center for Networked Systems is an affiliated Center of the California Institute for Telecommunications and Information Technology (Calit2).

 

Both world records are in the Indy category — meaning that the systems were designed around the specific parameters of the Sort Benchmark competition. The team is looking to generalize their results for the “Daytona” competition and for use in the real world.

“Sorting is also an interesting proxy for a whole bunch of other data processing problems. Generally, sorting is a great way to measure how fast you can read a lot of data off a set of disks, do some basic processing on it, shuffle it around a network and write it to another set of disks,” explained Rasmussen. “Sorting puts a lot of stress on the entire input/output subsystem, from the hard drives and the networking hardware to the operating system and application software.”

Balanced Systems

The data sorting challenges the computer scientists took on are quite different from the modest sorting that anyone with off the shelf database software can do by comparing two tables. One of the big differences is that data in terabyte and petabyte sorts is well beyond the memory capacity of the computers doing the sorting.

In creating their heavy duty sorting system, the computer scientists designed for speed and balance. A balanced system is one in which computing resources like memory, storage and network bandwidth are fully utilized and as few resources as possible are wasted.

“Our system shows what’s possible if you pay attention to efficiency — and there is still plenty of room for improvement,” said Vahdat, holder of the SAIC Chair in Engineering in the Department of Computer Science and Engineering at UC San Diego. “We asked ourselves, ‘What does it mean to build a balanced system where we are not wasting any system resources in carrying out high end computation?’” said Vahdat. “If you are idling your processors or not using all your RAM, you’re burning energy and losing efficiency.” For example, memory often uses as much or more energy than processors, but the energy consumed by memory gets less attention.

To break the terabyte barrier for the Indy Minute Sort, the computer science researchers built a system made up of 52 computer nodes. Each node is a commodity server with two quad-core processors, 24 gigabytes (GB) memory and sixteen 500 GB disks — all inter-connected by a Cisco Nexus 5020 switch. Cisco donated the switches as a part of their research engagement with the UC San Diego Center for Networked Systems. The compute cluster is hosted at Calit2.

To win the Indy Gray Sort, the computer science researchers sorted one trillion records in 10,318 seconds (about 172 minutes), yielding their world-record tying data sorting rate of 0.582 terabytes per minute per 100 terabytes of data. The winning sort system is made up of 47 computer nodes similar to those used in the minute sort.

According to wolframalpha.com, 100 terabytes of data is roughly equivalent to 4,000 single-layer Blu-Ray discs, 21,000 single-layer DVDs, 12,000 dual-layer DVDs or 142,248 CDs (assuming CDs are 703 MB).

The roster for TritonSort, the world record breaking sort team:

Alex Rasmussen, Radhika Niranjan Mysore and Michael Conley are computer science graduate students at UC San Diego. Alexander Pucher is a visiting student from Vienna University of Technology. Harsha V. Madhyastha is a postdoctoral researcher in computer science at UC San Diego. George Porter is a Research Scientist at the Center for Networked Systems at UC San Diego. Amin Vahdat holds the SAIC Chair in Engineering in the Department of Computer Science and Engineering and directs the Center for Networked Systems (CNS) at UC San Diego. Learn more about Sort Benchmark at http://sortbenchmark.org/

 

Related posts:

1. New world record in energy-efficient data processing
2. Guinness World Records Certifies NASA’s Aircraft Speed Record
3. Computer scientists scale ‘layer 2′ data center networks to 100,000 ports and beyond
4. 100-meter sprint world record could go as low as 9.48 seconds
5. New world record for solar-to-grid conversion efficiency

 

That is a lot of data in a very short amount of time.I would be interested in seeing the algorithms and system set up they were using.

A great tutorial for those who want to learn more about Ruby on Rails. The author helps you set up a good working environment and SCM using git. He also helps you set up a git push to Heroku so you can test out your app on the internet. In all a really good tutorial for people new to Ruby/ Rails.

Wow. I'm scared this guy makes and votes on bills that directly affect me.

I especially like their use of "hacking software". Nothing like a little FUD to start out the week.

Video demonstration of a variation of the Yoshimoto cube, invented in 1971. Link includes a video on how to make one yourself out of paper, as well as an introduction to the Banach-Tarski paradox ("a pea can be chopped up and reassembled into the Sun").

Made up of eight interconnected cubes, it’s capable of unfolding itself in a cyclic fashion. That means you could keep folding, or unfolding it, indefinitely.

In the toy Brocoum’s mom bought him, the cubes were also cut into two identical polyhedra, each capable of forming a Yoshimoto cube containing a hollow space inside with the exact shape of another Yoshimoto cube “open” as as dodecahedron (several other shapes are also possible).

If that sounded somewhat complicated, the animated GIF on the right may illustrate the miracle of the multiplication of Yoshimoto cubes better. It’s simply that a solid Yoshimoto cube can unfold into two hollow Yoshimoto cubes.

Folding the Yoshimoto Cube

Wow this is really cool. Reminds me of a cross between the Rubik's Cube, Rubik's Snake and the Rubik's Rings. I will probably have to get one for the kids.

Bittorrent a copy and keep the seed going.

Help encourage diversity in the IT workplace. Vote for Barbie to become a Computer Engineer.

26 people went before you!
Click here to watch them!

works for all platforms: (version 2.03.17)
Windows, Linux, Mac

Tired of your virtual life? This site makes it easy to kill off the "virtual you" so you can get back to living in the real world. Pretty cool idea.

The ModNation Racers beta starts today. If you haven't heard about it yet check out youtube for some video. If you have a PS3 I highly suggest trying to get your hands on a beta voucher. Lilly and Belles love it because they make their racer and cars all pink ;)

Seriously? $26 software allowing insurgents to monitor the video feed of a $15 million dollar aircraft and billion dollar program. General Atomics excuse is that their custom protocols don't easily allow for encryption. The biggest fail is that the US govt assumed that security through obscurity would work against an unorthodox opponent.

Our 1 Billionth Spam Message

PUBLISHED: DECEMBER 15, 2009

58 tweets TOP1K retweet

100Share

On Wednesday, December 9, 2009 at 06:20 (GMT) Project Honey Pot received its billionth email spam message. The message, a picture of which is displayed below, was a United States Internal Revenue Service (IRS) phishing scam. The spam email was sent by a bot running on a compromised machine in India (122.167.68.1). The spamtrap address to which the message was sent was originally harvested on November 4, 2007 by a particularly nasty harvester (74.53.249.34) that is responsible for 53,022,293 other spam messages that have been received by Project Honey Pot.

Every time Project Honey Pot receives a message we estimate that another 125,000 are sent to real victims. Our billionth message represents approximately 125 trillion spam messages that have been sent since Project Honey Pot started in 2004.

At this milestone, we wanted to take a second to report some of our findings. Our goal is not to rehash the same old insights but instead to give a new picture that only looking at five years and a billion data points can produce.

Who Are These Spammers?

Several organizations publish regular reports on the source countries for spam. We have one of our own. The problem is that these reports tell very little about the actual source of spam messages because of the nature of how spam is sent today.

Rather than sending spam directly, spammers primarily use "bot" machines in order to effectively launder their identities. These bots are PCs that have been compromised by a virus and whose owner usually does not know they are being used to send spam. The process is not unlike the stereotypical scene in a movie where the villain keeps his phone call from being traced by relaying it through a number of connections. Similarly, spammers' use of bots can make their messages look like they are coming from somewhere completely different than their actual location. As a result, lists of spam origin countries tell you very little about where the spammers are actually located.

On the other hand, they can help provide insight into a country's security policies because they give evidence on the number of bots operating within a country's borders. Since every country will have a different number of PCs, to make this number comparable we needed to create a ratio. We decided to look at the number of compromised machines operating within the country divided by the number of security professionals operating in the country. This gives us a relative IT security score. As a proxy for the number of security professionals we used members in Project Honey Pot. Here are the results:

Best IT Security #1 Finland #2 Canada #3 Belgium #4 Australia #5 Netherlands #6 United States #7 Norway #8 New Zeland #9 Sweden #10 Estonia

Worst IT Security #1 China #2 Azerbaijan #3 South Korea #4 Colombia #5 Macedonia #6 Turkey #7 Viet Nam #8 Kazakhstan #9 Macau #10 Brazil

Because sending spam remains the primary use of bots, Project Honey Pot has a unique perspective on bot network activity. Since 2004, active bots have grown at a compound annual growth rate of more than 378%. In other words, the number of bots has nearly quadrupled ever year. In 2009, you could find nearly 400,000 active bots engaged in malicious activity on any given day with several million active over the course of any month.

Fortunately, Project Honey Pot's coverage of active botnets has grown over time at an even faster rate. In 2006, we saw less than 20% of the active bots on any given day. Today we see more than 80%.

While tracking bots has become a critical aspect of Project Honey Pot, we remain curious where the spammers are actually located. To get at this information it's critical to look at spammer activities that are not laundered through bots. While sending email spam can easily be done in parallel (i.e., 1 million machines can send one message each) harvesting email addresses, which involves crawling web pages, cannot. This makes sense: crawling without centralized command and control will result in a lot of crawling of the same popular pages over and over again.

Our research indicates that, unlike the bots used to send spam, the machines used for harvesting tend to be more permanent, stable, and closely connected to the actual spammer's location. So where are the spammers actually located? We think the list below gives the most accurate approximation.

Where Harvesters Are
#1	United States
#2	Spain
#3	Netherlands
#4	United Arab Emirates
#5	Hong Kong
#6	Romania
#7	Great Britain
#8	China
#9	South Africa
#10	Germany

How Do They Operate?

On average, spammers today are faster than they've ever been before. The chart below indicates the average time from harvesting an email address from a web page to when the spammer sends the first email to that address.

2004	49 days 18 hours 54 minutes 15 seconds
2005	32 days 15 hours 39 minutes 41 seconds
2006	29 days 29 hours 10 minutes 24 seconds
2007	23 days 11 hours 53 minutes 03 seconds
2008	22 days 12 hours 36 minutes 54 seconds
2009	21 days 17 hours 17 minutes 28 seconds

We have found that speed is tied to the content of the message. "Product" spammers -- those selling an actual product of some kind, whether it be fake pharmaceuticals, college degrees, or mortgage loans -- tend to operate on a slower cycle, spending approximately a month gathering email addresses and then targeting those addresses with a set of spam campaigns. Product spammers tend to hold on to email addresses longer and send on average several messages a week to each address on their list.

On the other hand "Fraud" spammers -- those committing phishing or so-called "419" advanced fee scams -- tend to send to and discard harvested addresses almost immediately. The increased average speed of spammers appears to be mostly attributable to the rise in spam as a vehicle for fraud rather than an increasing efficiency among traditional product spammers.

One intriguing insight our data provides is that bad guys take vacations too. For example, there is a 21% decrease in spam on Christmas Day and a 32% decrease on New Year's Day. Monday is the biggest day of the week for spam, while Saturday receives only about 60% of the volume of Monday's messages.

The chart below shows the time of day spammers are most likely to send their messages. All times in the chart are set to the East Coast timezone of the United States (GMT -0500).

Whom Do They Target?

Spammers are a creative bunch and we have seen a wide variety of offers show up in the one billion messages we have received. Among products sold through spam, pharmaceuticals remain the most popular. To give you a sense, we've seen the word "Viagra" spelled at least 956 different ways in order to try and trick spam filters (e.g., VIAGRA, V1AGRA, V1@GR@, V!AGRA, VIA6RA, etc.).

While spammers will often alter their messages to look different, some are remarkably consistent. The table below shows the top message FROM/SUBJECT line pairs over the last five years. We have also included our estimate of how many of each message was sent Internet-wide.

RANK	FROM	SUBJECT	EST. INTERNET-WIDE VOLUME
#1	Instant Booster	Can you afford to lose 300,000 potential customers?	100 billion
#2	Internal Revenue Service	Notice of Underreported Income	91 billion
#3	Feed Blaster	Receive hundreds of targeted hits to your website	65 billion
#4	Hit-Booster	How to get free quality visitors to your website?	51 billion
#5	Feed Blaster	Feed Blaster puts your ad right to the screens of millions	44 billion

To give you some sense, assuming an average message storage requirement of 4KB, over the last 5 years the total storage requirement imposed on the Internet by just the spammers sending the top-20 spam campaigns was over 2.5 petabytes.

Beyond the product spam, fraudulent spam increasingly dominates our spam stream. The chart below shows the relative distribution of the most phished organizations online.

While banks and financial institutions still make up a majority of the phishing scams circulated via spam, social networks are increasingly targeted. In 2008, there were virtually no Facebook phishing message. Today Facebook is the second most phished organization online and, if current trends continue, is on track to take the top spot in 2010.

The Future of Spam

The good news for email users is that filtering technologies have done a terrific job keeping most of the volume of spam messages out of their inboxes. Behind the scenes, however, the volume of email spam continues to grow at a blistering pace. While spam may strike the average user as a minor annoyance, the real risk it continues to pose is providing a viable business model to finance the construction of bot networks. Our research indicates that these bots are increasingly multi-purposed into vectors for new types of attacks ranging from annoyances like comment spam to real threats like denial of service attacks (DDoS).

For example, if you run a blog you are aware of the comment spam attacks your site faces every day. This new breed of spammers uses the forms on websites to post advertisements and links to pages they are paid to promote. Project Honey Pot has been tracking their behavior for two years and has witnessed its gowth in volume and sophistication.

Where Comment Spammers Are
#1	United States
#2	China
#3	Brazil
#4	Japan
#5	Russia
#6	South Korea
#7	Ukraine
#8	Poland
#9	Germany
#10	Hong Kong

Looking at the data patterns, comment spam in 2009 resembles email spam when Project Honey Pot began in 2004. While comment spammers today are tending to use a relatively limited set of machines to post their messages, if this new breed of spammers follows the email spammers' lead to massive adoption of bot networks then it will pose a significant threat to websites everywhere.

To counter these increasing threats, web administrators need to continue to share data about attacks they see on their own sites through efforts such as Project Honey Pot. Over the next year, we will be launching a number of new initiatives to increase the protection we offer. In the meantime, if you run a website, we encourage you to become a member of Project Honey Pot today and encourage others to do so as well. Only by working together do we stand a chance to face the challenges that lie ahead.

Finally, thanks to all the current Project Honey Pot members as well as the organizations that have helped us build our infrastructure.

162 tweets TOP1K retweet

100Share

The Honey Pot Project just received it's Billionth SPAM email since it started in 2004. The sad thing is that this is only a small part of the 125 Trillion messages that have been sent out since they started.

Too bad they didn't include elinks and lynx statistics pre 2002.