"tech-news" via ehartwell in Google Reader

Suffering-oriented programming

Nathan — 2012-02-06T10:36:33-08:00

Someone asked me an interesting question the other day: "How did you justify taking such a huge risk on building Storm while working on a startup?" (Storm is a realtime computation system). I can see how from an outsider's perspective investing in such a massive project seems extremely risky for a startup. From my perspective, though, building Storm wasn't risky at all. It was challenging, but not risky.

I follow a style of development that greatly reduces the risk of big projects like Storm. I call this style "suffering-oriented programming." Suffering-oriented programming can be summarized like so: don't build technology unless you feel the pain of not having it. It applies to the big, architectural decisions as well as the smaller everyday programming decisions. Suffering-oriented programming greatly reduces risk by ensuring that you're always working on something important, and it ensures that you are well-versed in a problem space before attempting a large investment.

I have a mantra for suffering-oriented programming: "First make it possible. Then make it beautiful. Then make it fast."

First make it possible

When encountering a problem domain with which you're unfamiliar, it's a mistake to try to build a "general" or "extensible" solution right off the bat. You just don't understand the problem domain well enough to anticipate what your needs will be in the future. You'll make things generic that needn't be, adding complexity and wasting time.

It's better to just "hack things out" and be very direct about solving the problems you have at hand. This allows you to get done what you need to get done and avoid wasted work. As you're hacking things out, you'll learn more and more about the intricacies of the problem space.

The "make it possible" phase for Storm was one year of hacking out a stream processing system using queues and workers. We learned about guaranteeing data processing using an "ack" protocol. We learned to scale our realtime computations with clusters of queues and workers. We learned that sometimes you need to partition a message stream in different ways, sometimes randomly and sometimes using a hash/mod technique that makes sure the same entity always goes to the same worker.

We didn't even know we were in the "make it possible" phase. We were just focused on building our products. The pain of the queues and workers system became acute very quickly though. Scaling the queues and workers system was tedious, and the fault-tolerance was nowhere near what we wanted. It was evident that the queues and workers paradigm was not at the right level of abstraction, as most of our code had to do with routing messages and serialization and not the actual business logic we cared about.

At the same time, developing our product drove us to discover new use cases in the "realtime computation" problem space. We built a feature for our product that would compute the reach of a URL on Twitter. Reach is the number of unique people exposed to a URL on Twitter. It's a difficult computation that can require hundreds of database calls and tens of millions of impressions to distinct just for one computation. Our original implementation that ran on a single machine would take over a minute for hard URLs, and it was clear that we needed a distributed system of some sort to parallelize the computation to make it fast.

One of the key realizations that sparked Storm was that the "reach problem" and the "stream processing" problem could be unified by a simple abstraction.

Then make it beautiful

You develop a "map" of the problem space as you explore it by hacking things out. Over time, you acquire more and more use cases within the problem domain and develop a deep understanding of the intricacies of building these systems. This deep understanding can guide the creation of "beautiful" technology to replace your existing systems, alleviate your suffering, and enable new systems/features that were too hard to build before.

The key to developing the "beautiful" solution is figuring out the simplest set of abstractions that solve the concrete use cases you already have. It's a mistake to try to anticipate use cases you don't actually have or else you'll end up overengineering your solution. As a rule of thumb, the bigger the investment you're trying to make, the deeper you need to understand the problem domain and the more diverse your use cases need to be. Otherwise you risk the second-system effect.

"Making it beautiful" is where you use your design and abstraction skills to distill the problem space into simple abstractions that can be composed together. I view the development of beautiful abstractions as similar to statistical regression: you have a set of points on a graph (your use cases) and you're looking for the simplest curve that fits those points (a set of abstractions).

The more use cases you have, the better you'll be able to find the right curve to fit those points. If you don't have enough points, you're likely to either overfit or underfit the graph, leading to wasted work and overengineering.

A big part of making it beautiful is understanding the performance and resource characteristics of the problem space. This is one of the intricacies you learn in the "making it possible" phase, and you should take advantage of that learning when designing your beautiful solution.

With Storm, I distilled the realtime computation problem domain into a small set of abstractions: streams, spouts, bolts, and topologies. I devised a new algorithm for guaranteeing data processing that eliminated the need for intermediate message brokers, the part of our system that caused the most complexity and suffering. That both stream processing and reach, two very different problems on the surface, mapped so elegantly to Storm was a strong indicator that I was onto something big.

I took additional steps to acquire more use cases for Storm and validate my designs. I canvassed other engineers to learn about the particulars of the realtime problems they were dealing with. I didn't just ask people I knew. I also tweeted out that I was working on a new realtime system and wanted to learn about other people's use cases. This led to a lot of interesting discussions that educated me more on the problem domain and validated my design ideas.

Then make it fast

Once you've built out your beautiful design, you can safely invest time in profiling and optimization. Doing optimization too early will just waste time, because you still might rethink the design. This is called premature optimization.

"Making it fast" isn't about the high level performance characteristics of a system. The understanding of those issues should have been acquired in the "make it possible" phase and designed for in the "make it beautiful" phase. "Making it fast" is about micro-optimizations and tightening up the code to be more resource efficient. So you might worry about things like asymptotic complexity in the "make it beautiful" phase and focus on the constant-time factors in the "make it fast" phase.

Rinse and repeat

Suffering-oriented programming is a continuous process. The beautiful systems you build give you new capabilities, which allow you to "make it possible" in new and deeper areas of the problem space. This feeds learning back to the technology. You often have to tweak or add to the abstractions you've already come up with to handle more and more use cases.

Storm has gone through many iterations like this. When we first started using Storm, we discovered that we needed the capability to emit multiple, independent streams from a single component. We discovered that the addition of a special kind of stream called the "direct stream" would allow Storm to process batches of tuples as a concrete unit. Recently I developed "transactional topologies" which go beyond Storm's at-least-once processing guarantee and allow exactly-once messaging semantics to be achieved for nearly arbitrary realtime computation.

By its nature, hacking things out in a problem domain you don't understand so well and constantly iterating can lead to some sloppy code. The most important characteristic of a suffering-oriented programmer is a relentless focus on refactoring. This is critical to prevent accidental complexity from sabotaging the codebase.

Conclusion

Use cases are everything in suffering-oriented programming. They're worth their weight in gold. The only way to acquire use cases is through gaining experience through hacking.

There's a certain evolution most programmers go through. You start off struggling to get things to work and have absolutely no structure to your code. Code is sloppy and copy/pasting is prevalent. Eventually you learn about the benefits of structured programming and sharing logic as much as possible. Then you learn about making generic abstractions and using encapsulation to make it easier to reason about systems. Then you become obsessed with making all your code generic, with making things extensible to future-proof your programs.

Suffering-oriented programming rejects that you can effectively anticipate needs you don't currently have. It recognizes that attempts to make things generic without a deep understanding of the problem domain will lead to complexity and waste. Designs must always be driven by real, tangible use cases.

You should follow me on Twitter here.

The FBI file of Steven Paul Jobs

Xeni Jardin — 2012-02-09T23:12:14-08:00

In 1991, the FBI began interviewing Steve Jobs and people he worked with, as the CEO of Next Inc. "began to be considered as a candidate for sensitive, presidential appointments."

Here is Steve Jobs' FBI file, released under the Freedom of Information Act.

"Several individuals questioned Mr. Jobs' honesty stating that Mr. Jobs will twist the truth and distort reality in order to achieve his goals," reads the FBI summary.

Other elements of note: as a student, he had a 2.65 GPA. There was a bomb threat against him in 1985. There's a passing reference to a "hippie friend" on whose apple orchard the man who would later co-found Apple worked. And there's an excellent specimen of early 1990s FBI fax art, page 129.

You'll be shocked, shocked I say, to learn that Apple has declined to comment on the file's release. More context: WaPo, Wired, LA Times, SF Chron.

(Photo: Jobs beneath a photograph of him and Apple-co founder Steve Wozniak from the early days of Apple during the launch of the iPad in San Francisco, January 27, 2010. REUTERS.)

Software piracy is vital to preservation

Cory Doctorow — 2012-01-27T06:09:00-08:00

A PC World editorial by Benj Edwards recounts the history of "copy protection*" for software, and discusses how the cracks-scene, which busted open these software locks, is the only reason the legacy of old software is available today. There's a trite story about the persistence of paper and the ephemerality of bits, which goes something like this: "We can still read ancient manuscripts, but we can't read Letraset Ready, Set, Go! files from the 1980s." This is only true in a very limited sense: if you can crack the copy-protection on R,S,G! you can run it perfectly well in a little Mac emulator on a modern computer, with lots of headroom to spare (the laptop I'm typing this on being approximately ten bazillion times more powerful than the last machine I used R,S,G! on). The business of software preservation and data longevity is a lot simpler than the story would have you believe** (assuming you don't care about breaking the law to bust open copy protection and to get old copies of Mac System 6.x to run things on).

It may seem counterintuitive, but piracy has actually saved more software than it has destroyed. Already, pirates have spared tens of thousands of programs from extinction, proving themselves the unintentional stewards of our digital culture.
Software pirates promote data survival through ubiquity and media independence. Like an ant that works as part of a larger system it doesn’t understand, the selfish action of each digital pirate, when taken in aggregate, has created a vast web of redundant data that ensures many digital works will live on...
For a sample slice of what’s at stake when it comes to vanishing software, let’s take a look at the video game industry. The Web’s largest computer and video game database, MobyGames, holds records of about 60,000 games at present. Roughly 23,000 of those titles were originally released on computer systems that used floppy disks or cassette tapes as their primary storage or distribution medium.
23,000 games! If game publishers and copyright law had their way, almost all of those games would be wiped from the face of the earth by media decay over the next 10 years. Many would already be lost.

The article is long and thoughtful, and covers a lot of ground. I highly recommend it.

Why History Needs Software Piracy (Thanks, Rainman!)

* The term "copy-protection" is pretty misleading. Speaking as a former systems administrator, the way I "protect" my stuff was by making copies -- that is, backups. True, these are encrypted, but they're encrypted to a key that I posses.

** There's a separate question about media preservation, because old floppies and Zip carts and such are basically shit. But that's OK, since a modern hard drive can store pretty much all the floppies you ever handled without breaking a sweat. If you have (or had) the presence of mind to move all your data from floppies to your HDD, and if you keep your HDD backed up, you are pretty well-preserved. Much better-preserved than your hardcopy book library, which can't be backed up offsite without a photocopier, an army of interns and a lot of time, bother, and shipping containers.

Solving the Shakespeare Million Monkeys Problem in Real-time with Parallelism and SignalR

Scott Hanselman — 2011-11-11T17:09:22-08:00

A little over 18 months ago I was talking to Stephen Toub (he of the Parallel Computing fame) about parallelism and the kinds of problems it could solve.

I said, naively, "could we solve the million monkey's problem?"

He said, "the what?"

"You know, if you have an infinite number of monkeys and an infinite number of keyboards they will eventually write Shakespeare."

We brainstormed some ideas (since Stephen is a smarter than I, this consisted mostly of him gazing thoughtfully into the air while I sat on my hands) and eventually settled on an genetic algorithm. We would breed thousands of generations of (hypothetical) monkeys a second and then choose which ones would be allowed to perpetuate the species based solely on their ability to write Shakespeare.

We used the .NET 4 Task Parallel Library to make it easier for the algorithm to scale to available hardware. I mean, anyone can foreach over a million monkeys. But loops like that in parallel over 12 processors takes talent, right? Well, kinda. A lot of it is done for you by the Parallelism Features in .NET and that's the point. It's Parallel Processing for the Masses.

We created a WinForms version of this application and I've used it on and off to demonstrate parallel computing on .NET. Then Paul Batum and I went to the Keeping It Realtime conference to demonstrate SignalR this last week. I didn't want to do the same "here's a real-time chat app" or "here's a map that shows its results in real-time" demos that one always does at these kinds of things. I suggested that we port our WinForms Shakespeare Monkey demo to ASP.NET and SignalR and that's what Paul proceeded to do.

When doing something that is crazy computationally intensive but also needs to return real-time results you might think to use node for the real-time notification part and perhaps spawn off another process and use C or something for the maths and then have them talk to each others. We like node and you can totally run node on IIS or even write node in WebMatrix. However, node is good at some things and .NET is good at some things.

For example, .NET is really good at CPU-bound computationally intensive stuff, like, I dunno, parallel matrix multiplication in F# or the like. ASP.NET is good at scaling web sites like Bing, or StackOverflow. You may not think IIS and ASP.NET when you think about real-time, but SignalR uses asynchronous handlers and smart techniques to get awesome scale when using long-polling and scales even more in our labs when using an efficient protocol like WebSockets with the new support for WebSockets in .NET 4.5.

So, we wanted to see if you combined asynchronous background work, use as many processors as you have, get real-time status updates via SignalR over long-polling or Web Sockets, using C#, .NET 4.5, ASP.NET and IIS.

It takes about 80,000 generations of monkeys at thousands of monkey generations a second (there's 200 monkeys per generation) to get the opening line of Hamlet. So that's ~16,000,000 monkeys just to get this much text. As they say, that's a lot of monkeys.

Here's the general idea of the app. The client is pretty lightweight and quite easy. There's two boxes, two buttons and a checkbox along side some text. There's some usual event wireup with started, cancelled, complete, and updateProgress, but see how those are on a monkey variable? That's from $.connection.monkeys. It could be $.connection.foo, of course, as long as it's hanging off $.connection.

Those functions are client side but we raise them from the server over the persistent connection then update some text.

<script src="Scripts/jquery-1.6.4.min.js"></script>    
<script src="Scripts/json2.min.js"></script>
<script src="Scripts/jquery.signalR.min.js"></script>
<script src="signalr/hubs"></script>
<script>
    $(function () {
        $('#targettext').val('To be or not to be, that is the question;\nWhether \'tis nobler in the mind to suffer\n\The slings and arrows of outrageous fortune,\n\Or to take arms against a sea of troubles,\n\And by opposing, end them.');

        var monkeys = $.connection.monkeys,
            currenttext = $('#currenttext'),
            generationSpan = $('#generation'),
            gpsSpan = $('#gps');

        monkeys.updateProgress = function (text, generation, gps) {
            currenttext.val(text);
            generationSpan.text(generation);
            gpsSpan.text(gps);
        };

        monkeys.started = function (target) {
            $('#status').text('Working...');
            $('#targettext').val(target);
            $('#cancelbutton').removeAttr('disabled');
        };

        monkeys.cancelled = function () {
            $('#status').text('Cancelled');
            $('#cancelbutton').attr('disabled', 'disabled');
        };

        monkeys.complete = function () {
            $('#status').text('Done');
            $('#cancelbutton').attr('disabled', 'disabled');
        };

        $.connection.hub.start({}, function () {
            $('#startbutton').click(function (event) {
                $('#status').text('Queued...');
                monkeys.startTyping($('#targettext').val(), $('#isparallel').is(':checked'));
            });

            $('#cancelbutton').click(function (event) {
                monkeys.stopTyping();
            });
        });

    });
</script>

The magic start with $.connection.hub.start. The hub client-side code is actually inside ~/signalr/hubs. See how that's include a the top? That client-side proxy is generated based on what hub or hubs are on the server side.

The server side is structured like this:

[HubName("monkeys")]
public class MonkeyHub : Hub
{
    public void StartTyping(string targetText, bool parallel)
    {
    }

    public void StopTyping()
    {
    }

}

The StartTyping and StopTyping .NET methods are callable from the client-side via the monkeys JavaScript object. So you can call server-side C# from the client-side JavaScript and from the C# server you can call methods in JavaScript on the client. It'll make the most sense if you debug it and watch the traffic on the wire. The point is that C# and Json objects can flow back and forth which blurs the line nicely between client and server. It's all convention over configuration. That's how we talk between client and server. Now, what about those monkeys?

You can check out the code in full, but StartTyping is the kick off point. Note how it's reporting back to the Hub (calling back to the client) constantly. Paul is using Hub.GetClients to talk to all connected clients as broadcast. This current implementation allows just one monkey job at a time. Other clients that connect will see the job in progress.

public void StartTyping(string targetText, bool parallel)
{
    var settings = new GeneticAlgorithmSettings { PopulationSize = 200 };
    var token = cancellation.Token;

    
    currentTask = currentTask.ContinueWith((previous) =>
    {
        // Create the new genetic algorithm
        var ga = new TextMatchGeneticAlgorithm(parallel, targetText, settings);
        TextMatchGenome? bestGenome = null;
        DateTime startedAt = DateTime.Now;

        Hub.GetClients<MonkeyHub>().started(targetText);

        // Iterate until a solution is found or until cancellation is requested
        for (int generation = 1; ; generation++)
        {
            if (token.IsCancellationRequested)
            {
                Hub.GetClients<MonkeyHub>().cancelled();
                break;
            }

            // Move to the next generation
            ga.MoveNext();

            // If we've found the best solution thus far, update the UI
            if (bestGenome == null ||
                ga.CurrentBest.Fitness < bestGenome.Value.Fitness)
            {
                bestGenome = ga.CurrentBest;
                
                int generationsPerSecond = generation / Math.Max(1, (int)((DateTime.Now - startedAt).TotalSeconds));
                Hub.GetClients<MonkeyHub>().updateProgress(bestGenome.Value.Text, generation, generationsPerSecond);

                if (bestGenome.Value.Fitness == 0)
                {
                    Hub.GetClients<MonkeyHub>().complete();
                    break;
                }
            }
        }                
    }, TaskContinuationOptions.OnlyOnRanToCompletion);
}

If he wanted, he could use this.Caller to communicate with the specific client that called StartTyping. Inside ga.MoveNext we make the decision to go parallel or not based on that checkbox. This is where we pick two random high quality parent monkeys from our population for a potential future monkey. Hopefully one whose typing looks more like Shakespeare and less like a Regular Expression.

By simply changing from Enumerable.Range to ParallelEnumerable.Range we can start taking easily parallelizable things and using all the processors on our machine. Note the code is the same otherwise.

private TextMatchGenome[] CreateNextGeneration()
{
    var maxFitness = _currentPopulation.Max(g => g.Fitness) + 1;
    var sumOfMaxMinusFitness = _currentPopulation.Sum(g => (long)(maxFitness - g.Fitness));

    if (_runParallel)
    {
        return (from i in ParallelEnumerable.Range(0, _settings.PopulationSize / 2)
                from child in CreateChildren(
                    FindRandomHighQualityParent(sumOfMaxMinusFitness, maxFitness),
                    FindRandomHighQualityParent(sumOfMaxMinusFitness, maxFitness))
                select child).
                ToArray();
    }
    else
    {
        return (from i in Enumerable.Range(0, _settings.PopulationSize / 2)
                from child in CreateChildren(
                    FindRandomHighQualityParent(sumOfMaxMinusFitness, maxFitness),
                    FindRandomHighQualityParent(sumOfMaxMinusFitness, maxFitness))
                select child).
                ToArray();
    }
}

My 12 proc desktop does about 3800 generations a second in parallel.

Big thanks to Paul for the lovely port of this to SignalR and to Stephen Toub for the algorithm.

The code for the SignalR monkeys demo is on my BitBucket. Right now it needs .NET 4.5 and the Visual Studio Developer Preview, but you could remove a few lines and get it working on .NET 4, no problem.

Note that that SignalR works on .NET 4 and up and you can play with it today. You can even chat with the developers in the SignalR chat app in the 'aspnet' room at http://chatapp.apphb.com. Just /nick yourself then /join aspnet.

No monkeys were hurt in the writing of this blog post.

There’s more than one way to skin a patent troll

Rob Beschizza — 2011-08-11T11:42:16-07:00

Nilay Patel argues that we've become distracted by easy outrage at the patent system and are missing more promising opportunities for reform. The real problem is not how patents are awarded or what they cover, but patent trolls' ability to exploit the legal system: limiting damages and establishing a mechanical licensing system, for example, could dismantle their business models faster than a constitutionally-suspect fight over patents themselves. [TIMN]

An even easier way to get Windows Media Player to single-step a video

Raymond Chen - MSFT — 2011-08-02T07:00:01-07:00

Since my original article explaining how to get Windows Media Player to single-step a video, I've learned that there's an even easier way.

Pause the video.
To single-step forward, Ctrl+Click the Play button.
To single-step backwrd, Ctrl+Shift+Click the Play button.

Backward-stepping is dependent upon the codec; some of them will go backward to the previous keyframe.

The person who tipped me off to this feature: The developer who implemented it.

Remember: Sharing a tip does not imply that I approve of the situation that led to the need for the tip in the first place.

Microsoft Word does regex!

davidw — 2011-07-26T08:20:14-07:00

After literally decades of using Microsoft Word I just found out that it does regex!

I discovered this because I needed to delete comments inserted throughout my book manuscript, in the form . Hundreds of them. I was contemplating exporting to HTML so I could use a text editor that can handle this type of search and replace, but came across an article on how to use regular expressions in Word. Regexes let you use magical incantations that no one understands but that cause text to dance in little circles and transform themselves in puffs of smoke.

For example, to get rid of the pesky markup in my manuscript, I just had to tell the Replace dialogue to use wildcards, and then had it search for \<AU:?\>. The backslashes are necessary so that the angle brackets are not read as regex instructions. The question mark tells Word to find everything between <AU: and >. Simple! And it accepts far more complex regular expressions that. (Here’s a site that lets you test your regular expressions.)

Take a well deserved bow, Microsoft Word! (And then fix auto-numbers.)

Standards

(author unknown) — 2011-07-19T21:00:00-07:00

Win7's no-reformat, nondestructive reinstall [Newsletter Comp Version]

Eric Hartwell — 2011-07-13T23:43:44-07:00

If your software garbles this newsletter, read this issue at WindowsSecrets.com.

YOUR NEWSLETTER PREFERENCES Change
Delivery address: eric@exoware.com
Alternate address:
Locale: Canada M8Y2H1
Reader number: 84649-13263

Home

Newsletter

Lounge

Search

Polls

Contact

This issue

Library

Upgrade

Preferences

Unsubscribe

Windows Secrets Newsletter • Issue 297 • 2011-07-14 • Circulation: over 400,000

Repair and optimize your Windows
Reimage is not only a PC optimizer, registry fix, or an antivirus: Reimage reverses damage to your Windows OS, eliminating the need for reinstalling your PC. It scans and diagnoses, then repairs your damaged PC with technology that not only fixes your Windows but also reverses the damage already done with a full database of replacement files. Coupon code WS20OFF gives you 20% off any repair.
Reimage Online PC Repair

Table of contents
TOP STORY: Win7's no-reformat, nondestructive reinstall
LOUNGE LIFE: Yikes! My software hates my hardware!
WACKY WEB WEEK: Watch out! This photo booth has attitude
LANGALIST PLUS: ID those cryptic programs in Control Panel
KNOWN ISSUES: Looking for green (energy) in cloud computing
PATCH WATCH: Office File Validation patch leads to problems

You're receiving only our free content. Use the following link to upgrade and get our paid content immediately:

More info on how to upgrade

ADVERTISEMENTS

Recommended download: PC Matic
PC Matic boosts internet speed, enhances security, increases stability and maximizes performance. PC Matic is safe, secure, and simple to use software that automates the regular maintenance necessary to keep your PC fast and safe. Developed by PC Pitstop - where over 200 million free scans have been run. Free download & scan. Try it today.
PC Matic

Free download: smart productivity guide
With all the distractions that life brings, it's not easy to keep your productivity up. In this free twenty-page PDF, you will learn the most common productivity mistakes and applications to improve your typing, your time-organization, and your workflow. Although this PDF is primarily aimed at a computer-user, it comes packed with a lot of non-digital tips and advice as well. Download it today.
TradePub

Start your risk free trial today
Have you considered how much of your life is on your computer? Photos, emails, music, financial records, documents, and so much more. How would you feel if you woke up tomorrow and everything was gone? Every year, 43% of computer users lose their music, photos, documents, and more. Protect yourself with Carbonite Online Backup. Try Carbonite FREE for 15 days, risk-free - no credit card required.
Carbonite

See your ad here

TOP STORY

Win7's no-reformat, nondestructive reinstall

By Fred Langa

Microsoft won't tell you this, but you can do a fast, nondestructive, in-place, total reinstall of Windows 7 without damaging your user accounts, data, installed programs, or system drivers.

That means you may never have to do a full, from-scratch reinstall again, even when your system is misbehaving so badly that a full reformat-and-reinstall seems the only answer!

As I'm sure you know all too well, from-scratch reinstalls are ordeals. They take hours. And when a reinstall is done, you still have to recreate all your settings, reinstall all your software, and so on. It can take days to fully recover from a total reformat/reinstall.

Windows' little known, in-place reinstall takes only a fraction of that time and effort and yet completely rebuilds, repairs, and refreshes an existing Windows installation. It leaves your other software alone (no reinstallation needed!) while also leaving user accounts, names, and passwords untouched.

When you're finished, your Windows installation is just as it was before, except that all the system files are fully repaired, refreshed, and ready to go.

This nondestructive-reinstall ability has been in Windows since XP. (See this XP reinstall article that I wrote for another publication, years ago, when XP was new.) But — for reasons unknown — Microsoft has never made nondestructive reinstalls an official repair. In fact, it's not even listed in Win7's System Recovery Options (Help & How-to page).

(Vista users, you're not forgotten! The nondestructive reinstall process for Vista is nearly identical to that described in the rest of this article.)

You need three things before you begin

First, you need access to a standard Win7 installation DVD. Ideally, you have your original setup DVD tucked away somewhere. But if not, it's perfectly OK to borrow one from a colleague or friend, as long as it's the same 32- or 64-bit version as your installation. Ideally, it should also match the general type — retail disk or OEM/vendor-supplied disk — as well.

Why is it OK to borrow? A standard Win7 DVD actually contains all editions of Win7. For example, a 32-bit Win7 DVD has all the files for the 32-bit editions of Win7 Home, Win7 Professional, Win7 Ultimate, and so on. Your license key unlocks whichever edition you paid for.

This means it's perfectly legitimate for you to use someone else's Win7 setup DVD to install Windows on your system, as long as you use your own, original, paid-for product key during installation. Sharing disks is fine. Sharing keys is not.

This also provides an easy workaround for the all-too-common problem of PCs that ship without setup DVDs. As long as you can borrow a standard setup DVD of the same general type (as described above), you should be able to rebuild your system using it, with your own original, unshared product key.

And that's the second thing you need: your original 25-character product key. It's usually found on a sticker on your computer or in the paperwork that accompanies a retail copy of Windows 7.

If you've lost track of your product key, no problem: you can use a free keyfinder tool to dig it out. One such tool is the excellent, but absurdly named, Magical Jelly Bean (info/download). There also are many other free product keyfinders, as this About.com list shows.

The third and final thing you need is a current backup. Although the reinstall process works reliably, it's not infallible. Deep-seated system errors, OEM customizations, hardware trouble, or other variables may foil your reinstall efforts. Having a complete and current backup is a sensible precaution. (See the May 12 Top Story, "Build a complete Windows 7 safety net.")

Avoiding problems with Win7 Service Pack 1

If you're not running Win7 SP1, skip ahead to the next section.

You can also skip ahead if you're repairing an SP1 setup with a Win7 setup DVD that already contains the SP1 files — but such disks are still relatively rare as of this writing.

If you're still reading this paragraph, then you're most likely attempting to repair a Win7 SP1 setup with an original, pre-SP1 DVD. That's OK, provided you take an important preparatory step.

Win7 SP1 replaced many of your original system files with newer versions. If you try to install the older, original Win7 files over the newer SP1 files, the setup process will balk at what it sees as an erroneous downgrade.

So, if you're attempting to repair Win7 SP1 with a pre-SP1 DVD, you need to remove SP1 from the target PC before proceeding. Fortunately, that's easy, as Figure 1 shows.

Figure 1. Control Panel's Uninstall an update feature makes it a cinch to get the SP1 files out of the way.

Here's how: Open Control Panel's default view and click on Uninstall a program. In the left-hand pane of the uninstall applet, select View installed updates. When the Uninstall an update dialog box opens, scroll down to SP1 — listed as Service Pack for Microsoft Windows (KB976932) — and select Uninstall.

The system will churn for a while, but when it's done, SP1 will be gone — and you'll be able to use an original Win7 setup DVD to repair your system.

Start your Win7 in-place reinstallation

With Windows running — or limping, if it's badly broken — insert the Win7 setup DVD. When the AutoPlay dialog box pops up, click to run setup.exe. (See Figure 2.) Alternatively, you can run setup.exe manually by navigating to the DVD drive and selecting the setup file.

Figure 2. Run the Win7 installation DVD's setup.exe from inside your current Windows, either via AutoPlay (shown) or by manual selection.

When setup starts to run, the User Account Control asks whether you want to make changes to your PC. Answer Yes.

After a few moments, the Win7 installation process starts. (See Figure 3.)

Figure 3. When asked, click Install now to get the ball rolling.

Next, you'll see several information screens such as that in Figure 4.

Figure 4. Screens such as this one are just FYI and require no user intervention.

After several moments, you'll be asked whether you want to check online for updates related to the installation process, as shown in Figure 5. I recommend that you allow this online access to ensure essential files are current.

Figure 5. It's usually smart to allow the setup program to check for installation-related updates.

You can accept or reject the I want to help make Windows installation better option shown further down in the same dialog box. Your reinstall will proceed in the same way, regardless of your answer.

Assuming you allowed it, Windows then goes online to collect essential updates. (See Figure 6.)

Figure 6. It normally takes only a few moments to collect any needed installation updates.

Now we come to the heart of the matter. Windows asks you Which type of installation do you want? (See Figure 7.) But there's no Reinstall option listed. Instead, you have to bend the normal installation routine to your wishes by selecting the upgrade option.

Figure 7. Tell Windows a little white lie — that you're upgrading, even though you're really not.

You're not really upgrading; you're reinstalling the same version of the OS that's already on the PC. But the upgrade option leaves your files, settings, and programs in place, undisturbed. By pretending that you're upgrading, you can trick the setup program into doing an in-place reinstall!

And then, finally, the installation process begins in earnest, as shown in Figure 8.

Figure 8. Depending on the complexity of your setup and the speed of your hardware, reinstallation can take up to several hours.

There's no need for you to baby-sit the installation; no further user input is required until near the end. As you check in from time to time, note the progress indicators (see Figure 9) to keep track of the installation's evolution.

Figure 9. The progress bar at the bottom of the screen and the numeric indications (e.g., percent complete) let you easily monitor the process.

Windows reboots several times during the installation; you see various information screens— some unusual — along the way. (See Figure 10.)

Figure 10. Don't be alarmed at unusual-looking screens, such as this one.

At the very end of the installation process, you're asked to enter the product key from your original installation of Win7. (See Figure 11.)

Figure 11. When prompted, enter your original product key.

You also can elect to either Activate the fresh install of Windows immediately or wait. But unactivated instances of Windows get only limited access to Windows Updates and related services. I recommend activating without delay so you have immediate access to all Windows Update services.

If you defer activation, your desktop background will be set to an ominous black with the words This copy of Windows is not genuine (circled in yellow in Figure 12) in the lower-right corner. You'll also see Activation nag screens, as shown in Figure 12.

Figure 12. An unactivated Windows gets an ominous desktop background and recurring nag screens.

Personally, I think it's simpler and better to just activate and get it over with. (See Figure 13.)

Figure 13. Once it's activated, your reinstalled Windows desktop background can be set normally, and no further activation nags will appear.

By the way, activation is not a one-time-only thing — Microsoft allows for periodic reactivations of a given product key. Unless you've done several reactivations in a relatively short time, you should have no trouble. But if you do, just follow the remedies listed in the Activation Failed dialog box. As long as your product key is legit, Microsoft will work with you to resolve an accidental activation mishap.

Your Win7 reinstall is almost done!

At this point, the basic reinstall is finished. Your Win7 setup now has fresh copies of all the original system files. All that's left is to bring the installation up to date.

This generally takes several iterations of running Windows Update. (See Figure 14.) Run WU once, let the updates install, and then reboot. Repeat until WU reports that no further updates are available.

Figure 14. Your new Windows installation immediately needs to be brought up to date with the latest security patches from Windows Update. Don't put off this step!

When you're done, you have a fully refreshed, up-to-date copy of Windows with all your essential files, settings, and programs intact and working. In fact, if all has gone as planned, the only significant change you'll notice is that the original problem is gone!

With just a smidgen of luck and this article, you'll never again have to face a dreaded, start-over-from-scratch reformat/reinstall of Win7!

Have more info on this subject? Post your tip in the WS Columns forum.

Help people find this article on the Web:
Digg	Delicious	Twitter	Facebook	Other	Permalink

Please tell us how useful this article was to you:
Poor	Fair	Good	Great	Superb

Fred Langa is a senior editor of the Windows Secrets Newsletter. He was formerly editor of Byte Magazine (1987-91), editorial director of CMP Media (1991-97), and editor of the LangaList e-mail newsletter from its origin in 1997 until its merger with Windows Secrets in November 2006.

Table of contents

LOUNGE LIFE

Yikes! My software hates my hardware!

By Kathleen Atkins

When a Lounge member installed SP1, his Windows 7 stopped talking to his monitor.

WSG appealed to his fellow Lounge members for assistance — and they responded with advice.

But dissatisfaction still prevails: WSG's monitor just doesn't look as good as it did before SP1. More»

The following links are this week's most interesting Lounge threads, including several new questions to which you might be able to provide responses:

Office Applications
General Productivity	Data saved to disk	☼
Word Processing	Formatting equations
Spreadsheets	Macro requirement	☼
Databases	Validation issues	☼
Visual Basic for Apps	Outlook instance abnormally quits after Send command
Microsoft Outlook	Trouble with Outlook rules
Non-Outlook E-mail	Thunderbird 5 — trouble with links in e-mail
Windows
General Windows	Getting updates for software I'm not using	☼
Windows 7	Laptop will not boot Need audio driver after upgrading from XP to Win7 Windows 7 SP1 hoses PnP monitor	☼
Windows Vista	Can't launch programs from Start menu
Windows XP	Flash player stopped working
Windows Servers	Need SBS 2003 to restrict most of Internet
Internet/Connectivity
Internet Explorer	IE 9 — certain sites will *not* open
Third-Party Browsers	Info on updating Firefox
Application Servers	Exchange 2007 Outlook Web Access blocks JPG	☼
Networking	Using Wi-Fi in an apartment complex	☼
Other Technologies
Security & Backups	Who knows a good e-mail virus/malware scanner?	☼
Other Applications	Music CD for car stereo from YouTube videos
The Lounge
Forum Feedback	Attachments procedure	☼

☼ starred posts — particularly useful

If you're not already a Lounge member, use the quick registration form to sign up for free. The ability to post comments and take advantage of other Lounge features is available only to registered members.

If you're already registered, you can jump right in to today's discussions in the Lounge.

Help people find this article on the Web:
Digg	Delicious	Twitter	Facebook	Other	Permalink

The Lounge Life column is a digest of the best of the WS Lounge discussion board. Kathleen Atkins is associate editor of Windows Secrets.

Table of contents

WACKY WEB WEEK

Watch out! This photo booth has attitude

By Tracey Capen

Remember the photo booth? It was right up there with riding the Big Dipper and eating cotton candy as things you had to do at the amusement park.

Usually, you'd drop in your change, sit on the bench, and make funny faces to the camera. But what would you do if the booth proved unusually — and vocally — bossy? Play the video

Help people find this article on the Web:
Digg	Delicious	Twitter	Facebook	Other	Permalink

Table of contents

BONUS DOWNLOAD

A free download for all subscribers
You're more advanced than the typical Windows user — you read Windows Secrets! So what do you do when you have PC trouble? You jump in and start troubleshooting. Troubleshooting Windows 7 Inside Out, by Mike Halsey, is packed with prevention tips, troubleshooting techniques, and advice on system tools you can use to get Win7 running perfectly.

This month, all Windows Secrets subscribers can download an excerpt: Part 3, Chapter 16, Windows Problems Demystified. You'll learn about Windows 7's core files, Win7 security and policy features, advanced file restore, how to create a slipstreamed installation DVD, and more.

If you want to download this free excerpt, simply visit your preferences page, save any changes, and a download link will appear.

Info on the printed book: United States

Stanford CS101 Adopts JavaScript

Soulskill — 2011-07-11T11:40:00-07:00

mikejuk writes "In case further proof were needed that JavaScript shall indeed inherit the earth, we have the news that Stanford has adopted JavaScript to teach CS101 — Introduction to Computing Principles: 'The essential ideas of computing via little phrases of JavaScript code.' You can even try it out for yourself at Stanford's course page."

Read more of this story at Slashdot.

Bliki: SocialNetworks

(author unknown) — 2011-07-11T06:36:00-07:00

I wasn't cool enough to be in the first wave of invitations, but I have now got onto Google+, the Maybe Next Big Thing in social networks. It seems somewhat appropriate to mark this Momentous Event by writing a little bit about how I've used social networks so far, and some uninformed speculation about the impact of Google+

My initial reaction to social networks was suspicion. At heart I'm a misanthropic hermit with enough intelligence to put on a veneer of sociability. So I was never tempted by the early social networks like MySpace, and I still regularly refuse any requests to Link In.

I'd kept half an eye on Twitter, but was slow to get involved. Someone registered the name @martinfowler and started feeding my blog posts into it. I was worried that it may be a squatter, but it was actually a nice programmer who happily handed over the account when I asked for it. Eventually my colleague Doc List got me going on Twitter and now it's a regular part of my life. Shortly afterward I joined Facebook too.

I use the two networks quite differently, which is partly the function of my fate of being a geek celebrity. Twitter is a public facing stream, where anything I tweet goes out 35,000 followers. With Facebook I only link to people who I know personally. At a simple level I can think of Twitter as my professional side and Facebook as my social side - but that's a bit of a stretch since so many of my friends are also professional links.

Twitter is a fascinating social construct. It's a mix of features that I can't imagine someone would design up-front [1]. The 140 character limit is one of those silly technology fiats, yet it works well. As Elliotte Harold tweeted of Google Buzz: "Favorite Buzz feature: I'm not restricted to 140 chars. Least fav Buzz feature: everyone else not restricted to 140 chars :-)". Similarly the asymmetric follow (35,000 follow me, but I only follow 300) coupled with mentions (which get my attention) works better than I would have ever thought if someone described it to me. All in all it allows me to keep a good sense of connection without drowning in the flood of text.

Almost all the people I follow on Twitter are people I know personally. I read pretty much all the tweets from people I follow or people who mention me. I check the stream pretty regularly, several times a day most days. I post a few times a day when at home, a bit less when traveling. I choose my public posts conscious of the fact that I have so many followers, so I try to post things that are interesting to my perception of that crowd. Mostly these are links to interesting articles, which naturally include anything I write. I reply a bit more casually as I see things from friends or mentions. I find Twitter a useful source of web links to read which supplement my regular blog feeds. I also enjoy the feedback on my own contributions that appear whenever I tweet a new post.

With Facebook, I know that people who see my posts are only people I know reasonably well. The other difference is that my non-geek friends don't use Twitter but many do use Facebook. I don't check Facebook so often, usually only once or twice a day. My posts there don't include geek stuff, mostly they are either interesting social things we are doing. One common thing I links to my photo albums.

One issue for me is how to deal with people who are on both services. My approach is to almost never post the same thing to both places, along the geek/social divide. But several of my friends copy all posts to both services. I'm still unsure how to deal with that, I could block them on Facebook, but then I'd miss some conversations.

I use a consistent avatar photo on any site that I use. People are used to recognizing faces, so I think it's only reasonable to use a regular face photo as an avatar. It's frustrating when someone posts and I can't remember who it is because I haven't learned to match their face to their name yet. [2]

Overall I've come to be very glad of both services. The nature of my life means I have friends all over the globe and it's good to keep some low-level regular contact with them. I was never one for sending off emails, and have always disliked phone calls. Posting short messages about what we're up to work well for me.

So how will Google+ fit into this? Of course I don't really know, there's no sensible way of predicting the outcome. The question for me is whether Google offers something that the current Twitter/Facebook combo does not. From the small amount of use I've put it too so far it looks well put together. Circles seem to me to be a very useful feature that matches how I see most people's social interactions. [3] (Although as George Dinwiddie pointed out - remember that Google (and Doubleclick) are in all your circles.)

On the whole I'm very happy with Twitter and will need a lot of persuading to shift off it. Facebook, I'm less excited about and could easily see me moving. The circles notion could be just right for that context.

But the key issue here is that it doesn't matter what I think or do, it's what all my contacts do that counts. I mostly have a Facebook account because people I want to keep in contact with have a Facebook account. If they don't move to Google+ I'm staying with Facebook even if Google+ is nicer. That's why Google+ has a huge mountain to climb, especially since I don't want to check Yet Another Social Network. I suspect there may only be room for one of Google+ and Facebook.

So far this makes Google+ only marginally interesting to me. It's rather like Wave was - a cool idea but of little use if nobody else is using it. What may be the difference is tying in with Gmail. I always got the sense that the Wave and Gmail teams weren't on speaking terms, hence their software never seemed to be either. Google+ allows you to link to people who aren't on Google+ but do have your email address, your posts go to these people as email. I could do with this already for some non-Facebook friends, and it could be an important booster to getting people to use Google+. In general there seems to be a determination with Google+ to tie in with other Google services. This may be the lever they need to shift Facebook, other than their obvious advantage, but it will still be a big challenge.

1: I'm sure that somewhere on the web is a really interesting article on the evolution of Twitter's features and their impact on social interaction. When you find it, please tweet it to me.

2: If you do use a face as your avatar, please use full face only (not head and shoulders) otherwise it will be too small to see. Also make sure there's enough light on your face to be able to see it properly at that small size.

3: I've heard some people say that circles are a big innovation. I find that astonishing, surely it's obvious that people interact differently with different groups of contacts? I think the fact that Facebook didn't do this is that they are fixation in their view of the world that people should be public - which also explains their regular missteps over privacy.

Augmented-reality mood-detection goggles: "a primer in the law of unintended consequences"

Cory Doctorow — 2011-07-06T03:36:21-07:00

Sally from New Scientist writes, "The first few months after I moved to London from New York were like a Henry James novel of social horrors. Cultural differences that had seemed so subtle from across the Atlantic were staggering up close. I couldn't go five minutes without committing some disastrous social gaffe. Why did my conversations always end in mortified silence? What was I doing wrong? Why wouldn't anyone help me? Naturally, as a tech journalist, I started looking for technologies that would solve my problem. The good news is that these do exist--augmented reality applications are coming that can help you decipher the emotional cues of the people you're talking to. The bad news is that when this stuff gets rolled out in the workplace--which it now is--unintentional oversharing might the problem worse. This technology will be a primer in the law of unintended consequences."

When Picard and el Kaliouby were calibrating their prototype, they were surprised to find that the average person only managed to interpret, correctly, 54 per cent of Baron-Cohen's expressions on real, non-acted faces. This suggested to them that most people - not just those with autism - could use some help sensing the mood of people they are talking to. "People are just not that good at it," says Picard. The software, by contrast, correctly identifies 64 per cent of the expressions...
Picard says the software amplifies the cues we already volunteer, and does not extract information that a person is unwilling to share. It is certainly not a foolproof lie detector. When I interviewed Picard, I deliberately tried to look confused, and to some extent it worked. Still, it's hard to fool the machine for long. As soon as I became engaged in the conversation, my concentration broke and my true feelings revealed themselves again.

Specs that see right through you (Thanks, Sally!)

(Image: xray, a Creative Commons Attribution (2.0) image from shockadelic's photostream)

What's in a text message? The real cost of SMS and just how badly wireless carriers are ripping you off

Knowlton Thomas — 2011-07-08T08:07:03-07:00

Text messaging, or SMS, has become one of the core elements of the mobile world. Smartphones or dumbphones, prepaid or postpaid, everyone texts. You may not watch videos or take pictures or check your email—or you may do all of that plus download apps on the daily. Either way, you text.

The question is, why do you pay so damn much to do it?

SMS has been called "possibly the greatest ongoing con job of American capitalism" by William Poundstone. Sound sensationalist? Hardly. William points out in his book, Priceless: The Myth of Fair Value, that "the so-called market price of a text message has nothing to do with bandwidth or any technological reality." And he's right. Indeed, the cost of SMS is determined by how much consumers can be persuaded to pay.

In essence, wireless carriers ask you: just how badly can we rip you off?

And the answer is, quite a bit.

A text message is a package of bandwidth—AKA data. We all know that data costs money; that's why some of us stick with our dumbphones and why some of us get heart attacks when our monthly wireless bill rolls in. But when we think of data, we think of data plans; 500 megabytes for $10, two gigabytes for $25, or what have you. Text messaging, as has become fairly standard, is about $5 for unlimited SMS.

And that's the magic of it. Five bucks doesn't seem like a lot of money, and unlimited sounds like a lot of texting. Overall, it seems like a worthwhile feature and as such has become a staple in virtually all modern cellphone plans. Even discount wireless carriers such as Wind Mobile charge this flat fee for unlimited text messages. If those guys feel the price is right, it's gotta be a fair deal, right?

Wrong.

Texting your friend "Hey, what's up, Joe?" costs a wireless carrier roughly 1/1,000 of a penny. That is to say, sending roughly 1,000 text messages should cost, at wholesale pricing, about a single cent. If you send a lot of lengthier texts, it might round up to two cents. The number crunching at this point is simple: unless you're sending 500,000 text messages per month, your wireless carrier is profiting. Most people send under 1,000, and many send less than 100—all the while paying $5.00 per month.

Myriad people have pointed out the unfathomable markup of text messaging. In 2007, one suggest markup was 7,314 percent. In 2009, a markup of 4,900 was calculated. In 2010, in an article titled "America's Biggest Ripoff," the markup was believed to be around 6,500.

The trouble is that consumers don't have a clue about what a text message is really worth—or rather, how damn little a text messaging is really worth. A Simon-Kucher & Partners survey found that consumers believed a multimedia message or email containing photo(s) and/or video(s) was worth roughly 3 to 4 times what a basic text message was worth.

In reality, an MMS or email is anywhere from a thousand to a hundred thousand times more data than an SMS.

Now imagine the people coughing up as much as 20 cents for a single message on pay-as-you-go plans. They're practically throwing money at telcos. But as if this isn't ridiculous enough already, there's icing on the cake: SMS are piggybacked onto cellular networks. This means that text messages occupy, as William puts it, "otherwise unused space in a control channel used for network maintenance." So texting, most of the time, doesn't even cost your wireless carrier that fraction of a penny. It costs them, quite literally, nothing.

From the early 2000s until now, prices for individual text messages on pay-as-you-go plans have quadrupled from an average of 5 cents to 20 cents. This is obviously wildly misaligned with inflation. It is instead more in tune with the rapidly growing volume of text messages: a high-demand, ubiquitous service can be marketed to boast a perceived value that is some ways infinitely greater than its real value.

Now go text this article to your friends. 500,000 of them, if you can.

Spam is way down, but new malware is really tough

Cory Doctorow — 2011-07-02T00:23:05-07:00

Brian Krebs looks at the remarkable drop in spam that the Internet has experienced this year (25-50 billion spams/day today, down from a peak of 225 billion spams/day last July), and at the vicious new malware that's appearing as spam-crooks get more desperate. One such vector is TDSS (AKA "TLd-4"), a rootkit that infects your computer, kicks out all the other malware running on it, and then helps hackers distribute malware. Krebs says that there's plenty of gains to be realized by attacking the financial instruments used by criminals and he's promised a series on how these work.

The evolution of the TLd-4 bot is part of the cat-and-mouse game played by miscreants and those who seek to thwart their efforts. But law enforcement agencies and security experts also are evolving by sharing more information and working in concert, said Alex Lanstein, a senior security researcher at FireEye, a company that has played a key role in several coordinated botnet takedowns in the past two years.
"Takedowns can have an effect of temporarily providing relief from general badness, be it click fraud, spam, or credential theft, but lasting takedowns can only be achieved by putting criminals in silver bracelets," Lanstein said. "The Mega-D takedown, for example, was accomplished through trust relationships with registrars, but the lasting takedown was accomplished by arresting the alleged author, who is awaiting trial. In the interim, security companies are getting better and better about working with law enforcement, which is what happened with Rustock."

Where Have All the Spambots Gone?

Kilobots — Cheap Swarm Robots Out of Harvard

Soulskill — 2011-06-18T13:51:00-07:00

An anonymous reader writes with news of a research project at Harvard into controlling large swarms of small robots. This article describes what they call Kilobots. (Which, for clarity's sake, have nothing to do with killing. Yet.) Quoting: "They're fairly simple little robots about the size of a quarter that can move around on vibrating legs, blink their lights, and communicate with each other. On an individual basis, this isn't particularly impressive, but Kilobots aren't designed to be used on an individual basis. Costing a mere $14 each and buildable in about five minutes, you don't just get yourself one single Kilobot. Or ten. Or a hundred. They're designed to swarm in the thousands."

Read more of this story at Slashdot.

News: Hackers steal details of 200,000 Citi account holders. Fark: by changing the account numbers in the address bar. Ultrafark: Expert on the case says "'It would have been hard to prepare for this type of vulnerability" [Fail]

(author unknown) — 2011-06-14T09:48:57-07:00

[link] [186 comments]

How spam works, from end to end

Cory Doctorow — 2011-05-21T21:52:35-07:00

"Click Trajectories: End-to-End Analysis of the Spam Value Chain" is a scholarly research paper reporting on a well-designed study of the way that spam works, from fast-flux DNS to bulletproof hosting to payment processing to order fulfillment. The researchers scraped mountains of spam websites, ordered their pills and fake software, and subjected it all to rigorous comparison and analysis. They were looking for spam ecosystem bottlenecks, places where interdicting one or two companies could have a major impact on spam.

Figure 1 illustrates the spam value chain via a concrete example from the empirical data used in this study. On October 27th, the Grum botnet delivered an email titled VIAGRA R Official Site. The body of the mes- sage includes an image of male enhancement pharma- ceutical tablets and their associated prices (shown). The image provides a URL tag and thus when clicked directs the user's browser to resolve the associated domain name, medicshopnerx.ru. This domain was registered by REGRU-REG-RIPN (a.k.a. reg.ru) on October 18th -- it is still active as of this writing. The machine providing name service resides in China, while hosting resolves to a machine in Brazil. The user's browser initiates an HTTP request to the machine, and receives content that renders the storefront for "Pharmacy Express," a brand associated with the Mailien pharmaceutical affiliate program based in Russia.
After selecting an item to purchase and clicking on "Checkout", the storefront redirects the user to a payment portal served from payquickonline.com (this time serving content via an IP address in Turkey), which accepts the user's shipping, email contact, and payment information, and provides an order confirmation number. Subsequent email confirms the order, provides an EMS tracking number, and includes a contact email for customer questions. The bank that issued the user's credit card transfers money to the acquiring bank, in this case the Azerigazbank Joint-Stock Investment Bank in Baku, Azerbaijan (BIN 404610). Ten days later the product arrives, blister-packaged, in a cushioned white envelope with postal markings indicating a supplier named PPW based in Chennai, India as its originator.

Click Trajectories: End-to-End Analysis of the Spam Value Chain (PDF) (via MeFi)

Google banks on Canada long-term with launch of major Kitchener-Waterloo hub

Knowlton Thomas — 2011-05-19T09:00:41-07:00

Google is banking on Canada's long-term business relevance with the official launch of its enormous 34,000 square-foot office, a thorough revamping of its originally puny Canadian headquarter.

The hub, perhaps now the most major one in the world outside of Google's home base in the Valley, sits in the Kitchener-Waterloo area, not far from Research in Motion.

Quoth The Globe and Mail:

Outside the company's global headquarters in Mountain View, Calif., few offices have proved as vital to Google as the Kitchener-Waterloo location. Initially, Google's presence in the region was minimal – the result of the acquisition of a mobile technology company in the area. As Stuart Feldman, Google's vice-president of engineering for the east coast put it, Google's first office had 22 employees and 20 chairs. Today, 170 engineers work at Google's new office, making it the largest in Canada.

The company hosted a gala to unveil its new offices. The event drew MPs, Kitchener city council, and myriad engineers.

The Arduino Project Gets a Core Memory Accessory

timothy — 2011-05-14T06:31:00-07:00

Stoobalou writes "A pair of mathematicians have created an electronics project that nostalgic computer buffs will likely recognize straight away: a magnetic-core memory shield for the Arduino electronics prototyping platform." The creators' web site has more, including schematics, if you'd like to make your own.

Read more of this story at Slashdot.

Telehack Re-Creates the Internet of 25 Years Ago

timothy — 2011-05-13T19:43:00-07:00

saccade.com writes "Telehack.com has meticulously re-created the Internet as it appeared to a command line user over a quarter century ago. Drawing on material from Jason Scott's TextFiles.com, the text-only world of the 1980s appears right in your browser. If you want to show somebody what the Arpanet looked like (you didn't call it the "Internet" until the late '80s) this is it. Using the 'finger' command and seeing familiar names from decades ago (some, sadly, ghosts now) sends a chill down your spine."

Read more of this story at Slashdot.