O'Reilly Radar - Insight, analysis, and research about emerging technologies

Four short links: 18 June 2013

2013-06-17T07:15:52Z

Our Backbone Stack (Pamela Fox) — fascinating glimpse into the tech used and why.
Automating Card Games Using OpenCV and Python — My vision for an automated version of the game was simple. Players sit across a table on which the cards are laid out. My program would take a picture of the cards and recognize them. It would then generate valid expression that yielded 24, and then project the answer on to the table.
Ray Ozzie on PRISM — posted on Hacker News (!). In particular, in this world where “SaaS” and “software eats everything” and “cloud computing” and “big data” are inevitable and already pervasive, it pains me to see how 3rd Party Doctrine may now already be being leveraged to effectively gut the intent of U.S. citizens’ Fourth Amendment rights. Don’t we need a common-sense refresh to the wording of our laws and potentially our constitution as it pertains to how we now rely upon 3rd parties? It makes zero sense in a “services age” where granting third parties limited rights to our private information is so basic and fundamental to how we think, work, conduct and enjoy life. (via Alex Dong)
Larry Brilliant’s Commencement Speech (HufPo) — speaking to med grads, he’s full of purpose and vision and meaning for their lives. His story is amazing. I wish more CS grads were inspired to work on stuff that matters, and cautioned about adding their great minds to the legion trying to solve the problem of connecting you with brands you love.

Why We Started the Velocity Conference

2013-06-18T18:59:00Z

Back in 2006, Debra Chrapaty, then VP of Operations for Windows Live (later CIO at Zynga, and now CEO of Nirvanix) made a prescient comment to me: “In the future, being a developer on someone’s platform will mean being hosted on their infrastructure.” As it often turns out, things don’t work out quite as planned. A few months later, Amazon announced EC2, and it was Amazon, not Microsoft, that became the platform whose infrastructure startups chose to host their applications on. But Debra certainly nailed the big idea!

I wrote a blog post about that conversation, entitled Operations: The New Secret Sauce, which included the statement “Operations used to be thought of as boring. It’s now ground zero in the computing wars.” Jesse Robbins, then “Master of Disaster” at Amazon and later co-founder and CEO of Opscode, told me that everyone in operations at Amazon printed out that blog post and posted it in their cubicles. Operations had been a relatively low-status job. Jesse told me that was the first time anyone had made a strong public statement about how important it was becoming.

As a result of that post, Jesse, Steve Souders, and a group of others came to me the following year and said “We need a gathering place for our tribe.” That gathering place became the Velocity Conference, now in its sixth year. We chose to include not just web operations, but also web performance and the emerging field of “DevOps” – the development model for applications hosted in the cloud.

This seems to be part of the secret sauce of some of our most successful events: the recognition that it’s not just about technology but the people who put it into practice. At the heart of conferences like Velocity and Strata are new job descriptions, new skills, and new opportunities to grow careers and companies. That’s also why we increasingly think of these events not as conferences but as gathering places for communities. Technology matters. The people who put it into practice matter more.

The Velocity Conference starts tomorrow in Santa Clara. There is still time to attend.

Four short links: 17 June 2013

2013-06-17T07:09:19Z

Weekend Reads on Deep Learning (Alex Dong) — an article and two videos unpacking “deep learning” such as multilayer neural networks.
The Internet of Actual Things — “I have 10 reliable activations remaining,” your bulb will report via some ridiculous light-bulbs app on your phone. “Now just nine. Remember me when I’m gone.” (via Andy Baio)
Announcing the Mozilla Science Lab (Kaitlin Thaney) — We also want to find ways of supporting and innovating with the research community – building bridges between projects, running experiments of our own, and building community. We have an initial idea of where to start, but want to start an open dialogue to figure out together how to best do that, and where we can be of most value..
NAND to Tetris — The site contains all the software tools and project materials necessary to build a general-purpose computer system from the ground up. We also provide a set of lectures designed to support a typical course on the subject. (via Hacker News)

Networked Things?

2013-06-14T15:46:19Z

Well over a decade ago, Bill Joy was mocked for talking about a future that included network-enabled refrigerators. That was both unfair and unproductive, and since then, I’ve been interested in a related game: take the most unlikely household product you can and figure out what you could do if it were network-enabled. That might have been a futuristic exercise in 1998, but the future is here. Now. And there are few reasons we couldn’t have had that future back then, if we’d have the vision.

So, what are some of the devices that could be Internet-enabled, and what would that mean? We’re already familiar with the Nest; who would have thought even five years ago that we’d have Internet-enabled thermostats?

From the thermostat, it’s an easy jump to the furnace. A furnace may be the dullest appliance in the house. Except when it breaks. The first November after we bought our house, the furnace broke down. The furnace guy came and fixed it. And fixed it again a few days later. And fixed it again a few days later … seven times during the month of November, until he finally said, “I have a bad feeling about this,” and replaced the part he had replaced on the first service call, which turned out to be defective in exactly the same way as the original.

Fortunately, we had a service contract and got most of a new furnace out of the deal. But what impressed me was that I didn’t really appreciate waking up at 2 a.m. wondering why it was cold, noticing that the furnace wasn’t running, calling the answering service, and waiting for the guy to get here. And I wondered why the furnace couldn’t have a little Internet-connected controller that would notice it wasn’t working and send the repair service a message saying “fix me” — bonus points for sending diagnostic information so the technician could make sure he had the right part. Then the tech could call me and say, “We’ll be at your house in 15 minutes; just unlock the door and go back to bed.” That’s a good reason for the phone to ring at 2 a.m.

So that’s one. My next Internet-enabled device fantasy is our water system. Like many residents of formerly rural suburbs, I have a private well. And one of the things I dread is groundwater pollution: every few years, you read about an area where the well water goes bad because someone dumped pollutants (dry cleaning fluid, used motor oil, you name it) that eventually made it to the water table. The polluter has frequently gone out of business, so there’s nothing you can do but try to persuade the water company to extend service to your area. Wouldn’t it be worthwhile if the state were flooded with water sensors (one on every well) that would allow you to trace these events and predict their spread? Wouldn’t it be great if the water company could start running the pipes before your tap water started smelling like tetrachloroethylene? That’s easily within modern sensor network technology. A daily update to a central database from each of a few hundred thousand homes isn’t a big deal. I bet the sensors themselves would be quite simple, and I’m sure the local water companies already have sensors (and networks) on their wells.

Let’s move downstream from the well to more familiar appliances. What about dishwashers? At Foo camp last year, Jason Huggins (@hugs), founder of Sauce Labs and BitBeam, said, “Robotics is always in the future. Every once in a while, a piece of it breaks off and becomes part of the present. Then we get used to having it around, and it stops being robotics. But someone who lived in the 40s would be amazed by a modern dishwasher.” So, how smart could a dishwasher really be? Loading and unloading sounds like a hard problem. But for those occasions when you’ve just left home and realized that you’ve forgotten to start the dishwasher, being able to start it remotely would be useful. Being able to sense what was in it and adjust the cycle accordingly would also be useful. An Internet-enabled dishwasher could also access a rate API from the utility company and run itself at times when power is least expensive.

Bread makers: what if a bread maker or ice cream maker could download recipes from the web? (There is an XML-based microformat for recipes.) Then making bread would be as simple as loading up the machine with ingredients; it would take care of everything else by itself, based on a recipe you’ve selected from an online library. For that matter, what about your stove and oven? Why get up to turn off the oven when the timer goes off — why can’t it know what you’re cooking and turn itself off at the right time? There’s also a recipe standard for beer (BeerML); I can imagine fully automated brewing. I’ve seen a fully automated Internet-enabled still (at a properly licensed distillery).

What makes these ideas interesting isn’t the intelligence, it’s the connectivity. Furnaces newer than mine have microprocessor-based controllers that probably display error codes. I’m sure that modern dishwashers have microprocessors to regulate temperature and water usage; and while bread makers went out of fashion a while ago, my rather elderly microwave has a “sensor” mode that will automatically turn itself off when it thinks the food is cooked. I suspect it would be hard to find any modern device that doesn’t have at least one microprocessor. The magic starts when you add the ability to communicate over a network. Given that these devices already have processors, adding a simple network connection is trivial.

None of these examples are staggeringly inventive. A dishwasher that can be turned on from remote locations and optimize energy consumption is almost dull. So is the Nest thermostat, if you think about what it does rather than what it means. A thermostat is supposed to be dull, and the smarter a thermostat is, the more you can ignore it. The Nest is most certainly a part of the future that has broken off and now belongs to the present. So, here’s the challenge: rather than think about what can’t be done or how silly it would be to have a networked refrigerator, let’s take what we already have and think about what we might be able to do if our things were part of an Internet of Things. What “Things” would you Internet-enable?

Radar podcast: the Internet of Things, PRISM, and defense technology that goes civilian

2013-06-14T13:49:39Z

On this week’s podcast, Jim Stogdill, Roger Magoulas and I talk about things that have been on our minds lately: the NSA’s surveillance programs, what defense contractors will do with their technology as defense budgets dry up, and a Californian who isn’t doing what you think he’s doing with hydroponics.

The odd ad in The Economist that caught Jon’s attention, from Dassault Systemes. Does this suggest that contractors, contemplating years of American and European austerity, are looking for ways to market defense technologies to the civilian world?

Because we’re friendly Web stewards, we provide links to the more obscure things that we talk about in our podcasts. Here they are.

As the Vietnam War wound down, Boeing dabbled in both futuristic and not-so-futuristic public transit systems.
In his farewell address, Dwight Eisenhower anticipated the rise of a military-industrial complex–a permanent, infrastructural presence for military contractors. In his “Chance for Peace” address, also very moving, Eisenhower enumerated the costs of military preparedness: “Every gun that is made, every warship launched, every rocket fired signifies, in the final sense, a theft from those who hunger and are not fed, those who are cold and are not clothed. This world in arms is not spending money alone. It is spending the sweat of its laborers, the genius of its scientists, the hopes of its children.”
The Guardian interviewed Edward Snowden on video in Hong Kong, where he has fled to avoid prosecution for leaking the NSA PowerPoint deck that has caused a firestorm.
A programmer has linked Arduinos with hydroponics to optimize growth patterns.
For an overview of how software and industry might come together, take a look at my research report on the industrial internet, including a bit of background on Sight Machine, which makes quality control software for factories. (Full disclosure: O’Reilly’s sister firm, O’Reilly Alpha Tech Ventures, has become an investor in Sight Machine since I wrote the report.)

If you enjoyed this podcast, be sure to subscribe on iTunes, on SoundCloud, or directly through our podcast RSS feed.

Four short links: 14 June 2014

2013-06-14T11:02:46Z

How Geeks Opened up the UK Government (Guardian) — excellent video introduction to how the UK is transforming its civil service to digital delivery. Most powerful moment for me was scrolling through various depts’ web sites and seeing consistent visual design.
Tools for Working Remotely — Braid’s set of tools (Trello, Hackpad, Slingshot, etc.) for remote software teams.
Git Push to Deploy on Google App Engine — Enabling this feature will create a remote Git repository for your application’s source code. Pushing your application’s source code to this repository will simultaneously archive the latest the version of the code and deploy it to the App Engine platform.
Amazon’s 3D Printer Store — printers and supplies. Deeply underwhelming moment of it arriving on the mainstream.

Four short links: 13 June 2013

2013-06-13T05:12:12Z

The Unengageables (Dan Meyer) — They signed their “didactic contract” years and years ago. They signed it. Their math teachers signed it. The agreement says that the teacher comes into class, tells them what they’re going to learn, and shows them three examples of it. In return, the students take what their teacher showed them and reproduce it twenty times before leaving class. Then they go home with an assignment to reproduce it twenty more times. Then here you come, Ms. I-Just-Got-Back-From-A-Workshop, and you want to change the agreement? Yeah, you’ll hear from their attorney. Applies to management as much as to teaching.
Fixing Signin — The general principle can be stated simply, in two parts: first, give users a trust-worthy way to identify themselves. Second, do so with as little information as possible, because users don’t want to (and simply can’t) remember things like passwords in a secure way. (via Tim Bray)
Retro Gaming with Raspberry Pi (Adafruit) — finally, a clear incentive for kids to work through the frustration of setting up their own Linux box.
Mieko Haire — Apple’s fictious demo lady. Or is she fictitious? This is a new aesthetic-esque glitch, but while most glitches are glitches because you see something that doesn’t exist, this is glitchy because the fictions are actual people. Ok, maybe I need to lay off the peyote.

Four short links: 12 June 2013

2013-06-12T07:16:32Z

geogit — opengeo project exploring the use of distributed management of spatial data. [...] adapts [git's] core concepts to handle versioning of geospatial data. Shapefiles, PostGIS or SpatiaLite data stored in a change-tracking repository, with all the fun gut features for branching history, merging, remote/local repos, etc. BSD-licensed. First sound attempt at open source data management.
Introducing Loupe — Etsy’s monitoring stack. It consists of two parts: Skyline and Oculus. We first use Skyline to detect anomalous metrics. Then, we search for that metric in Oculus, to see if any other metrics look similar. At that point, we can make an informed diagnosis and hopefully fix the problem.
Bluetooth-Controlled Robotic Cockroach (Kickstarter) — ’nuff said. (via BoingBoing)
Nature Sounds of New Zealand — if all the surveillance roboroach anomaly detection drone printing stories get to you, put this on headphones and recharge. (caution: contains nature)

Four short links: 11 June 2013

2013-06-11T12:05:37Z

For Example — amazing discussion of 3D visualization techniques, full of examples using the D3.js library and bl.ocks.org example gist system. Gorgeous and informative.
Anti-Gravity 3D Printer — uses strands to sculpt on any surface. (via Slashdot)
How 3D Printing Will Rebuild Reality (BoingBoing) — But even though home 3D-printing has received substantial publicity of late, it is in the industrial sector where the technology will probably make its most significant near-term impact on the world both by manufacturing improved commercial products and by stimulating industry to develop next-generation fab methods and machines that could one day truly bring 3D-printing home to users in a real way.
The Emotional Side of Big Data — Personal Democracy Forum 2013 talk by Sara Critchfield, on reframing emotion as data for decision-making. (via Quartz)

Four short links: 10 June 2013

2013-06-11T01:03:23Z

Anatomy of Two Memes — comparing the spread of Gangnam Style to Harlem Shake. Memes are like currencies: you need to balance accessibility (or ‘money supply’) and inflation. Gangnam Style became globally accessible through top-down mainstream sources (High Popularity), but this gave it high social inflation so it wasn’t valuable to share (Low Shareability). However, scale sustained its long term growth. Harlem Shake was not as easily accessible because it was driven more by small communities (Low Popularity), but for the same reason, being less easily accessible, it remained highly valuable (High Shareability). Lack of scale was what made Harlem Shake growth short-term and eventually killed it prematurely. Caution: contains fauxconomics.
Handedness (Github) — determine left or right handedness from pinch gesture.
Innovation Cartography — video of a talk by Richard Jefferson of Cambia’s lens, on the imperative to innovate held at the Skoll World Forum on Social Enterprise. His story of maritime cartography (starts around 5m50s) is awesome.
Statically Recompiling NES Games into Native Executables with LLVM and Go — or “crack for Nat” as I like to translate that title.

Four short links: 7 June 2013

2013-06-07T05:26:18Z

Accumulo — NSA’s BigTable implementation, released as an Apache project.
How the Robots Lost (Business Week) — the decline of high-frequency trading profits (basically, markets worked and imbalances in speed and knowledge have been corrected). Notable for the regulators getting access to the technology that the traders had: Last fall the SEC said it would pay Tradeworx, a high-frequency trading firm, $2.5 million to use its data collection system as the basic platform for a new surveillance operation. Code-named Midas (Market Information Data Analytics System), it scours the market for data from all 13 public exchanges. Midas went live in February. The SEC can now detect anomalous situations in the market, such as a trader spamming an exchange with thousands of fake orders, before they show up on blogs like Nanex and ZeroHedge. If Midas sees something odd, Berman’s team can look at trading data on a deeper level, millisecond by millisecond.
PRISM: Surprised? (Danny O’Brien) — I really don’t agree with the people who think “We don’t have the collective will”, as though there’s some magical way things got done in the past when everyone was in accord and surprised all the time. It’s always hard work to change the world. Endless, dull hard work. Ten years later, when you’ve freed the slaves or beat the Nazis everyone is like “WHY CAN’T IT BE AS EASY TO CHANGE THIS AS THAT WAS, BACK IN THE GOOD OLD DAYS. I GUESS WE’RE ALL JUST SHEEPLE THESE DAYS.”
What We Don’t Know About Spying on Citizens is Scarier Than What We Do Know (Bruce Schneier) — The U.S. government is on a secrecy binge. It overclassifies more information than ever. And we learn, again and again, that our government regularly classifies things not because they need to be secret, but because their release would be embarrassing. Open source BigTable implementation: free. Data gathering operation around it: $20M/year. Irony in having the extent of authoritarian Big Brother government secrecy questioned just as a whistleblower’s military trial is held “off the record”: priceless.

Four short links: 6 June 2013

2013-06-06T08:41:33Z

ShareFest — peer-to-peer file sharing in the browser. Source on GitHub. (via Andy Baio)
Media for Thinking the Unthinkable (Bret Victor) — “Right now, today, we can’t see the thing, at all, that’s going to be the most important 100 years from now.” We cannot see the thing. At all. But whatever that thing is — people will have to think it. And we can, right now, today, prepare powerful ways of thinking for these people. We can build the tools that make it possible to think that thing. (via Matt Jones)
McKinsey Report on Disruptive Technologies (McKinsey) — the list: Mobile Internet; Automation of knowledge work; Internet of Things; Cloud technology; Advanced Robotics; Autonomous and near-autonomous vehicles; Next-generation genomics; Energy storage; 3D Printing; Advanced Materials; Advanced Oil and Gas exploration and recovery; Renewable energy.
The Only Public Transcript of the Bradley Manning Trial Will be Produced on a Crowd-Funded Typewriter — [t]he fact that a volunteer stenographer is providing the only comprehensive source of information about such a monumental event is pretty absurd.

Phishing in Facebook’s Pond

2013-06-05T17:15:10Z

A recent blog post inquired about the incidence of Facebook-based spear phishing: the author suddenly started receiving email that appeared to be from friends (though it wasn’t posted from their usual email addresses), making the usual kinds of offers and asking him to click on the usual links. He wondered whether this was a phenomenon and how it happened — how does a phisherman get access to your Facebook friends?

The answers are “yes, it happens” and “I don’t know, but it’s going to get worse.” Seriously, my wife’s name has been used in Facebook phishing. A while ago, several of her Facebook friends said that her email account had been hacked. I was suspicious; she only uses Gmail, and hacking Google isn’t easy, particularly with two-factor authentication. So, I asked her friends to send me the offending messages. It was obvious that they hadn’t come from my wife’s account; they were Yahoo accounts with her name but an unrecognizable email address, exactly what this blogger had seen.

How does this happen? How can a phisher discover your name and your Facebook friends? I don’t know, but Facebook is such a morass of weird and conflicting security settings that it’s impossible to know just how private or how public you are. If you’ve ever friended people you don’t know (a practice that remains entirely too common), and if you’ve ever enabled visibility to friends of friends, you have no idea who has access to your conversations.

The day I read this post, I also read about Facebook’s deal with Acxiom and other information vendors. If you know anything about Acxiom, you know that they’re one of the biggest brokers of personal data in the country. Acxiom’s data is supposedly “anonymized,” but if you know anything about data de-anonymization, and how much easier de-anonymization becomes when you have access to multiple data sources, you know that’s not much comfort. As Jeff Jonas has pointed out, given sufficient data and a few pieces to the puzzle, it’s easy to locate, say, the Turkish guy who lives near the O’Reilly employee in Connecticut. If you’ve never searched for yourself online, you should; you’ll be surprised what’s known about you.

Facebook is buying data, not selling it, and they would certainly argue that there’s no way someone reading Facebook pages could reverse-engineer the information that they’ve bought from Acxiom. I’m not so certain, particularly given Facebook’s history as a company that pushes the limits, then apologizes, and adds even more arcane security settings. It’s not as if personal information hasn’t leaked out many times over the years, going back to a surprise marriage proposal that was spoiled when Facebook told the groom’s friends that he had just bought an engagement ring. Facebook is trying to build a legit ad placement business on top of their social graph, but in doing so, have they inadvertently built the greatest asset for cybercrime that the world has ever seen?

The issue isn’t that Facebook will be phishing you themselves. It’s that your Facebook pages will be scraped, whether Facebook likes it or not, and all the data that can be extracted about you will be in the phisher’s hands. The recent phishes that I’ve seen have been primitive. It’s fairly easy to look at a message that says “Hey, Mike, lower your credit card rate” and realize that it’s spam, even if it looks like it came from one of your friends. But that’s not the end of the road. It’s not hard to craft a message that really looks like it came from a friend, and offers you something that you might genuinely be interested in. Such a message might refer to things you’ve said online or know facts that you’ve only shared with friends. At that point, it’s much harder to resist. And we’re not necessarily talking about phishes trying to sell bogus credit card services: we’re talking about attempts to get at corporate data (“Hey, Mike, who’s going to be in the 10 a.m. meeting tomorrow? I’ve forgotten. BTW, loved your Radar post on Facebook”), other personal data, passwords, etc. And any message that can be crafted by humans could, without too much work, be generated by machines.

Our future will inevitably include lots of carefully personalized, machine-generated spam; that spam might be so good that it will be indistinguishable from a message you might legitimately receive from a friend. And that’s not going to be pleasant.

Four short links: 5 June 2013

2013-06-05T11:24:41Z

OATV Fund III Pitch Deck (Slideshare) — contains a list of what they were investing in, and what they want to invest in with the new round. Then: Quantified self; Internet subsystems; Smart networks of things; Manipulation and visualization of big data; sustainability; Maker movement. Now: Quantified Self Pro; Maker Pro; Hacking Education; Hidden Economies; Operations as Competitive Advantage; A Router in Every Pocket; The Internet Operating System. The move to “Pro” interests me, too. (via Bryce Roberts)
The Network is Reliable — Many applications silently degrade when the network fails, and resulting problems may not be understood for some time—if they are understood at all. [...] much of what we know about the failure modes of real-world distributed systems is founded on guesswork and rumor. [...] In this post, we’d like to bring a few of these stories together. We believe this is a first step towards a more open and honest discussion of real-world partition behavior, and, ultimately, more robust distributed systems design.
Wisee (PDF) — recognising gestures using disturbances in the (wifi) force. Our results show that WiSee can identify and classify a set of nine gestures with an average accuracy of 94%. (via BoingBoing)
Why Your Users Hate Agile Development (IT World) — What developers see as iterative and flexible, users see as disorganized and never-ending. Here’s how some experienced developers have changed that perception. (via Slashdot)

Radar podcast: anthropology, big data, and the importance of context

2013-06-05T15:22:14Z

Jim Stogdill, Roger Magoulas and I enjoyed a widely discursive discussion last week, available as a podcast above. Roger, fresh from our Fluent conference on JavaScript, opens by talking about the emergence of JS as a heavyweight computing tool and the importance of openness in its growth. A few other links related to our discussion:

WeatherSpark.com presents weather forecasts, averages and historical data with interactive graphs and maps

WeatherSpark, a great nerdy way to see the weather forecast.
AngularJS is an open-source MVC framework for JavaScript, developed by Google.
Urban Outfitters and Anthropologie are based in a beautiful campus on the site of the Philadelphia Navy Yard.
George Dyson’s excellent Turing’s Cathedral is a history of early computing. (Related to another part of our conversation, about legacy systems, he has also written about companies that still use punch cards.)
The New Yorker on efforts to find the legendary White City in Honduras using laser rangefinders.

If you enjoyed this episode, subscribe to our podcast series on iTunes or SoundCloud.