I could have focused on the Governments Search for Google Data visualization from Chris Canipe and Madeline Farbman of the Wall Street Journal. Or, I could have focused on Neal Ungerleider’s piece that covers Eric Fisher and MapBox for Gnip’s twitter metadata visualizations.  Yet, my curiosity took over once I came across The Economist’s High Spirits graphic. Not only do I make my own bitters which qualifies me for preliminary booze nerd status, I also needed a brief break away from the transparency issues currently dominating the data-oriented conversations. Following my booze nerd curiosity led me to this interactive data visualization of common cocktail ingredients:

Analysis of 25,000 recipes from drinksnation.com and drinksmixer.com

Be careful though…it is easy to get sucked into it. I must have spent about 10 minutes before I realized that I should revisit it later.

Yet, it was a good 10 minute break.

Hat tip to Marko Plahuta of Virostatiq for highlighting this.

More visualizations:

• NYC Citi Bike
• Stravinsky’s “The Rite of Spring”
• Hospital procedure charges across the U.S., compared
• CIA rendition flights of terror suspects
• Real-time Wikipedia edits

HBase has had a reputation of being a bit difficult to use – its core users have been data engineers, not data scientists. The good news is that as HBase gets adopted by more companies, tools are being developed to open it up to more users. Let me highlight some tools that will appeal to data scientists.

SQL Interfaces
Any data store that wants to appeal to data scientists and business analysts needs to be accessible via SQL. Judging from the number of sessions² devoted to SQL, HBase users already have several options to choose from (with more to follow in the near future). Widely used inside Salesforce, Phoenix is a relatively new open source project that targets interactive analysis. Phoenix is an embedded JDBC driver, and it compiles SQL to optimized native HBase calls. Generally available since the start of May, the open source, distributed query execution engine Impala supports both HDFS and HBase.

Model development, deployment, and maintenance
I recently highlighted the growing number of open source, analytic engines, that make it easier to develop big data applications. Beyond these engines, tools for developing apps on top of Hadoop are also starting to emerge. The kiji project is a framework³ that makes big data applications easier to develop, maintain, and deploy. With kiji, developers don’t need to concern themselves with serialization, schema evolution, and other low level details. As an example, kiji-express lets data scientists encode algorithms in Scalding. Data scientists can also develop models in other frameworks and import PMML files into kiji.

While there are many frameworks for developing models and algorithms, there are far fewer tools⁴ for deploying and maintaining algorithms in “production”. Data scientists usually have to explain their models to engineers who rewrite them for production environments (a process that takes weeks/months in some companies). Just as important, once algorithms are deployed “in the wild”, they need to be carefully maintained and monitored – e.g., models that are severely underperforming need to be revisited. At this stage, the kiji project has tools for integrating model development (kiji-express) and deployment (kiji-scoring). Over time, kiji will include tools for monitoring, maintaining, and combining models.

Model deployment and maintenance are areas that many more data scientists are paying attention to. Some choose to work in a single framework (such as kiji), others piece together different tools. New workflow tools such as Chronos, are allowing business analysts to develop and maintain long, complex, data processing pipelines. I’m looking forward to seeing more tools address these critical pain points.

Related posts:

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

Pew Research Center national survey

Rebecca J. Rosen at The Atlantic took a look at legal scholar Daniel J. Solove’s argument that we should care about the government’s collection of our data, but not for the reasons one might think — the collection itself, he argues, isn’t as troubling as the fact that they’re holding the data in perpetuity and that we don’t have access to it. Rosen quotes Solove:

“The NSA program involves a massive database of information that individuals cannot access. … This kind of information processing, which forbids people’s knowledge or involvement, resembles in some ways a kind of due process problem. It is a structural problem involving the way people are treated by government institutions. Moreover, it creates a power imbalance between individuals and the government. … This issue is not about whether the information gathered is something people want to hide, but rather about the power and the structure of government.”

In a similar vein, Moxie Marlinspike at Wired tackled the response from some that the NSA’s data gathering efforts aren’t worrisome if individuals have nothing to hide. He quotes from <a href="“>James Duane, a professor at Regent Law School and former defense attorney, and from Supreme Court Justice Breyer to make the point that federal criminal laws span 50 titles of the United States Code in 27,000 pages and that the actual total number of laws and regulations isn’t exactly known, creating the problem of knowing what you may or may not need to hide. Marlinspike writes:

“If the federal government had access to every email you’ve ever written and every phone call you’ve ever made, it’s almost certain that they could find something you’ve done which violates a provision in the 27,000 pages of federal statues or 10,000 administrative regulations. You probably do have something to hide, you just don’t know it yet.”

Marlinspike also argues that having something to hide is an important part of our growth as a society—the recent legal victories in same-sex marriage and legalization of marijuana “would probably not have been possible without the ability to break the law,” he writes. Furthermore, he argues, a dystopian world where law enforcement is 100% effective opens wide avenues for abuse of power: “…if everyone’s every action were being monitored, and everyone technically violates some obscure law at some time, then punishment becomes purely selective,” Marlinspike writes. “Those in power will essentially have what they need to punish anyone they’d like, whenever they choose, as if there were no rules at all.” You can read his full piece at Wired.

Mark Jaquith writes on Medium that based on news reports of the PRISM program, we don’t actually know what the NSA is doing or how they’re gathering data, making it impossible for citizens to judge whether or not to be outraged. Jaquith notes the troubling lack of technical details in Glenn Greenwald and Ewan MacAskill’s account of the PRISM program at The Guardian and their follow-up article on Edward Snowden, who leaked the story. Jaquith points out the the authors — not Snowden — described the program as “[allowing] the agency to directly and unilaterally seize the communications off the companies’ servers.” Likewise, he notes, the Washington Post reported that “[f]rom inside a company’s data stream the NSA is capable of pulling out anything it likes.”

All of which, if true, calls for outrage. But on the other side, a report at The New York Times tells a slightly different story. Jaquith quotes from the piece:

“But instead of adding a back door to their servers, the companies were essentially asked to erect a locked mailbox and give the government the key, people briefed on the negotiations said. … The data shared in these ways, the people said, is shared after company lawyers have reviewed the FISA request according to company practice. It is not sent automatically or in bulk, and the government does not have full access to company servers.”

Jaquith points out this is “indirect and moderated” access — in direct opposition to the access Greenwald and MacAskill described as “direct and unilateral.” “The difference between these two explanations isn’t some nuanced distinction that only tech geeks should care about,” Jaquith says. “This is the difference between companies voluntarily giving the government direct and unilateral access to arbitrary customer data and companies merely complying with the law in a technically efficient way that doesn’t change the nature of the data received by the government.” He notes that the only way Greenwald and MacAskill can be correct at this point, without offering further corroborating evidence from Snowden, is if everyone — all the companies involved, the sources for The New York Times, the NSA, and the U.S. President — are lying, which he allows isn’t impossible, but stresses that the technical details do matter. “There is no aspect of this story more important,” he says, “than finding out which account is accurate.” You can read his full report at Medium.

Facebook opens an Open Compute data center

Facebook opened its first data center in Europe this week in Luleå, Sweden, housed only with its Open Compute servers. In a post announcing the launch, the company described the data center as “likely to be one of the most efficient and sustainable” centers in the world and explained that the equipment is powered by 100% renewable, locally generated hydro-electric power — power so reliable, they’ve been able to “reduce the number of backup generators required at the site by more than 70 percent.”

Jon Brodkin notes at Ars Technica that the data center’s power usage effectiveness (PUE) rating is an impressive 1.07 and that Facebook plans to post near real-time PUE data for this center, as it does for its US data centers. Brodkin reports that Facebook’s next goal is to provide companies with an alternative to Cisco and other network vendors by releasing an Open Compute design for a top-of-rack switch that will work with any networking software.

Local police now “databanking” DNA

In a post at The New York Times, Joseph Goldstein took a look at the growing DNA-gathering practices of local law enforcement agencies. Instead of waiting for state and federal agencies, local agencies are developing their own DNA databases, and their methods are causing some concern. Goldstein writes:

“These local databases operate under their own rules, providing the police much more leeway than state and federal regulations. And the police sometimes collect samples from far more than those convicted of or arrested for serious offenses — in some cases, innocent victims of crimes who do not necessarily realize their DNA will be saved for future searches.”

Barry Scheck, a co-director of the Innocence Project, told Goldstein that they’ve warned local law enforcement that the public would be “disturbed” when these “rogue, unregulated” databases came to light. Goldstein reports that DNA samples are being taken from people “on the mere suspicion of a crime” and entered into a database regardless of whether or not the subject was charged or found guilty. Samples also are gathered from people to rule them out of a crime — say, a homeowner burglary victim — but then kept on file. Goldstein notes that the Supreme Court’s recent decision in Maryland v. King was the first to address this sort of DNA “databanking” and that it could serve to accelerate the practice. “While that decision said nothing explicit about the authority of local law enforcement to keep DNA databases,” Goldstein reports, “it could well encourage local jurisdictions to push ahead, several experts said.”

Tip us off

News tips and suggestions are always welcome, so please send them along.

Related:

• The NSA is spying on us
• Court case sheds light on FBI stingray surveillance
• FBI documents “strongly suggest” they read citizens emails sans warrants
• What Facebook’s Open Compute Project means
• More Strata Week coverage

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

Transparency…or Lack Thereof

• OpenData Latinoamérica: Driving the demand side of data and scraping towards transparency (Neiman Journalism Lab)
  “There’s a saying here, and I’ll translate, because it’s very much how we work,” Miguel Paz said to me over a Skype call from Chile. “But that doesn’t mean that it’s illegal. Here, it’s ‘It’s better to ask forgiveness than to ask permission.” Paz is a veteran of the digital news business. The saying has to do with his approach to scraping public data from governments that may be slow to share it.
• The real story in the NSA scandal is the collapse of journalism (zdnet.com)
  On Thursday, June 6, the Washington Post published a bombshell of a story, alleging that nine giants of the tech industry had “knowingly participated” in a widespread program by the United States National Security Agency (NSA). One day later, with no acknowledgment except for a change in the timestamp, the Post revised the story, backing down from sensational claims it made originally. But the damage was already done.
• We are shocked, shocked… (davidsimon.com)
  Having labored as a police reporter in the days before the Patriot Act, I can assure all there has always been a stage before the wiretap, a preliminary process involving the capture, retention and analysis of raw data. It has been so for decades now in this country. The only thing new here, from a legal standpoint, is the scale on which the FBI and NSA are apparently attempting to cull anti-terrorism leads from that data. But the legal and moral principles? Same old stuff.
• Big Data Has Big Stage at Personal Democracy Forum (pbs.org)
  Engaging News Project’s Talia Stroud tackled the issue of public engagement in news organizations. Polls on websites don’t yield scientifically accurate results, nor do they get people to address difficult issues, she said. “These data are junk. We know they’re junk,” Stroud said. “City council representatives know they’re junk. Even news organizations know that the results of these data are junk. The only reason that this poll is being included on the news organization’s site is to increase interactivity and increase your time on page.”

Tools and Resources

• Latin America’s first DataBootCamp set to spark media innovation (IJNet.org)
  Not only will the upcoming DataBootCamp of Bolivia be the first event of its kind in Latin America, it will be the highest-altitude data journalism event the world has seen so far. Journalists, programmers and designers will gather in the capital, La Paz, whose altitude reaches more than 11,800 feet (3,600 meters) above sea level. But the event itself will represent just the first step of a steep yet unstoppable climb toward using open data in the region.
• AAJA-Asia and the Journalism and Media Studies Centre at The University of Hong Kong present at the New.Now.Next Media Conference | Panel 2a, Take a Byte: Data Journalism in the Era of Big Information (YouTube)
  Reuters data expert Irene Jay Liu will talk how she was involved in building Connected China, an app that visualises the complex networks of China’s leadership. Eric Ulken, assistant managing editor for digital at the Seattle Times, will show what smaller organizations can do to “dip a toe or two” into data journalism. AP’s Global Interactive Editor and AAJA National President, Paul Cheung, will tell us what we can really do with data, big or small. Moderator: Henry Williams, data journalist

Data Visualization 

• Why The Global Mail has focused in on data and investigations (journalism.co.uk)
  The not-for-profit investigative outlet launched data visualisation “Behind the Wire” today, as part of a new focus on data and ‘more editorially ambitious’ stories. Part of the evolution of Behind The Wire will rely on users actively engaging with the data, both in terms of flagging certain reports and pursuing FOI requests. Bungey said while there have been similar attempts by other news outlets to get the audience involved in looking through data, he has not previously seen a visualisation which “actually changes” based on users’ inputs, and gives the ability for them to actually contribute “graphical evidence”.
• Q&A: Data Viz Expert, Amanda Hickman (reporthers.tumblr.com)
  There have always been charts in the newspaper. What’s new is the really incredible mapping and collaborating and exploring that we’re doing online. As data visualization gets more popular in newsrooms, newsrooms are more and more likely to show their work, to publish their raw data, which I think is incredibly valuable. We expect to be able to explore the news, to question reporters. That seems like a good thing to me. The open data movement has gone hand in hand with this transformation–as more journalists do more with data it gets more appalling when a government agency tries to pass off a photograph of a data table as “information.”

Click here for the full visualization.

Users can zoom in and select a particular station to pull up detailed information on how many bikes and spaces are available. The screenshot below shows 32 bikes and 3 spaces available at Lafayette St. and Jersey St. at the time of writing.

Click here for the full visualization.

The bar across the top of the map also links to real-time bike share information in other cities around the world, including London, Toronto, Washington D.C., and Milan. A full list of all available cities can be viewed in the drop-down list in the upper right corner.

In a blog post, O’Brien explains that the data is updated automatically every two to 10 minutes, noting that data generally comes from the bike share provider’s website or their official API, but he also gathers data from third-party data collectors, such as citybik.es.

Hat tip to Susi at Velojoy for highlighting O’Brien’s work.

More visualizations:

• Stravinsky’s “The Rite of Spring”
• Hospital procedure charges across the U.S., compared
• CIA rendition flights of terror suspects
• Real-time Wikipedia edits
• Building collapse rescue efforts

Key highlights include:

• Emil explains graph databases [Discussed at 0:29]
• Facebook Graph Search is a well-known example of a graph database [Discussed at 3:28]
• But really, graph databases can be used more much more than social search [Discussed at 4:50]
• Neo4j, the original graph database [Discussed at 5:25]
• Graph databases ‘shape’ data [Discussed at 6:20]

You can view the full interview here:

This post was originally published on O’Reilly’s Programming blog.

Related:

• Subscribe to the free programming podcast through iTunes

The health privacy summit, now in its third year, provides a crucial spotlight on the worries patients and their doctors have about their data. Did you know that two out of three doctors (and probably more–this statistic cites just the ones who admit to it on a survey) have left data out of a patient’s record upon the patient’s request? I have found that the summit reveals the most sophisticated and realistic assessment of data protection in health care available, which is why I look forward to it each year. (I’m also on the planning committee for the summit.) For instance, it took a harder look than most observers at how health care would be affected by patient access to data, and the practice of sharing selected subsets of data, called segmentation.

What effect would patient access have?

An odd perceptual discontinuity exists around patient access to health records. If you go to your doctor and ask to see your records, chances are you will be turned down outright or forced to go through expensive and frustrating magical passes. One wouldn’t know that HIPAA explicitly required doctors long ago to give patients their data, or that the most recent meaningful use rules from the Department of Health and Human Services require doctors to let patients view, download, and transmit their information within four business days of its addition to the record.

At both the Datapalooza and the privacy summit, patient access was not universally acclaimed but considered a minimum requirement for improving health care. It allows the patient to find critical errors, simplifies the sharing with other doctors who are treating the patient, and helps the patient comply with medical advice. Old complaints that patients don’t care or might not understand the contents of the record no longer hold water. Law professor Barbara Evans pointed out that, “To police privacy, you have to know what’s in your records.” In fact, four days is seen as unreasonably long to withhold information, given how quickly medical conditions can become dangerous.

Read access is one thing, but write access raises its own concerns–actually, the same old concerns with a new urgency. Doctors worry that they cannot trust data from patients, and might be held liable for decisions they make based on wrong data. The question of provenance thus requires electronic records to contain fields that indicate the source of data.

Write access by patients cannot be a matter for compromise. Fitness devices and other medical advances are opening up huge new sources of data that can save lives. For instance, one can wear a device that monitors one’s heart rate or blood pressure and sends information to the medical record at regular intervals. There are probably encryption and signing techniques that can assure doctors the patient has not falsified results. The bottleneck to deploying these technologies is the medical record itself, which is rarely set up to accept such data. The doctor must also learn how to use it.

A still higher stage of patient control would take the records away from the doctors altogether and store them with the patient. This is eminently logical–why should you have to go to a dozen different providers to gather a few years’ worth of records? Separate health information exchanges–organizations that charge doctors for exchanging data and, even so, have trouble surviving financially–would evolve into more useful services. Patient control over records would also give patients some control over the dizzying array of places their data goes, often without their consent.

Skeptical doctors point out how little interest their patients (especially the ones who need to show the most interest) take in their health. Moving data to patients will start among a small, educated cohort, but could be widespread if the medical field seriously promoted it, and if a cloud solution like Microsoft HealthVault made it easy.

Cloud computing was discussed at the summit. Lie most aspects of health care, panelists insisted on more transparency there, such as knowing what company runs the underlying storage network, but their general view seemed positive.

Segmentation–still a lot of mire

Ideally, a health provider dealing with a broken bone or a cut would not be prejudiced to learn of your sexual orientation, mental health, or drug habits. But in many places, patients have to protect themselves. They also may choose to participate in research studies without releasing certain sensitive information.

Therefore, one of phantoms being chased by privacy advocates is data segmentation, the ability of a patient to share data with certain people while withholding it from others. Segmentation even made it into a major report by the President’s Council of Advisors on Science and Technology on health data.

Scads of problems are raised by segmentation, not so much technical as procedural, with some policy thrown in.

For instance, patients have trouble telling what might reveal a condition they want hidden. If an HIV patient reveals the list of medications he’s taking, a knowledgeable outsider can easily tell he’s HIV-positive. Even a note in the record that a patient is sensitive to some medication can reveal the condition that makes him sensitive. Free text, which is a large part of medical records, is difficult to handle in segmentation.

Although we don’t seem to have corralled this beast yet, many advances were described at the summit. Pilot systems can tag data with the patient’s preferences. When requests for data come in, these systems hide any data that the patient has restricted and that the requester is not supposed to see. One such system analyzes the record to remove all information that could inadvertently reveal the hidden diagnosis. When there’s a sensitive condition, the system usually omits all free text.

As many people pointed out at the summit, segmentation is an imperfect solution to the stigma attached to many medical conditions. Different diseases have been frowned on at different stages in history, and public education has alleviated some of the shame. Such education is needed so that people don’t have to be afraid about sharing their mental health, or other such conditions.

Quentin Hardy’s recent post in the Bits blog of The New York Times touched on the gap between representation and reality that is a core element of practically every human enterprise. His post is titled “Why Big Data is Not Truth,” and I recommend it for anyone who feels like joining the phony argument over whether “big data” represents reality better than traditional data.

In a nutshell, this “us” versus “them” approach is like trying to poke a fight between oil painters and water colorists. Neither oil painting nor water colors are “truth”; both are forms of representation. And here’s the important part: Representation is exactly that — a representation or interpretation of someone’s perceived reality. Pitting “big data” against traditional data is like asking you if Rembrandt is more “real” than Gainsborough. Both of them are artists and both painted representations of the world they perceived around them.

The problem with false arguments like the one posed by Hardy is that they obscure the value of data — traditional data and big data — and the impact of data on our culture. I’m now working my way through “Raw Data” is an Oxymoron, an anthology of short essays about data. I recommend it for anyone who is seriously interested in thinking about the many ways in which data has influenced (and continues influencing) our lives. I especially recommend “facts and FACTS: Abolitionists’ Database Innovations,” by Ellen Gruber Garvey. As its title suggests, the essay focuses on what proves to be an absolutely fascinating period of U.S. history in which the anti-slavery movement harvested data from real advertisements in Southern newspapers to paint a vivid and believable picture of the routine horrors inflicted by the slave system on real human beings.

That 19th century use of data mining built support for the anti-slavery movement, both in the U.S. and in England. The data played a key role in making the case for abolishing slavery — even though it required the bloodiest war in U.S. history to make abolition a fact.

Data itself has no quality. It’s what you do with it that counts.

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

Another collection of open source, Hadoop-compatible analytic engines, the Berkeley Data Analytics Stack (BDAS), is being built just across the San Francisco Bay. Starting with a batch-processing framework that’s faster than MapReduce (Spark), it now includes interactive SQL (Shark), and real-time analytics (Spark Streaming). Sometime this summer, frameworks for machine-learning (MLbase) and graph analytics (GraphX) will be released. A cluster manager (Mesos) and an in-memory file system (Tachyon) allow users of other analytic frameworks to leverage the BDAS platform. (The Python data community is looking at Tachyon closely.)

Next up: Applications
Many developers aren’t familiar with the intricacies of deploying, managing¹, and tuning distributed systems. The good news is that as the infrastructure gets simpler, companies can start focusing on building interesting applications. I’m starting to hear of many more researchers and startups building interesting solutions on top of one of these integrated platforms (BDAS, Cloudera, and other Hadoop distributions).

One can create Big Data applications by cobbling together different (“best-of-breed”) systems, but it’s usually² much easier to use engines built on top of the same platform. There’s a tradeoff: in many cases it’s hard³ to beat highly optimized/targeted solutions. It’s easier to use an integrated stack, but you likely have to sacrifice a little bit of performance in exchange. I suspect that in many use cases, the performance of integrated platforms will be “good enough”, and convenience will trump performance. Over time, analytic engines built on top of BDAS and Hadoop will improve and the performance gap will narrow further.

A good place to learn more about interesting Big Data applications (and how they’re built), is at the combined Hadoop World + Strata conference in NYC this October.

Related posts:

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.

Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England

There has been much coverage recently of the Internet of Things, connecting everything from washers and dryers to thermostats to cars to the Internet. Wearable sensors — things like FitBit and health-care-related sensors that can be printed onto fabric or even onto human skin — are also in the spotlight.

Kevin Fitchard reports at GigaOm that researchers at CEA-Leti and three French universities believe these areas are not mutually exclusive and have launched a project around wireless body area networks called CORMORAN. The group believes that one day soon our bodies will be constantly connected to the Internet via sensors and transmitters that “can be used to form cooperative ad hoc networks that could be used for group indoor navigation, crowd-motion capture, health monitoring on a massive scale and especially collaborative communications,” Fitchard writes. He takes a look at some of the benefits and potential applications of such a collaborative network — location-based services would be able to direct users to proper gates or trains in busy airports and train stations, for instance — and some of the pitfalls, such as potential security and privacy issues. You can read his full report at GigaOm.

In related news, wearable sensors — and even our bodies — may not only be used to connect us to a network, but also to identify us as well. Casey Johnston at Ars Technica reports this week on two password protection systems that Motorola presented at All Things Digital’s D11 conference: electronic tattoos and consumable pills that would replace the security tokens from the two-factor authenticator.

The tattoos, Johnston reports, aren’t branded onto skin, but are more like flexible stickers, “islands of high-performance silicon connected by accordion-like structures,” that move with a person’s body and remain adhered to the skin. Johnston does point out, though, that the long-term plan likely is to embed the silicon and wires into the skin “to make the user a proper bionic human.”

The pills are just that — a pill that the user swallows that “turns one’s entire body into an authenticator,” Johnston writes. Regina Dugan, senior vice president of the Advanced Technology and Projects group at Motorola Mobility explained that the pills use stomach acid to power a switch to “[create] an 18-bit EKG-like symbol in your body, and your body becomes the authenticator.” You can read Johnston’s full report at Ars Technica.

The NSA is spying on us

Headline news this week surrounded the U.S. National Security Agency. First, Glenn Greenwald posted a report at The Guardian publishing a leaked copy of a court order that “requires Verizon on an ‘ongoing, daily basis’ to give the NSA information on all telephone calls in its systems, both within the US and between the US and other countries.”

The order covers “all call detail records or ‘telephony metadata,’” Greenwald reports, including telephone numbers, location data, call duration and time of call, and unique identifiers — but it doesn’t cover conversation content. Because the information is classified as “metadata,” or transactional information, as opposed to “communications,” it does not require a warrant, Greenwald explains.

Jane Mayer at the New Yorker took a close look at this “metadata” to see just how bad it could be. Susan Landau, a mathematician and former Sun Microsystems engineer, told her that people don’t understand — “[metadata] is more intrusive than content.” With the amount of information that can be gathered, she explained, “you know exactly what is happening — you don’t need the content.” You can read Mayer’s in-depth report at the New Yorker.

Megan Garber at The Atlantic put together a FAQ on the situation, covering exactly what data is being gathered, who has access to it, what they’re doing with it (“as far as we know”), why it doesn’t violate the Fourth Amendment, if the surveillance is limited to Verizon customers, and much more. You can read the FAQ at The Atlantic.

An additional NSA surveillance program came to light this week as well. Glenn Greenwald and Ewen MacAskill at The Guardian report they received a 41-slide PowerPoint presentation describing a program called PRISM that allows the NSA direct access to the servers at nine U.S. Internet companies: Microsoft, Yahoo, Google Facebook, PalTalk, YouTube, Skype, AOL and Apple, and according to Greenwald’s and MacAskill’s report, “[t]he program is continuing to expand, with other providers due to come online.” According to their report Dropbox is “described as ‘coming soon.’”Harrison Weber

Victor Luckerson at Time broke down the program particulars, noting that the program was established in 2007 with Microsoft and that the program allows NSA officials direct access to 10 types of data: email messages, instant messages, videos, photos, stored data, voice chats, file transfers, video conferences, log-in times and profile details on social networks.

In an in-depth report at the Washington Post, Barton Gellman and Laura Poitras quote from a statement issued by Director of National Intelligence James R. Clapper: “information collected under this program is among the most important and valuable foreign intelligence information we collect, and is used to protect our nation from a wide variety of threats. The unauthorized disclosure of information about this important and entirely legal program is reprehensible and risks important protections for the security of Americans.” The Washington Post also published a selection of the slides with annotations.

Philip Rucker, Sean Sullivan and and Aaron Blake report at the Washington Post that President Obama defended the program, saying that such programs “make a difference in our capacity to anticipate and prevent possible terrorist activity” and that “[y]ou can’t have 100 percent security and also then have 100 percent privacy and zero inconvenience.”

Harrison Weber at The Next Web notes (as do the reports at the Washington Post and The Guardian) that the major Internet companies named as participants have denied involvement in and knowledge of the PRISM program. Weber points out that the NSA wouldn’t have given the Internet companies the program’s name, and that “those who receive national security letters are prevented from discussing their existence by law.”

Temporal cloaks hide telecommunication data

Zeeya Merali reports at Scientific American that ultrasecure communications may soon be at hand via a temporal cloak. “Electrical engineers have used lasers to create a cloak that can hide communications in a ‘time hole’,” Merali writes, “so that it seems as if they were never sent.” The first working temporal cloak was built last year by a team at Cornell, but the cloaking windows opened too rarely to hide data at telecommunication speeds. Merali describes how Joseph Lukens, an electrical engineer at Purdue University and author of this latest study, solved the problem:

“To speed up the cloaking rate, Lukens and his colleagues exploited a wave phenomenon that was first discovered by British inventor Henry Fox Talbot in 1836. When a light wave passes through a series of parallel slits called a diffraction grating, it splits apart. The rays emanating from the slits combine on the other side to create an intricate interference pattern of peaks and troughs. Talbot discovered that this pattern repeats at regular intervals, creating what is now known as a Talbot carpet. There is also a temporal version of this effect in which you manipulate light over time to generate regular periods with zero light intensity, says Lukens. Data can be then be hidden in these holes in time.”

When testing the cloak, the research team determined the cloak was able to hide data added at a rate of 12.7 gigabits per second. You can read more about the research and watch a video showing how it works at Scientific American. You can also access the team’s published paper at Nature.

Tip us off

News tips and suggestions are always welcome, so please send them along.

Related:

• The power of the Internet, wielded by machines and things
• Industrial Internet, the machines are talking (Free Radar Report)
• Court case sheds light on FBI stingray surveillance
• Data brokers know more about us than we know
• More Strata Week coverage

O’Reilly Strata Conference — Strata brings together the leading minds in data science and big data — decision makers and practitioners driving the future of their businesses and technologies. Get the skills, tools, and strategies you need to make data work.Strata Rx Health Data Conference: September 25-27 | Boston, MA
Strata + Hadoop World: October 28-30 | New York, NY
Strata in London: November 15-17 | London, England