And for those who want to engage me in a project to improve the quality of their own content — I specialize in writing about data security, privacy, workplace sociology  & science-y topics — please drop a line to me at Metaphorly.

As crazy as it all was, there was at least a small part of these pipe dreams that was based on reality. That was not the case with Theranos’s Edison gadget. It was known by the medical science community that Holmes was a fraud years before the John Carreyrou story broke the story in the WSJ. In reading some of his reporting and watching the enlightening Inventor: Out for Blood in Silicon Valley, I saw Theranos as the complete realization of something Dan Lyons wrote in his startup memoir Disrupted. And that is startups are really about making movies and the investors and VCs are the producers.

With Theranos, they skipped the part about the product actually working and went full Hollywood. Read the piece and learn how animatronic Barbie is the perfect actress for fleecing investors and fooling the public.

Image credit: SAWS

Yeah, NASA is taking us back to the moon. And Bezos, Musk, and Branson are trying to turn space travel into ,yawn, a commercial service that will ferry things and people up to space and back — like a rocket-based Mack truck. My take on all this is that the early pioneers, like the amazing Russian amateur rocket scientist, Konstantin Tsiolkovsky, and the incredible engineers at NASA  are not well known but we’re all living off their amazing work. That means you Mr. Musk: you’re standing on the shoulders of giants.

The other point in the article is that getting into space still ain’t easy, and that living out there even less so.  Do I believe Mr. Mush when he predicts they’ll be a huge self-sustaining city on Mars by 2050?  It makes for good PR and brand boosting but the whole project is nuts. Back to reality: NASA is hoping to set up a tiny space basecamp on the moon in the next few years as part of its Project Artemis. I’l settle for a few astronauts able to live on the moon for a week or two as a major accomplishment. Bonus points if they can eventually mine lunar ice to make their own water and O2!

One of my first pieces was on the secret history of search engines, which as I learned stretches back thousands of years. I’ve written about some of the  techy aspects of ranking algorithms on my Medium account. In my research for this Magellan article,  I was shocked to discover that the Googly idea of indexing massive amounts of data was born about the time of …  the Library of Alexandria (circa 4th century BC).

In the article, you’ll learn about the ancient world’s great data scientist, Callamachus of Cyrene — the Larry Page of his day. Then there’s a fascinating history that carries this indexing concept forward to the present with appearances from Aristotle, the Dewey Decimal System. and HG Wells. And for the wonks out there, Page and Brin got the idea of scanning and indexing university library books directly from our greatest sci-fi writer ever,  Mr. Wells and his World Brain project.

Rather than bore with you the screed about how the .1% are immiserating workers, I wrote this very short sci-fi story called I Love My Boss. In the very near future, we’ll all become high-productive knowledge workers who won’t be able to afford the basics. Enjoy!

In this case it’s cellular carriers and ISPs, who already know a lot about you thanks to some deregulation work of the current regime. Yeah, Comcast, Verizon, and the rest are allowed to scoop up our web browsing histories. With “Actionable Intelligence” — sounds like a CIA-NSA program with the consumer being the enemy—  Facebook shares more details about the person using the phone to the digital Draculas running our telecom companies. Sure Facebook can kind of claim, cough, it doesn’t sell their users personal data: Actionable Intelligence is a small sweetener that’s added to their existing advertising relationships with companies.

The more interesting point is that the Intercept references the obscure technical term homophily, which is the idea that we connect to similar people. Known to sociologists, who have studied who we “friend” as a serious academic topic, and to rest of us as “birds of a feather flock together, homophily allows carries and ISPs (and the NSA) to make good guesses about the behaviors and preferences of those in a  Facebook friend network, or in your real neighborhood, or organizations you belong to. Scary.  By the way, if you want to read a very good science fiction take on this topic, pick up a copy of Tom Purdom’s Barons of Behavior.

Once upon a time I wrote about homophily for a certain data security company, and though the post was taken down, it’s still a good read. I also wrote another piece about Subject Attribute Networks or SANs, which is the computer science technique used to study homophily. It has benign applications in corporate environments to help admins understand who’s likely to look at files and folders. Courtesy of Facebook and other information cartel members, the same idea has far more menacing consequences. I’ve dug the two posts out of the archives and republished below for your reading pleasure …

Homophily and Medadata

True or false: metadata can be used to discover important patterns and behaviors in social networks? This is as true now as it has been since the start of electronic communications: those whom you telegraphed and called, and more recently, email and “like”, can reveal important, non-trivial facts about both sender and receiver, even without analyzing the actual content exchanged. But more significantly, a single piece of metadata also provides details about the surrounding network neighborhood.

Let’s leave aside current headlines about metadata for a moment. E-retailers have understood since the first online purchase that metadata is a powerful way to cluster customers into like-minded communities. What are book and movie suggestions on Amazon and Netflix, but metadata that’s been used to slot members into virtual neighborhoods with uniform preferences?

That’s the key point: metadata of those in a virtual neighborhood are very likely to be similar.

These principles have been validated by empirical sociology research going back to at least the 1950s. The rest of us intuitively understand this as “birds of a feather flock together”: we connect up with people who have similar education, income, age, religion, ethnicity, aspirations, and on and on. So it’s not a surprise that we recreate these flocking patterns in 21^st-century cyber neighborhoods.

It’s easier to see these ideas play out in social networking services, where users choose their neighbors. Recently, Facebook started to directly exploit its metadata network with Graph Search. Based on my brief testing, it still in an early stage of development. But the intention behind Graph Search is clear: use metadata—i.e., likes and dislikes—of friends and friends of friends to provide more directed answers to queries. Nearby connections are far more predictive of users’ preferences and other attributes than those connections further down the chain.

For Facebook wonks, there’s also an informative post from their development team about how close “birds of a feather” connections — these friends make connections.

This is an issue in Facebook— by no means the only service with this loophole— wherein friends of friends can always view their friends’ “friends lists”. In other words, someone two hops away from me can see my connection to our common friend, and then surmise much about me from my neighbor. While this does encourage new connections, making friends lists so freely available in this way leaves the door wide open for stalkers and identity thieves to enter and gain information about targets.

Back to some of the recent headlines about metadata and phone calls. Compared to the deregulated Internet services, regulated voice carriers give metadata tighter privacy protections. Who would be surprised (and outraged) if our cell phone companies, in their monthly bills, published the numbers of people with similar interests or suggested local businesses to visit based on mining call records?

We all would. But this, of course, is common practice in social networks.

I have two takeaways to put all this into context. One, social network graphs combined with metadata are essentially inference engines. Two, the surprise for many of us is that metadata inference can work quite well in even old-fashioned phone systems with quaint 10-digit URLs.

Subject Attribute Networks and Big Data (Brother)

Like many others, I think of Big Data as enormous data sets that are worthy of distributed processing, say in the multi-petabyte range. A petabyte for those who need a quick refresher is over 1 million gigabytes—a warehouse full of thumb drives. Typically, organizations enter the Big Data zone by collecting transactional data from tens of millions of customers or, if you’re a social media company, by storing status posts, images, and videos from a huge subscriber base.

But there’s another way to cross the Big Data threshold, and it is right under our noses. Internal file systems for large companies can easily pass the 1 petabyte line. We chatted with an IT exec who manages a 1.5 petabyte file system, which is based only on the human-generated data from the company’s 40,000 employees. Your company’s files system may not be that large, but if it’s in the enterprise range (over 1,000 employees), you likely have multi-terabytes or more of storage space. That’s short of a petabyte but still very significant.

But Is Enterprise Search Big Data?

The Big Data problem space is somewhat fuzzy, and there’s no agreement on many of parameters. But there are other considerations for deciding whether something is Big Data—complexity of computation coupled with high performance. If you have difficult calculations or algorithms that need to be speedily carried out on a large data set, then you’re in the realm of Big Data.

What type of Big Data problem do I have in mind for internal file systems? Similar to web-based search, enterprise search lets employees query their company’s file systems, generating results—a la Google—as an ordered list based on relevancy and also honoring file permissions. The last requirement means that the search app has to figure out, unlike web search, whether users can view the search results based on the permissions (ACLs) for the contents.

And an enterprises search app should give similar lightning fast performance to the search engines of the web world, but using far less computing resources.

Taken altogether, enterprise search starts to look worthy of the Big Data classification. And in case you’re wondering, there is a metadata connection to both web and enterprise search.

Ranking the Results

If we dive a little more deeply into enterprise search, we can get a sense of its bigness and how metadata plays an important role. As in the consumer search world, the “Page 1” results that are returned from a query should be the most relevant. Formally, this is known as the ranking problem. And it was most dramatically solved by Google’s founders, who developed the PageRank algorithm. While Google has long since moved on to other ways to calculate rank, the underlying idea behind it is instructive: PageRank essentially uses core metadata, in this case, incoming links to a web page as a kind of vote.

In other words, the more popular pages—the ones that will show up higher in the rank list of those pages that match the keyword query—have more incoming links. For the wonky, the original paper by Sergey and Larry can be read here. By the way, there are other algorithms in the ranking genre, but they generally depend on this voting idea and using  link count metadata.

The Big Question: is there an equivalent to this voting metadata metaphor for enterprise search that sort file results matching the keyword query based on a popularity metric?

Social Searching in Enterprise Search

It turns out there’s a nice analogue to page link voting. One can think of the metadata related to file access—the number of users viewing or modifying a file as a proxy for popularity. As in the case with internet search, additional metadata is also an advantage in enterprise search, and we can apply similar popularity algorithms to these plain old files. With file activity metadata now in the mix, this definitely becomes a business-class Big Data challenge—the type that your boss at work would love solved.

There are lots of way to slice this problem, but awhile back I wrote about the “birds of a feather” principle. This says if we’re both attracted to the same general class of things, we’re likely to have other things in common—I will like what you like. You can also describe this as herding behavior—we’re following each other. This phenomena is exploited in social-based searching by many of the usual suspects in the social networking world—see Facebook’s Graph Search for more details.

We can do something similar for enterprise search by fine tuning the voting.  For example, supposed user A accesses a file, say “Marketing Roadmap for Product X’, which is also accessed by user B. And User B also has accessed a file “Sales Data For Product X’, which is not accessed by A. By the birds of a feather principle, you would want to allocate a bit of A’s vote to the sales file, even though it’s not directly accessed. Suppose user A was searching on some keywords that are used in the  “Sales Data For Product X” file—let’s say “metadata software”. The file would appear higher in the results because of its SAN weighting than if A and B were not “birds of a feather”. 

Briefly, SANs

No, not Storage Area Networks. I’ve just described a ranking model more formally known as a Social-Attribute Network (SAN), which takes into account two types of metadata—the users, the social part, and their relationships, along with the actual data and its relationships. PageRank, unlike a SAN, doesn’t directly account for social metadata since its ranking algorithm is based solely on data or content relationships.

There are a few great survey papers on SANs, but all roads lead back to the godfather of these models and the inventor of a ranking algorithm that preceded PageRank, Cornell University’s amazing Jon Kleinberg.

The actual computation for a SAN ranking for enterprise search—and I promise to be brief—often involves a giant table, which by the way is also used in PageRank. Think of each row as representing a file, and each column a user. The initial entry represents whether a user accesses the file—say give it a 1. The SAN algorithm is iterative and works by adjusting the votes based on following a chain of likes. Eventually you’ll get a number—technically it’s a probability, but never mind—that ranks a file’s relevance for each user. In other words, unlike PageRank, SAN provides user specific rankings.

This table is huge—perhaps several thousand users by maybe a hundred thousand files—the calculations complex, and they have to be carried out until the vote rankings converge.

We’ve come up with the imaginatively titled Varonis Coffee Series to describe four morning chats with an amazing multi-disciplinary team of experts. My colleague Cindy Ng has spent years talking to some of the more interesting people in the crazy world of data security — and you should check out her podcasts — and she has picked out the most popular of her speakers for these morning talks. I’m patting myself on my upper dorsal for discovering the Black Hills pen testers, who will also be presenting.

Here’s our amazing lineup: Allison Avery on leveraging diversity for better red-teaming  (March 12), the Black Hills gang, Brian Fehrman, Beau Bullock, and Derek Banks, on the incredible effectiveness of simple post-exploitation techniques (March 19), Dr. Gemma Gladon Clavell on the stunning revelation that good privacy ethics improves data security (March 26), and Dr. Ann Cavoukian, the godmother behind Privacy by Design (PbD), will explain that PbD is not just a good idea but will help you comply with GDPR and the new breed of data laws (April 2).

Yeah, so if you register early, you’ll get a $5 gift card to a well-known coffee chain. The virtual conferences start at 10am EST and so you can have your coffee. listen and watch these experts, have a bite of a cronut, and then ask questions during the Q&A at the end.

The enormous difference is that our grandparents at least had a sense of financial security and were entitled to a pension, whereas a new generation of entrepreneurs and bankers managed to convince artsy 20-somethings that they have creative freedom, and their stock options will allow them to retire early. And in return, they’re expected to put in the hours of a 19th century cotton mill worker, but monitored with 21st century Minority Report technology.  Just remember, they’re all having fun.

I had the chance to interview Professor Zinaida Benenson recently. She delivered this eye-opening talk at Black Hat 2016 on phishing. In short: we’re doomed! Benenson performed a clever experiment on college kids to test their susceptibility to clicking: they flunked. Her message for corporate IT security is that you’ll have to assume that employees will take the bait and malware will be running rampant.

So check out my two-part podcast with Professor Benenson and the transcript here.

I’m also becoming somewhat knowledgable on the General Data Protection Regulation (GDPR), which will reshape data security compliance in the EU. A lot of my legal education on this subject can be attributed to interviewing London-based Mintz Levin partner Sue Foster. You can listen in as well and soak up GDPR knowledge.

I didn’t, until I spoke to Adam. His new book tells the story of one such very profitable broker, IMS Health, which is based in Connecticut but has a global reach. I’ve been covering data security and privacy regulations for a few years now, and I thought I understood HIPAA, which protects our medical data,  but I learned there’s a small HIPAA loophole that IMS has exploited.

If the drug data is anonymized— stripped of identifiers — then it’s no longer covered by HIPAA and therefore doesn’t require consumer consent to release to a third-party data broker.  IMS and other brokers receives this scrubbed drug data.

Now if the data was truly anonymized it wouldn’t be of use to the drug companies. However, the prescribing doctor’s name remains in the data. Obviously, drug companies now can target doctors whose business they want based on the medications they are prescribing. This drug sales data then is quite valuable to them.

But that leaves open the possibility of drug companies using other marketing data about you to put together scary profiles. There’s been much research in the last few year showing it’s possible to re-identity the data in cases where there’s “low entropy”— e.g., a drug that’s not often prescribed—or associate the drug information to a small pool of consumers.

And there’s the very real possibility of hackers or cybercriminals using hacked medical insurance records (Anthem and the rest) along with this drug data to blackmail high-profile individuals. Adam assures me that IMS and other brokers have not been hacked, at least as far as we know about!

The interview is in two parts and well worth the 15 minutes or so it takes to listen.