Here’s a great example of how one company takes Big Data and makes it fun.  Help is a drug company that strives to simplify the pharmaceutical choices for customers.  Their website now features a map highlighting sales data from Target and Walgreens called “What’s wrong U.S.?”.  A bar chart for each state shows how many people are buying products for particular ailments versus the national average; you can also click on the state and get region by region details.  For example, Central Texas, home to our home, Austin, TX has a higher than average number of blisters.  Maybe it’s all the running and biking we do!

(via Flowing Data)

The main ingredients of the project were Ruby on Rails, MongoDB, and everyone’s favorite, Amazon Web Services. The result is an alpha stage of the tentatively named S3Chimp. It is an addition to Dashpot, our Analytics & Operations Dashboard for the Infochimps Platform. Dashpot boasts an easy-to-use analytics and operations dashboard that provides business metrics and visualization, cluster management capabilities, and system monitoring on top of the Infochimps Platform. Integrating a lightweight metadata system into the workflow makes it possible for Dashpot to also track and organize distributed massive-scale data assets. What was once time-consuming (according to us as well as various people in the industry), can now be a dynamic part of an organization’s internal analytics.

Before I could begin making S3Chimp, organizing the Infochimps Amazon S3 Buckets was key. Perhaps a company that boasts about its command of data should have a beautifully organized set of buckets? Perhaps….  But let’s pretend that is not the case. And let us imagine that a young and excited Information Studies graduate student decides to tackle the S3 clutter. The essential steps in such a scenario include designing a thought-out schema guideline tailored to the company’s needs and data types, and insensately enforcing those guidelines.

Next on the list was learning Ruby on Rails, over several weeks. It was a baptism by fire. I learned the very basics of Ruby on Rails and how to love the MVC trinity. Ruby on Rails is a smart and fun web app framework and it was an enjoyable experience, relative to PHP. Relative to a Saturday afternoon at Barton Springs? Not so much.

With a snazzy script written in the enchanted Infochimps Data Mine, I was able to take the most exciting leap which was taking metadata from the now beautifully organized S3 buckets, and injecting it into MongoDB, a NoSQL database. The result is the S3Chimp genesis. S3Chimps is a system that that tells you what data and how much of it is in AWS, all from your analytics dashboard. Future plans for this product include making a tool to capture provenance metadata, and other goodies.

You can find me at the upcoming MongoDB NYC conference, if you’d like to ask me about our awesome new Ironfan Platform, Dashpot, or my CapStone project.

I’d like to thank my Field Supervisor, Flip Kromer as well as my Faculty Adviser, Dr. Melanie Feinberg.

Keep an eye out for my next blog post where I will be chronicling my personal Ruby on Rails adventure that is near and dear to my librarian heart. Travis Dempsey and I will make an in-house database of our office library’s catalog. The Bukfin Repostiry’s catalog is currently housed in Librarything.

To crib a DevOps aphorism, however, it’s not the technology that makes Infochimps work: it’s the culture. Specifically, it’s about culture that keeps the challenges from all that novel technology manageable.

Our hiring process is a big part of building and maintaining that culture. We have multiple interview passes, to efficiently separate the few who will fit from the large mass of potential hires. The first pass is with our office manager Holly, a sweet lady who weeds out obvious mismatches in personality, interest, or resume. Next is a phone interview with Adam, our technical team lead, to help weed out those with obviously insufficient skill-sets. After that comes a team interview at the office, to do a finer test on the cultural fit, and start sniffing out where the candidate’s skills and interests are. The last hurdle before an offer is a short initial contract job (a week or two long, paid on completion); nothing demonstrates work ethic and development style clearer than actual development work. Although there are two technical passes, in all cases the focus is less on existing experience, and more on attitude and potential: even the most experienced candidate will lack experience with most of our tools, so adaptability and initiative are important traits in a successful hire.

Our management style relies heavily on what we have won in the hiring process: a work environment full of capable, intelligent, and self-motivated people. Management structure is very flat; I regularly consult with the C-level folks, and everyone else, as a particular task requires. Well-defined (but flexible) roles help keep communication open, as it’s usually obvious who should be included in a discussion. The technical leadership is focused on setting and tying together goals, leaving most choices about the implementation to those doing the work, but always available to clarify what choices align best with the bigger picture. Shared language for common pains and frustrations (e.g. spending currency as an analogy for causing developer frustration, or our various terms for types of technical debt) help encourage empathy, and a shared focus on troubleshooting over blame-assignment. Above all, management strives to avoid mandatory overhead for development (i.e. regular status reports and meetings), instead relying on each employee’s good judgment and occasional casual check-ins to decide how they communicate status and needs.

Beyond core management style, Infochimps goes to great lengths to support their employees. Developers aren’t always the best at remembering to eat in the middle of deep code delve, so lunches (and gentle reminders from Holly) are supplied, in addition to a fully stocked kitchen. There’s an employee joy fund, which employees propose and vote on uses: past choices have ranged from “a new coffee maker that doesn’t suck” to bimonthly yoga classes. There are company outings, both formal and impromptu, and some fun and games around the office too (including the occasional Magic: the Gathering free-for-all).

Employee career development is another big key to employee joy. An employee’s focus is largely self-directed, with interest trumping experience in all but the most time- or stability-sensitive projects. To paraphrase Flip, one of our founders, our aim is to make employees awesomely valuable to the open job market, and totally disinterested in it.

Our development culture is heavily agile, embracing elements from Scrum, Kanban, and DevOps without slavish adherence to any of them. Though core technology choices often come from the C-level, good ideas can and do come from anywhere, from the newest hire to the office manager. In a similar way, although operations are my core responsibility, they are not mine alone: we are closer to the ideal of DevOps (or perhaps NoOps+1 or AllOps), in that ultimately everyone shares the goal (and some of the load) of keeping everything operational. Repeatability is key to many of our core products, but we balance with an understanding that automation is best done to address boredom or terror, not just inefficiency; a task must be either be too predictable to be interesting, or too complex to be feasible, to be a good reason to add further infrastructure. We are also consciously risk-taking, preferring failure from audacity to failure from inaction, and failing forward instead of rolling back wherever possible.

Our infrastructure choices are made with similar goals in mind: developer experience and ergonomics are important criteria for tool choice. Resources are open by default (in cultural assumption, where security concerns prevent it in actuality), so that developers may get to what they need to easily. Components should ideally be small, decoupled, and late-binding wherever possible; reducing the interdependence the system improves both how manageable it is, and how flexible your architecture can be in the face of changing business needs. Making infrastructure repeatable (by making it from code, via Ironfan) means that building anew is an attractive option, which can free you from some of the worst of legacy code upgrade cycles. Archiving unused code and data from production systems, as opposed to supporting everything without question, makes the resultant systems easier to understand and trust.

So now that we’ve got this great workplace, what’s next? We foresee (and are even starting to experience) some growing pains as we shift into our enterprise focused work. How do we handle the impedance mismatch between our model and our clients’ models? What do we do as the company grows beyond the size of the monkeysphere? How should we tackle user segmentation and security as we build our Platform out?

Ultimately, the answers boil down to the same thing we have been doing: find the best teammates we can, then tear down any barriers between them and being awesome.

When you think Big Data, the first words that come to mind are often Hadoop and NoSQL, but what do these technologies actually mean for your business?  Different Big Data technologies have different use cases where they work best.  For your real-time Big Data challenges often a very different class of tools must be implemented.

In this free white paper, we’ll explore:

• How to create a flexible architecture that allows you to use the best Big Data tools and technologies for the job at hand
• Where Hadoop analysis and NoSQL databases work and where they can fall short
• How Hadoop differs from real-time analytics and stream processing approaches
• Visual representations of how real-time analytics works and real world use cases
• How to leverage the Infochimps Platform to perform real-time analytics

When it comes to predicting the future, your best resource (short of a soothsayer) is historical data.  As data collection, storage and processing has become more sophisticated, the volume of data has exploded. A recent article in the McKinsey Quarterly, states that in the US, across most business sectors, companies with more than 1000 employees store, on average, over 235 terabytes of data – more data than contained in the entirety of the US Library of Congress.

What does this mean?  It means that companies are sitting on a goldmine of insights for competitive advantage.  The McKinsey Quarterly article mentions this example:

The top marketing executive at a sizable US retailer recently found herself perplexed by the sales reports she was getting. A major competitor was steadily gaining market share across a range of profitable segments. Despite a counterpunch that combined online promotions with merchandizing improvements, her company kept losing ground.

When the executive convened a group of senior leaders to dig into the competitor’s practices, they found that the challenge ran deeper than they had imagined. The competitor had made massive investments in its ability to collect, integrate, and analyze data from each store and every sales unit and had used this ability to run myriad real-world experiments. At the same time, it had linked this information to suppliers’ databases, making it possible to adjust prices in real time, to reorder hot-selling items automatically, and to shift items from store to store easily. By constantly testing, bundling, synthesizing, and making information instantly available across the organization—from the store floor to the CFO’s office—the rival company had become a different, far nimbler type of business.

The amount of data we produce is staggering and the underlying possibilities are incredible, but that doesn’t necessarily mean companies have the ability to extract true value from their data.

Looking to understand how Big Data can revolutionize how your organization does business?  Sign up for a free Big Data consultation with some of our leading data scientists to get started today!

A recent article from The Atlantic explores how Big Data has revolutionized the dairy industry.  In the past sixty years, through innovations in dairy science, milk production from an individual dairy cow has gone up from an average 5,000 pounds of milk in a lifetime to 21,000 pounds of milk.  This astonishing increase has largely been fueled by data-driven predictions that allow dairy breeders to optimize their herds.

Dairy breeding is perfect for quantitative analysis. Pedigree records have been assiduously kept; relatively easy artificial insemination has helped centralized genetic information in a small number of key bulls since the 1960s; there are a relatively small and easily measurable number of traits — milk production, fat in the milk, protein in the milk, longevity, udder quality — that breeders want to optimize; each cow works for three or four years, which means that farmers invest thousands of dollars into each animal, so it’s worth it to get the best semen money can buy. The economics push breeders to use the genetics.

Somewhat unsurprisingly, we are also outnumbered by chickens.  In 2009, we killed 52 billion chickens for food (to say nothing of the ones we kept alive).  Kind of makes you thankful they aren’t fighting back.

Happy Friday!

by NatGeo. Browse more data visualizations.

 

The boundaries of a neighborhood can be a topic of hot contention. Look to a tourist guidebook, a real estate agent, and a local and you’ll get four about whether or not north of 14th Street still counts as “The Village” in NYC.  Livehoods, a project by the School of Computer Science at Carnegie Mellon University takes a social spin on answering these questions and uncovers some truly insightful data of neighborhood boundaries, relationships, activity levels, character, and more.

Livehoods offer a new way to conceptualize the dynamics, structure, and character of a city by analyzing the social media its residents generate. By looking at people’s checkin patterns at places across the city, we create a mapping of the different dynamic areas that comprise it. Each Livehood tells a different story of the people and places that shape it.

One thing I found particular fascinating, though not wholly unexpected about the New York City map was the clustering of neighborhoods in New Jersey.  In NYC, with the relative proximity of… everything to everything, it’s not surprising to find that neighborhoods are small areas comprised of a tightly clustered businesses and homes.  In New Jersey, the “neighborhoods” span across a half dozen suburban towns in the same county.

Interested in experimenting with some Foursquare data yourself?  Check out our Foursquare Places API!

Living in Austin, TX, it was pretty obvious that last year with its record number of 100+ degree days without rain, thousands of square miles burned in wildfires, and billions lost on agriculture that we were in the middle of a serious drought. The impact across the state and throughout much of the South since October 2010 is staggeringly reviewed in this simple flipbook-style map from NPR.

The potential solutions to the problem are outlined in the Water Plan. It will be interesting to see how the continuation of this drought will affect job growth, home prices, population, and more throughout the state in the coming years.

Various plans for dealing with future droughts and growing demand for water in Texas exist, but most comprehensive — and accepted — is the state Water Plan. It offers a frank assessment of the current landscape, saying Texas “does not and will not have enough water to meet the needs of its people, its businesses, and its agricultural enterprises.” It predicts that “if a drought affected the entire state like it did in the 1950s,” Texas could lose around $116 billion, over a million jobs, and the growing state’s population could actually shrink by 1.4 million people.