Welcome to the second part of my journey to explore the world of big data (read part one here). Though I am an experienced database developer familiar with both transactional and business intelligence systems, the world of high-volume, complex, fast-moving data blithely labeled “big” is sending me back to school. Join me in my efforts to beat back ignorance and ship cool solutions.

At this point, you may be convinced that your company has big data that is underused or undervalued; and you want to find a way to import it, organize it, and start mining it for its incalculable business value. Allow me to step in and give you a single word of advice:

Don’t.

What? Consultants are supposed to write about why you should buy what they’re selling. But let’s look at the implementation details. Jumping into the deep end of your first big data project involves quoting and procuring huge amounts of storage; choosing, buying, and acquiring skills on new data analysis tools; and almost certainly staff augmentation, in terms of new head count or outside vendors. With that many variables, there is a lot of potential downside. One or two costly mistakes could cripple the project.

For your first adventure, I would advise using the tools and storage you already have, and learning to sip from the fire hose. Just because something is possible doesn’t mean it’s feasible, especially for a team charting new territory. Here are four ways to cut big data down to size, so you can acquire skills and demonstrate business value using only a few hundred gigabytes of data, instead of petabytes.

Subset. Let’s say you have access to a support forum or social media feed, with a large and active base of users and messages. Big data nirvana would be to import the entire data store, and use natural language processing to determine preference, intent, and intensity. Who likes us and why? What do they wish we did differently? Who are the opinion leaders? This project would be amazingly cool, but daunting in its complexity.

What if you didn’t import the text? By using only a subset of the available fields, you could import data more quickly, use less storage, and perform simplified but valuable analysis. If you import the date, and the IDs of the forum category, thread, message, and user, each record would be less than 50 characters wide. You could import billions of them, and analyze message volume over time, messages per thread, who posts, and how often. You could even identify clusters of users who post together or form communities. By tying in summary data regarding marketing activities and sales, you could see how forum activity drives or reacts to external events. All using a database easily handled by your existing tools, and a few gigs of unused storage. And all without importing or parsing a word of forum text.

Segment. A data store may be too big to process in its entirety. But you could learn a lot and develop skills for future projects by dividing the data by date or geography, and processing a smaller segment. You may have data for five continents and five years. But your pilot project might be to look at Canada last year, or the U.S. last quarter. Start small with big.

Sample. Instead of importing all data for a small segment, you could import a representative sample of the whole. Locate or create a varying numeric field, and use its last two digits to select a random 1% of the data to process. Sampling or segmentation can give your executive stakeholders a taste of both the value contained in big data, and the difficulty in obtaining that value—without making them commit to a full-scale project. Lessons learned can be used to guide planning for future phases.

Summarize. In this approach, you import all the raw data (or as much as you can, using segmentation or sampling), summarize key characteristics, and store only the summary for later analysis. This approach drastically reduces the total storage space required. This might be a good phase two approach, because your team will need to deal with full-scale velocity (can you process all the data as quickly as it is being generated?), but can make compromises regarding data variety/complexity and volume.

Your first big data implementations will drive lots of learning—new concepts, vocabulary, and expectations. By starting with segments, subsets, samples, or summary data, you can use the database tools you’re already proficient in, and not have to start over on a new platform as well. Plus the reduced size of the initial data sets will give you a scale model you can use to refine your techniques.

Medium—it’s like big, only smaller. Next time we’ll look at some of those new tools. I look forward to reading your comments, which will help guide the direction of future posts.

Welcome to the Projectline Weekly Roundup. We know that the week can move pretty fast. Since Fridays sometimes offer a chance for a breather, we wanted to share links to some of the articles we liked this week. As always, we’d love to get your take, so feel free to leave a comment or chat us up on Twitter. Happy reading and have a great weekend!

Business Intelligence
Data Is Useless Without the Skills to Analyze It—I’ve highlighted a couple of these kinds of blog posts, but it is so important. Machines can only do so much with data, and humans are critical to using it to your advantage.

Social Media
Facilitating Social Business in Large Organizations—This the transcript from this week’s #mmchat Twitter chat. It is worth it to scan through and pick up some ideas.

Traditional Strategy Is Dead. Welcome to the #SocialEra—I was really drawn to the opening of this post. “When I say, “Social is and can be more than media,” people resist. It’s as if the two words (social and media) are now permanently fused together. But they shouldn’t be.”

Projectline Posts
What’s The Big Deal? The World of Big Data—Just what is “big data?” Norm Bowler from our business intelligence group, shares his perspective in this blog post.

I’m an engineer, for better or worse. I live by what I can weigh and measure. When someone tells me they have “big data” and are going to “put it in the cloud,” I put my hand on my wallet. I can’t help it—I get a fairyland feeling. To me, “too big to measure” sounds suspiciously like “too big to fail,” and both terms make me nervous. What is this stuff? My engineer brain rebels.

But there really must be such a thing as big data, because suddenly everyone is talking about it. And people are sure they need it, even if they’re not exactly sure what it is. Join me as I try to shed a little light on the latest buzzword.

How Big Is Big?
Let’s start with a practical example: Take a piece of paper and write out your name, address, phone number, and e-mail, then count the characters. (Engineers like me love this stuff; give yourself a gold star if you used graph paper.) It will probably be a little under 200 characters. Using that as a baseline, we can extrapolate: five address book records per 1000 characters (a kilobyte, or “K”); five thousand entries per million characters (a megabyte, or ”Meg”); and five million entries per billion characters (a gigabyte, or “Gig”). Multiply that by another factor of 1000, and you’ve got five billion address book entries and a trillion characters of data—a terabyte. Almost one entry for each person in the world. That’s a lot of data, but it’s not “big” data—a terabyte or two is well within the capabilities of current SQL databases.

But the next step up is a petabyte—a thousand terabytes. That’s big data, by anyone’s definition. Where and how do you store that much data? What tools do you use to analyze it? And how in the world do you make a backup copy? Now we begin to grasp the problem.

More Than Just Big
Like so many convenient labels, the term “big data” can be somewhat misleading. When we give big data a closer look, we find that more than just its size makes it special. Big data is often discussed in terms of Volume, Variety, and Velocity. We just looked at the issue of Volume above.

Variety refers to the fact that the data is not always regularly formatted, or may link together in very complex ways. Importing millions of expense entries, where each entry has a date, an amount, a category, and a description is a fairly simple task. But how do we import, store, and extract meaning from Twitter or Facebook threads? The job gets messy quickly.

The third challenge is Velocity. Not only is there more data than ever before, it is moving faster. One of the challenges of big data processing is being able to load the data as quickly as it is being generated. Having a system that is never caught up because data never stops flowing in is acceptable; but having a system that is constantly falling farther behind is not.

It might be more accurate to describe big data as “hard-to-handle data.” But that doesn’t roll off the tongue so well. The reason we pursue big data processing even though it’s difficult is that there is a huge amount of business value in these large, fast-moving, complex data stores. There are enough fish in this ocean of data to feed us all. But we need a very big net, and a boat to haul it.

The New World
In my career as a database developer, I have gone from small-business systems running on a single machine, to client-server transactional systems driving web-based e-commerce platforms, to terabyte-scale data warehouses. But the new tools and new challenges of big data make me feel like a beginner again. And perhaps that is what I resent. It’s time for me to stop being the teacher and start being a student. I have a lot to learn, and hope to share that new knowledge in future posts.

Where are you in your big data journey? Skeptic, seeker, explorer, guide, or tycoon? Hopefully not a casualty. Let me know about your successes and struggles in the comments. Your input will help guide the discussion in future posts. Read part two of this series here.

Here at Projectline, we’ve recently started a project to capture all the places we work and bring together our team around the world. Every weekday morning at 10:42 am, our team is invited to send in a picture of where they are, what they’re doing, or who they’re with. Each Monday, we’ll choose our favorite picture of the previous week and share its story here.

Bearded water-gun slinger holds BI Analyst hostage high over the city of Bellevue. Actually, just an average day in the BI outpost of our Eastside fort.

I’m a Business Intelligence Analyst. I’m also a data geek. So when a client recently asked me to oversee the development of a comprehensive Social Media report, for monthly presentation to their upper management, I was both excited and wary.

Excited Me: Cool! The latest thing! A vast untapped resource of dialog and opinion straight from customers!

Wary Me: This stuff is mostly free-form! How are we ever going to glean relevant, actionable data from these sources? It’s like the Wild West out there…

Some basic social metrics are easy to gather and interpret. How many Facebook likes did we add this month? How many unlikes? How many Tweets were made with a hashtag we might expect, like #ProductName? How many people liked our client’s latest Facebook post? It’s a good start, but our client’s management wanted more in-depth insight.

There are several tools available for social media analytics. Visible Intelligence, Radian6, Sysomos (MAP and Heartbeat), Nielsen’s BuzzMetrics, Crimson Hexagon, etc. They all offer valuable insight but go about it in very different ways. Instead of picking a tool and prying out what we could, we worked backward from our client’s requirements. We tried various tools, analyzed the results, and chose the solutions that provided the most relevant analyses for their needs. Here’s a partial list of some of the in-depth analyses we were able to generate:

Buzz Volume: Using keyword searches across Facebook, Twitter, and select blog, video, and photo sites, we came up with a “buzz” volume that we could track over time. This is good to watch for overall social interest—for example, to see the buzz from rolling out a redesigned website or a new product version.

Sentiment: Some tools are quite good at classifying posts by sentiment: positive, negative, or neutral. Tracking counts of posts by sentiment month-to-month proved to be very useful, particularly around new product version releases and press releases. For further insight we picked out specific sample posts, positive and negative, and provided links to the actual posts.

Topics by Media Type: This breaks out a percent of posts by where they occurred: Facebook, Twitter, blogs, etc. Watching shifts in these volumes proved valuable. For example, we knew that discussions in this client’s forums tended to be mostly about troubleshooting technical issues, so an increase in forum buzz proportionally to other sources could indicate an emerging technical issue.

Top Contributors: We identified the URLs of forums, URLs of blogs, and handles of Twitter accounts providing the highest number of posts in the month. This provides a nice short list for further investigation of who is “buzzing” about our client’s product the most and might be worth approaching for a more formal relationship. For example, a person who frequently blogs about usability of the product might be an excellent person to have as a beta tester.

Competitive Analysis: Why just track buzz volume for the client’s product? We track buzz volume for competitors’ products as well, then combine and track over time to see how our client stacks up. The percentage that our client’s buzz contributes to the overall buzz in their category could be termed “Share of Voice” and is a good metric to track. When our client releases a new version, does their Share of Voice go up? If not, do they need to retool their launch marketing efforts?

Website Visits by Referring Domain: Most common web traffic tracking tools can report the domain from which a visitor came. Because many of our client’s social media posts include a call-to-action to click on a link to our website, we can track referrals from twitter.com, linkedin.com, etc. to determine how much web traffic is being driven by their social campaigns.

Success Events by Referring Domain: Going one layer further with web analytics, we worked with our client to define specific “success events” we wanted to track. These were desirable actions taken once the visitor landed on the client’s website, such as watching a video, reading a whitepaper, or downloading a software trial. Analyzing these success events by referring domain allowed us to see the level of engagement of visitors from social sites. Combining Visits and Success Events, both by Referring Domain, we calculated a “Success per 100 Visits” metric-another good way to see how effective our client’s social media posts are at engaging customers.

Blog Traffic Analysis: We used standard web analysis tools to track interaction on our client’s blog. Initially, their blog showed entire articles on the main page. To enable more detailed analysis, we drove changes to their site structure so that the home page shows only the first several lines of each article with a “Read more” link. This way, each full article receives a unique URL, and we’re able to track how many in-depth reads each article is receiving.

Blog Traffic by Referring Domain: Again, it’s interesting to see how people are getting to our client’s blog. This analysis reveals some noteworthy sources of links to their blog articles, and it can also identify sites that could be approached for a more formal relationship.

In its infancy, the process of analyzing social media data can yield some very helpful insights for marketers and managers. It’ll be exciting to see how the tools evolve—no doubt providing richer and more accurate data in the future—and to discover how we can further leverage this rich source of customer feedback data to benefit our clients.

Here at Projectline, we’ve recently started a project to capture all the places we work and bring together our team around the world. Every weekday morning at 10:42 am, our team is invited to send in a picture of where they are, what they’re doing, or who they’re with. Each Friday, we’ll choose our favorite picture of the week and share its story here.

This week it was hard to choose just one favorite—we’ve seen pictures of the Inc. Leadership Conference, an adorable baby, and the art near our lovely downtown neighborhood.

But Eric’s picture from Portland was irresistible, because we’re very excited about our new partnership with Webtrends (which is based in Portland). You can now find us on their partner page, and we can’t wait to see how their analytics tools enrich our ability to help clients with customer intelligence. (Not to mention giving us an excuse to visit Portland more often!)

Projectline recently released a white paper on Master Data Management (MDM) for the healthcare industry. Although it’s packed with content, much of it may not seem obviously applicable to our more marketing-based thinking. But it does offer some great ways to think about data, and we’d like to pull out two of those points from the white paper and apply them to the difficult process of measuring return on investment in marketing.

Point 1: People, Process, and Technology:

“The Projectline approach has three core dimensions, which revolve around people, processes, and tools. While many consulting firms focus primarily on tools and processes, Projectline believes the role of people is just as important because they possess the institution’s collective knowledge and are responsible for implementing information tools and processes. These core dimensions are Data Stewardship, Data Governance, and Technology. All of them must be taken into account as an organization undertakes a project to discover its master data and to design, build, and implement a solution.”

As in healthcare, new marketing measurement initiatives often focus on process or tools, but underplay the vital role of people in getting good data that leads to valuable insights. Without full participation from marketers, the data going into the system will always be flawed—and the stories the data tells will always be untrustworthy. People need to trust the technology and understand the process. In turn, the technology needs to be created in a way that honors existing processes. And the processes need to be developed with a full understanding of the technology and how it fits into people’s work.

Point 2: Quick Wins:

“Develop effective implementation plans for ‘quick wins.’ Quick wins are particularly important because they lead functional areas to recognize the importance of MDM and accept it right away.”

When you create a long-term system to optimize marketing by measuring marketing initiatives, it can be easy to get focused on crafting a pure system or shifting ingrained behaviors and taxonomies. But to gain the confidence of teams—and executives—it is vital to define early success and make it achievable. It will be much easier to justify continued investment in a program if you can point to success within the first quarter after implementation, rather than pointing to numbers far into the future.

Here at Projectline, we write a lot about marketing. We love marketing (as you may have noticed).

But it’s not all we do. We also have a Business Intelligence (BI) team, and they feel as passionate about BI as the customer evidence team does about case studies, good content, and social media. To share some of their expertise and wisdom, we’ve just published an article about Master Data Management (MDM) for the healthcare industry, and we’re thrilled to share it with you. Here’s a brief overview:

Market and regulatory issues in the healthcare industry are driving and increasing the need for accurate and efficient flow of information, with the result that healthcare providers must ensure that their data is relevant, up-to-date, and comprehensive. We see Master Data Management, an approach to creating and maintaining consistent data, as a key IT strategy for achieving this goal.

Our article looks at how you can use Master Data Management as a key strategy for addressing these challenges and discusses the Projectline Services approach to MDM for healthcare. We apply three core dimensions—processes, tools, and especially people—to maximize “data wellness” and help make healthcare organizations more efficient and agile to thrive in a dynamic marketplace.

If this sounds like something that would be helpful to you—or someone you know—download the whole article to learn more.

A few weeks ago, I ran the Warrior Dash, a demanding obstacle race. After months of anticipation leading up to the event, I came and conquered…well, not really conquered, but the event exceeded my thrill-seeking expectations.

The one thing that I had not expected was a weekend full of discussions with my friends about the event’s numbers. With close to 90,000 fans on Facebook, Warrior Dash has sold out ten events across the country. The Dash I attended drew almost 10,000 participants in one weekend, at sixteen heats per day and 350 people per heat, and I started to wonder: How do they keep track of their success? How do they measure repeat attendees? How do they keep their unique event vision creative and still mind the numbers?

In my working life, I often think of numbers in terms of success measures. Especially now, with tightened purse strings, metrics are essential for validating customer engagement programs. Often executives ask questions like: How many references are in the program? How often do they participate in requests? What opportunity dollars are associated with the reference activity? Are there gaps in the available reference pool? And it’s up to you to supply the answers and validate your program.

As proven by the Warrior Dash, metrics don’t have to stifle your creative, thoughtful vision. Use the metrics to your benefit. While parameters should be defined, metrics shouldn’t be restrictive in the creative sense. When thinking about metrics for your organization:

• First, consider what makes your program stand out among other reference programs. By knowing the unique identifiers for your program, you can keep the program integrity intact and align your metrics to your distinctive program.
• Second, ask yourself what metrics really matter and focus your energy on achieving the relevant numbers without compromising the program vision. For instance, if your program vision is get dynamic customer references across all verticals, assign a goal for each vertical and then report on how you’re tracking against that goal.
• Third, consider automation. By automating the reporting functionality of your efforts, you can spend more time on the program itself. Having the extra time to keep current program participants engaged and enlist new references will not only help you achieve your goals, but also give you time to carry out your program vision.

When you think about success measures as a tool for protecting your program creativity, you can not only meet your program mission, you can also achieve measurable results.

To find out more about how to maintain your true program spirit and creativity and obtain the metrics you need to support your vision, visit our page on reference marketing.

They say a picture’s worth a thousand words, and it can certainly be nice to present data in visual form. But on the IT and analytical side, we are sometimes skeptical of pretty pictures. We worry they can oversimplify data, mislead viewers, or prevent the kind of manipulation (i.e. sorting, filtering, and pivoting) we need to garner real business insights.

So, when should you use your data to create a visual representation of your business intelligence? I thinks it depends on two main factors:

• Who is your audience? Are you talking to other analysts, your marketing team, or executives? How much time will you have? What are their priorities? As with any presentation, this should guide how detailed your visualization is.
• Will the visualization make your data substantially clearer? Can you find a visual format that is easily comprehensible? You want the visualization to illuminate your data, rather than obscuring it. If your charts distract viewers from the important insights, causing them to fixate on red herrings, reconsider your visualization.

Because we have more data, faster processing, and more complex data sources, Business Intelligence now has access to a quantity (and quality) of data that was previously only accessible to enormous research firms—or NASA. Making sense of this data is harder than ever, and design can be a vital part of digesting and presenting data effectively. (A recent TED talk did a great job of showing just how helpful it can be.)

IT’s cultural hesitation about “just” pretty pictures is justified: how many times have we had someone draw up an elegant UI on a wireframe, but neglect to involve the technical team in discussions about feasibility and long term flexibility and support? We know that starting with the picture can run the risk of creating unrealistic expectations or ignoring technical realities. We prefer to start with data and the ability to sort, filter, pivot and manipulate it in whatever ways we need to make our analysis.

On the other hand, if I’m a Chief Marketing Officer and I can quickly illustrate—with hard data—my claim that this year’s campaigns have been more effective than last year’s, my CEO is much more likely to sign off on the budget I’m seeking for next year. A dramatically sloped line, intriguingly weighted pie chart, or clear spatial correlation can be stunningly persuasive. What if I could look at multiple campaigns and channels and overlay the impact one has when used in conjunction with another one? For instance, email and a quick follow up phone call vs. a newsletter sent out after a trade show?

If I’m a business owner, good data visualization can give me a snapshot of how my business is performing as well as helping me spot trends and trouble spots more quickly. For the techie in IT, visualization provides validation for my business intelligence initiatives by increasing understanding of them among those who operate more at the business level.

Recently I attended a Seattle Technical Start Up meeting, where Tableau Software presented their method for visualization of data about FAA accidents with various types of birds. The type of bird was represented by a colored circle which grew in size relative to the total cost of the collision with the airplane. At a glance, you could tell which type of bird caused the greatest damage and filter by time of day or type of aircraft, drilling down to the underlying source and raw data to appease their need to see the numbers.

If we can more quickly and clearly understand the vast amounts of data we all are confronted with on a daily basis and use this information to make changes in our business processes earlier on, we’re one step closer to the promised land of analytical enlightenment.

Contact me at brianh@projectlineinc.com if you would like to learn more.