With an estimated 250 million tweets posted daily, Twitter is a huge trove of data about things as mundane as celebrity wardrobe malfunctions or as weighty as rebellions in the Middle East.  What some have called the “Twitter firehose”  can be an invaluable information source, provided the right data can be found and filtered. That’s what DataSift brings to the table..

DataSift’s current service handles only very recent tweets. Gnip, the only other company Twitter has authorized to work with its data, can go back 30 days; DataSift Historics will dial back to January 2010.

While some might question how valuable months- or years-old information really is, DataSift CEO Rob Bailey said companies planning new products, promotions or price changes would do well to study the impact of their past actions before proceeding, and that Twitter is the perfect venue for that.

“Companies spend a lot of money designing new products and social network reaction can be a huge accelerant or a countervailing force,” he said.

A phone maker working on a new model would be well served to go back to see what customers thought of its last release — what they liked and didn’t like about the screen size, resolution, features, for example. If a price change is under consideration, a smart company might go back to see how past price actions were positioned and received.

In addition, it’s helpful for businesses to know which Twitter users have the most influence and followers and to study their tweets carefully.

Bailey said DataSift works closely with Twitter to make sure the data is handled properly. “Users are extremely sensitive to privacy so we spent four to five months checking out the data… if people have deleted tweets, we make sure they stay deleted,” Bailey said.

DataSift Historics will be broadly available in April.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet 2012: companies and technologies set to disrupt
• Finding the Value in Social Media Data
• The Internet of things: creating tomorrow’s health care

Drew Houston

With more than 45 million users already connected to his company’s cloud storage service, Dropbox CEO Drew Houston knows he has an infrastructure challenge ahead of him. The challenge is that as Dropbox adds users and devices, it approaches the echelon of web companies that includes heavyweights such as Facebook and Twitter. And as they’ve shown us time and time again over the past few years, there comes a time when you have to go big on infrastructure or go home.

That, Houston told me, is a big reason Dropbox bought collaboration startup Cove, the founders of which played major roles in building out some of Facebook’s big features while they were there, from 2005 through 2010.

Right now, Houston said Dropbox relies on Amazon Web Services for storing users’ files, but it also manages its own 1,200-plus server infrastructure that handles pretty much everything else. Dropbox also built its own file system that Houston says has to be one of the biggest around in terms of sheer scale. It has done this with just “a few dozen” engineers split across about seven or eight subteams, and only about six people dedicated to the backend infrastructure.

“Find smaller numbers of really amazing people, and you can accomplish a lot more than you might think,” Houston said about his lean engineering team.

But the company aspires to be on par with Facebook and other web leaders in terms of building out advanced web infrastructure, and that’s where the Cove team comes in. Not only does it give Dropbox some of the additional staff it will need to focus on infrastructure, it also gives Dropbox some great staff. After all, as Houston said, “There’s only so many people in the world who think big and want to have a big impact and also have built a product that has touched hundreds of millions of people.”

Cove Co-Founder Aditya Agarwal told me Cove is in the process of building its own backend infrastructure that aligns quite nicely with some of what Dropbox wants to do. Going forward, the plan is to integrate a large portion of the Cove and Dropbox systems, but there are some clear areas of integration early on, including around Cove’s search index.

One thing that probably won’t happen, Houston said, is a move away from Amazon S3 for storing file data. Other large web companies, such as Zynga, like to save money by moving cloud computing workloads back in-house once they reach a certain size and predictability, but Houston said AWS provides too much functionality and flexibility to consider abandoning it right now.

Ultimately, Houston said, Dropbox is more than just an app on an iPhone; it’s trying to build a fabric that ties together every service, device and app. Forty-five million users means even more connected devices — possibly hundreds of millions — and those numbers are only growing. “[C]onnecting all of that,” Houston said, “is a massive engineering challenge.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• Quality of the cloud: best practices for ISVs
• Infrastructure Q3: OpenStack and flash step into the spotlight

That is about to change, according to this OpenStack contributor page, which lists IBM as a participant as of February 2. There has been no announcement, but Krishnan Subramanian over at CloudAve certainly spotted this and got the word out. Another, more official-looking OpenStack contributor page makes no mention of IBM.

OpenStack was launched by Rackspace and NASA in 2010 to furnish an open-source cloud computing platform alternative to what VMware, Amazon Web Services and others offered.  Last fall, Rackspace turned leadership over to a more independent OpenStack Foundation to run.

While each of the big tech companies that back OpenStack has its own proprietary cloud products, they all are hedging their bets by joining the open-source effort in case it winds up becoming popular. As part of their backing, they contribute technology to the effort and support that technology. OpenStack is now available in beta. Ultimately, the idea is that each of the big tech companies would have its own version of OpenStack, but all of them would be interoperable.

IBM, which has an arsenal of hardware, software and middleware at its disposal, has done better than most legacy IT companies of moving in both the open source and commercial software world. It was, for example, largely responsible for the success of Eclipse, the open-source Java framework it launched and then released to an independent foundation.

IBM did not respond to requests for comment;

Updated: An IBM spokesman, responding via email Monday night, did not comment directly but said:

[IBM is encouraged to see] the OpenStack community growing rapidly in support of an open source IaaS Cloud technology. Most recently, OpenStack has announced the evolution of the organization into a more open governance process. We believe that this is a right step in the direction towards expanding the OpenStack ecosystem and in particular its support for standardized cloud APIs.

IBM supports the implementation of standards-based interfaces, he said. “The same holds true for cloud and we’ll continue to explore any efforts that legitimately work to address issues our customers face.”

An OpenStack spokeswoman had no comment.

It could be that one or both companies are holding off on a formal announcement — IBM’s big PartnerWorld Leadership Conference kicks off tomorrow in New Orelans.  This is the kind of announcement that would be ideal for that venue. There is also an OpenStack Conference in April.

Feature photo courtesy of karindalziel

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q2: Big data and PaaS gain more momentum
• The Structure 50: The Top 50 Cloud Innovators
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

Enterprises want Hadoop, badly

Bearden told me during a recent call he thinks the Hadoop market will be bigger than the open source application-platform and database markets because of how much value — and net new value — Hadoop brings to companies as they begin blazing their big data trails. Analytics is big business and only getting bigger as companies start trying to analyze data from entirely new sources such as sensors, social media and web pages, and Hadoop is at the core of most such efforts. He thinks Hadoop will be a billion-dollar market in two or three years.

Bearden points to the interest in Hadoop among large enterprises as proof of his thesis. “There’s not a Fortune 500 enterprise out there that doesn’t have three to five proofs of concept around Hadoop going on right now,” he said. The Fortune 200, he told me, is “absolutely” driving interest in Hortonworks’ suite of open source products and services. The story in every customer meeting is about exponentially growing volumes of unstructured data that are dwarfing structured data volumes, and those companies want Hadoop to be the answer.

But enterprise adoption needs a unified front

Rob Bearden

Despite enterprise interest, however, it will still take a lot of work from Hortonworks and its competitors, such as Cloudera and MapR, in order for Hadoop to fulfill its moneymaking potential.

For one, Bearden said, companies pushing Hadoop distributions need to remain as true as possible to the core Hadoop code as it’s maintained within Apache. This way, users don’t need to choose between what version of Hadoop they want to use and risk being locked in, because products all leverage the same core pieces. The easier it is to adopt, the faster companies will adopt it and the more money they’ll be willing to spend on support.

Both Hortonworks and Cloudera peddle Hadoop distributions based entirely on Apache code, although Cloudera does have proprietary management software and the two package the various components differently. They compete for customers, Bearden said, but both companies “are trying to move the ball down the field within the same boundaries.”

Bearden is not a fan of Hadoop distributions that include proprietary components within the core storage and/or processing framework. “[Cloudera CEO] Mike [Olson] and I both agree that MapR is disingenous in its approach,” Bearden said.

MapR sells a Hadoop distribution that includes a proprietary replacement for the Hadoop Distributed File System, and EMC bases its Greenplum MR offering on MapR’s M5 product. Presumably, Bearden’s also not a fan of new offerings like those from Fujitsu, which on Monday announced a new Hadoop distribution featuring its own proprietary file system. For what it’s worth, though, both investors and some users seem to think quite highly of MapR as a faster, more reliable alternative to Hadoop.

No dissension in the Hortonworks ranks

As I reported on Feb. 17, Bearden recently took over the role of Hortonworks CEO from co-founder and now-CTO Eric Baldeschwieler. My sources characterized the move as a referendum on Baldeschwieler’s leadership and strategy, but Bearden toed the company line, telling me it was just a matter of needing to scale both the technology and the business side in a hurry. In that sense, it makes more sense to have a technologist as CTO and an experienced open-source business mind as CEO, he said.

“The original thesis was that we could take the core components of Hadoop, add some functionality, and get that out the door in an easily consumable manner,” he told me, but “it has become very clear” that Hadoop actually actually has evolved from being a data repository into a data platform. It can be “the center of gravity for the next generation of data architecture,” he said, but that requires a lot of work both technologically and in developing a workable business model, because it means embracing a partner-heavy strategy.

One thing Hortonworks will not do — and something Cloudera’s Olson has told me before about his company — is move up the stack to start providing analytics applications or tuning Hadoop to specific use cases such as transaction processing. Those capabilities will have to come from partners.

Of course, for Hortonworks to become a truly meaningful part of the Hadoop market, it still needs to have a generally available product. Bearden said the first version of the Hortonworks Data Platform will be available within 60 to 90 days and will be updated with incremental improvements about every 45 days thereafter. Once the platform is open to the public, Bearden said, “there will be many ways to monetize that” in the form of customer support and even more technology partnerships like those Hortonworks has in place with Microsoft and Teradata.

You can learn a lot more about the future of Hadoop at our Structure:Data conference next month in New York, where executives from Hortonworks, Cloudera and many other Hadoop vendors will take the stage to talk about just that topic.

Image courtesy of Flickr user RachScottHalls.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Amazon’s DynamoDB: rattling the cloud market
• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

U.S. Venture Partners led the round, and USVP partner Rick Lewis is joining the board of Denver-based Wayin. USVP joins an interesting roster of investors, including enterprise software company TIBCO; Silicon Valley law firm Wilson Sonsini Goodrich and Rosati; former head of Microsoft Business Solutions and McNealy classmate Doug Burgum; Hollywood producer Burt Sugarman; and sportscaster Jim Gray.

As McNealy told GigaOM late last year, it’s easier to raise money if you have a good rolodex. Starting Wayin “was actually a little easier probably than it should have been,” he said. Early on, the startup was able to rake in $6.3 million in investment without even tapping the VC community, McNealy said at the time.

At launch time, GigaOM’s Ryan Lawler described the service as a website and mobile application that let users easily create and vote on polls, follow other users and share opinions. Frequent users get points for creating and voting on polls.

Clearly, social networks — especially Twitter and Facebook — are hugely influential. Companies pay good money to analyze and parse these streams of data to perform sentiment analysis, which is something McNealy has acknowledged Wayin intends to cash in on. Whether or not Wayin can become a data source on the level of Facebook or Twitter, however, is up in the air.

Photo courtesy of Flickr user Eddie Awad.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• NewNet 2012: companies and technologies set to disrupt
• Dissecting the data: 5 issues for our digital future

By all accounts, Dropbox provides a slick way to upload and store digital paraphernalia in the cloud. From there, users can access their stuff from any device and sync files across devices. The service has been hugely popular: As of four months ago, Dropbox claimed more than 45 million users. But the success of the five-year-old company has bred imitators and competitors, including the biggest companies in tech.

The new private camera upload feature will let users take their photos as always but then easily move them from smartphone, tablet, camera or SD card to their cloud data trove using Wi-Fi or their cellular data plans. Dropbox uploads the photos and videos in their original size and full resolution to the user’s camera upload location. The feature is available now for Android phones with Windows, Mac and iOS support to come, Dropbox said.

Hardware makers hedge with Dropbox

This is one example of how Dropbox is trying to stay ahead of the curve and make itself an indispensable tool for connected consumers. In that, it has some formidable partners. Just this weekend at the Mobile World Congress, HTC said buyers of its new HTC One phones will get 25 gigabytes of Dropbox storage free for two years. Handset makers like HTC see Dropbox alliances as a way to combat Apple’s iCloud initiative.

Dropbox’s popularity has certainly been noted. Companies from Microsoft to Apple and (probably) Google are trying to mimic its capabilities. “Everyone wants to be Dropbox,” Andres Rodriguez, the CEO of storage specialist Nasuni, told me recently. Steve Jobs, the late CEO of Apple, reportedly wanted to buy the company. When that didn’t work out, Jobs called Dropbox “a feature, not a company”  and launched iCloud.

That “feature versus company” meme has dogged Dropbox ever since and cropped up again this weekend when PandoDaily’s Farhad Manjoo weighed in on Jobs’ side of the debate:

Dropbox is a great little file-syncing app, and founder Drew Houston and crew are already making some nice money out of it. But is it a $40 billion company? I doubt it. And when I hear folks like Benchmark’s Bill Gurley suggesting that it might be, and calling Dropbox “a major disruption,” I wonder if they’ve simply been blinded by the thrill of using an obviously well-crafted utility.

Dropbox is slick, and it supports nearly all the relevant clients. But in Manjoo’s experience, that support is uneven. Dropbox is often flummoxed by OS- and application-level problems, he wrote.

But any neutral party without access to Apple’s native hardware hooks will be somewhat stymied. Plus, that only takes into account some of Dropbox’s value, argues Posterous co-founder and venture capitalist Garry Tan. On his blog, Tan writes that the tech giants (Google, Apple, Microsoft) that make their own OSes and applications have no incentive to make them sync well with others.

What are the odds of Apple getting their sync client right for PC’s? Just about zero, considering what they’ve done in the past with MobileMe sync.

Same goes for Microsoft writing sync software for the Apple platform. Arguably Google is in the best shape to provide a seamless multiplatform experience . . . well, except for iOS! The odds of a viable multi-platform option emerging from one of these big three seem slim to me.

Those who forget history . . .

The cautionary tale for Dropbox is that the best technology doesn’t always win. (I would insert the Betamax vs. VHS argument here, but no one remembers it anymore.) Should Microsoft, Apple or Google offer at least reasonably good cross-platform file storage and sync capabilities, Dropbox will be in trouble. Working in Dropbox’s favor is that CEO Drew Houston appears acutely aware of history.

According to this Forbes magazine account, when Apple announced iCloud, Houston shot off a memo to employees, reaffirming the company’s status as one of the fastest-growing companies in the world. Then he listed several other once-fast-growing companies: MySpace, Netscape, Palm and Yahoo.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 12 tech leaders’ resolutions for 2012
• NewNet Q4: Platform mania and social commerce shakeout
• Dissecting the data: 5 issues for our digital future

Better tools for big data.

So, if it’s safe to assume that big data is real, and that you should be investing, where do you start and what should you expect as you go through the adoption process? Big data today, is what the web was in 1993. We knew the web was something and that it might get big, but few of us really understood what “big” meant. Today, I believe we aren’t even scratching the surface of the big data opportunity.

A good example of potential big data use models can be found here.

Current issues with adoption.

There are a number of issues that will affect your ability to successfully adopt and make best use of a big data solution, but the three I believe are most critical are:

1. Useable enterprise tools — The tools that will allow any business to fully utilize big data aren’t ready.
2. Lack of staff expertise — The availability of data scientists or folks with a similar background is limited at best.
3. Data gravity — As Dave McCory pointed out in his post on data gravity, where data is created/sent is where it ends up being used. The applications and people need to come to the data as I explained.

How will these adoption issues affect big data as a business opportunity?

Useable enterprise tools — The current suite of products include Greenplum, Cloudera’s Hadoop and others, which are making headway in many large enterprises. However, these tools are still new and generally require large technical teams trying to solve issues for companies like eBay and Sears. A smaller company would be less likely to gain the appropriate return on investment, because of the high complexity of implementation combined with low overall volume.

Lack of staff expertise — This area is similar to enterprise tools. Even if you’ve got 10 people working on the refinement of the system, it’s likely going to boil down to that one wizard/expert who can work magic with your data. Putting a large number of people on the problem won’t guarantee success.

Data gravity — Considering the strong possibility that most organizations will struggle to fulfill the promise of their big data strategy with internal resources, we are likely to see a proliferation of services from various cloud providers. My concern here is that the use characteristics of Big-Data-as-a-Service aren’t being thoroughly examined.

The questions and big picture concerns.

Thanks to complex implementations, the data divide could grow.

I see big data quickly becoming a competitive advantage, which is the good news. However, I see significant parallels between the ability to pay for and adopt big data and the first decades of the mainframe. Only a few companies could afford mainframes, and those companies that could afford them were able to develop real advantage. With the introduction of the internet and cloud computing we have moved to a much more democratic model of IT availability, but big data has the potential to re-insert that gap between the haves and the have-nots.

When thinking about democratizing the use of data, the following questions come to mind. They can relate to your implementations, but also are worth thinking about in general. They are:

• Where will your data reside?
• How will you get your data to the service?
• Will tools be delivered across the wide area network (WAN) to be run locally against your in-house data?
• How will you collect and capture your own data?
• If you store your data with a service how often will you use it? Or will you likely be paying to keep it handy for rare uses? (I call this the problem of “Data in Waiting”)
• If you store your data on the service providers storage such as on S3 but you don’t want to pay for it when it’s not in use, will you delete it? How will you know it has been deleted?
• If your big data is running in a public cloud, what tools, and strategies will you use to make that data available to customers and other applications (integration)?
• Will big data cause you to buy more WAN capacity?
• Will big data cause you to rethink your enterprise application strategy?

So what’s the solution to bring data to as many businesses as possible?

To make big data available to everyone we need quite a few things to happen. We need to figure out simple use cases for data to solve common problem sets. Then we must make those available to developers so they can build tools that make solving those set problems easy. We need to continue to push the boundaries of cost-effective disk storage and network capacity, or provide ecosystem environments that allow for direct access over a private network. In an ideal world we will do both.

We’ll know big data has arrived when the use of the service is integrated into common business software tools that are used by the majority of your businesses employees. Also key will be the ability of any knowledge worker to run their own questions/queries against internal and external data sources. The average business won’t be able to call big data truly successful or accessible as long as its usability is being defined and managed by a small disconnected team of IT scientists.

Mark Thiele is executive VP of Data Center Tech at Switch, the operator of the SuperNAP data center in Las Vegas. Thiele blogs at SwitchScribe and at Data Center Pulse, where is also president and founder. He can be found on Twitter at @mthiele10.

Interested in big data? Come talk to us at our Structure Data event next month in New York City.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Why the big data startup boom will likely be short-lived
• 12 tech leaders’ resolutions for 2012
• Dissecting the data: 5 issues for our digital future

Before he passed away, Steve Jobs made his final goal clear: make Apple the backbone of the consumer cloud. First Apple launched cloud features within Mobile.me. Then it updated cloud-enabled versions of iOS. With the OS X Mountain Lion preview last week, we can finally see how Apple plans to change our world again.

Any move by Apple is obviously watched closely by Microsoft. But after digging into Mountain Lion’s features and philosophy, I think Apple’s new operating system poses a threat to another, less obvious, company — the current leader in the consumer cloud, Facebook.

Many companies have learned the hard way not to underestimate Apple. For Microsoft and Facebook, now may be the time to make bold moves to ensure their continued relevance in the consumer cloud.

Facebook and Apple, a rocky road

Ever since the last-minute removal of Facebook from Apple’s Ping launch, Apple and Facebook haven’t been on the best terms. The delayed Facebook iPad app, rumors of a Facebook phone and the inclusion of Twitter integration in iOS 5 all add to this division.

With the launch of the developer preview of Mountain Lion, Facebook is notably absent once again. If you believe, as I do, that integrations within operating systems will be critically important for social services going forward, this development is more than a slight – it’s a potential catastrophe for Facebook.

The good news for Facebook is that it already has a friend in the OS game (a friend with $240 million in goodwill) — one it may want to get even closer to.

A friend in Redmond

Rapid consumer adoption of iOS and OS X in any form clearly threatens Microsoft. But OS X Mountain Lion’s social cloud features, including messaging and photo storage, create an entirely new set of competitors beyond operating systems.

Apple has chosen a few strategic partners in Flickr, Vimeo and Twitter, but this isn’t necessarily great news for those companies. Remember, Apple has no qualms about mimicking successful apps from their app store (just ask Instapaper).

The only companies that are worse off are those not involved from the start.

Suddenly, Facebook and Microsoft find themselves on the same side of the table. If I were at either company, I’d be considering huge moves — all the way up to an acquisition or merger.

I realize how crazy this might sound at Facebook, so here is a bit of perspective: The most valuable company in the U.S. has just declared war on you. They want your users’ photos on their system. They want your users’ eyes and attention in their ecosystem and not focused on your site.

Oh, and they have $100 billion in cash, the largest mobile operating system and the fastest growth in personal computer share.

The Mountain Lion roars

The list of Mountain Lion’s features shouldn’t have come as a surprise to anyone. But in its totality, it offers fascinating insight into Apple’s view of the future, one where all content, connections and experiences aren’t tethered to websites, but to the cloud – the cloud as defined by OS X.

The move is such a game changer, it’s easy to miss by looking at features alone. In my estimation, it’s even larger than Om said last week, when he discussed the move to cloud-based operating systems:

“It doesn’t matter whether we use Windows, Mac, Linux, Android or iOS: we can do all the things we like to do as long as the Internet is there.”

Yes, Apple’s upcoming OS gives us a window to the Internet and other services using the cloud, but now our Apple devices will automatically know who we are, who our friends are, the content we care about and all the apps we use. With your reminders, contacts and photos automatically synced between each of your devices, you’ll never feel the Internet.

And you won’t miss it.

In other words, many of the reasons we love Facebook will be baked into all of our online devices, but that content won’t be controlled by Facebook. It will be controlled by Apple.

Of course the Internet will still play a role, but it will take a backseat to features and connections built into the cloud-based OS X Mountain Lion.

Only from Apple

Apple is the only company that can do this alone right now. To create a cloud OS that’s relevant, you need to hit a few critical tipping points:

• Hundreds of millions of active users at several access points (desktop and mobile)
• Billions of pieces of personal information and content generated on your system
• Control over your ecosystem to mandate these changes and force people to get on board en masse

Microsoft barely has a mobile installation base and barely any network, but it does have access to hundreds of millions of desktops.

Facebook may have nearly a billion users, but it is just a website/application at the mercy of the operating system.

But, together, the two form one of the only plausible challenges to Apple’s next move.

If I was at Microsoft, here’s what I’d be looking at:

• OS-level social collaborations: Office in the cloud has been in the works for years. But without a reliable network to tap into, it has failed to gain critical mass. Facebook’s 800 million members (with 100-plus connections each) could definitely help.
• Identity: Facebook probably has the most reliable database of personal information outside of the government. Imagine buying a Windows 9 computer or phone, logging in with your Facebook ID and finding all your content, connections and games installed automatically.
• Social data into Bing: Wholesale collaboration between Bing and Facebook could finally create a legitimate challenge to Google’s search dominance (at a time when Google is showing its first signs of weakness).
• Hooks into the Web: Microsoft was notoriously late to the Web and even later to the social Web. Short of an acquisition or merger, it seems impossible for a Microsoft branded “like” button to make any meaningful headway. This  kind of social curation is imperative to Bing’s success.
• Personal cloud data: As we document more of our lives on our mobile phones, Microsoft’s lack of a meaningful mobile presence puts it at a huge disadvantage. The company will have an increasingly difficult time acquiring users’ photos without a major player in cloud photo storage, such as Facebook.

What Facebook gets:

• Real stickiness: Make no mistake about it, the world is gunning for Facebook. From Pinterest to Tumblr to RunKeepeer, social threats to Facebook’s dominance emerge daily. OS-level integration would create a significant barrier for competitors, or at least require them to play on Facebook’s terms in the Windows sandbox.
• Top-level integration: By seamlessly integrating messaging across handsets, desktops and tablets, iMessage is a great example of an OS-level integration that threatens Facebook. Yes, Facebook messaging works across all of those devices now, but it’s at a disadvantage to native apps.
• Cash: Microsoft has nearly half of Facebook’s IPO value on hand in cash (or equivalents). If Facebook is going to take on Apple or Google (or both), Microsoft’s substantial financial assets could definitely come in handy.

Bonus: The FB Phone 2, powered by Windows Phone

Facebook strategically understands the need for its own handset, but building off of the Android platform is less than ideal, for several reasons. Microsoft has a great mobile OS and quality hardware partners, but it lacks the differentiation needed to make a dent in the market. A Facebook-branded, Microsoft-subsidized and -engineered phone could be the only legitimate competitor to iOS and Android.

Throw in Facebook’s friendship with Zynga, and the partnership could lure talented developers away from their focus on Apple apps.

Time to act

OS X Mountain Lion is a different kind of operating system, and as such threatens different types of companies than operating systems ever have before.

For Microsoft and Facebook, now may be the time to bring their social and OS expertise and users together — before it’s too late.

Edward Aten is the founder of Swift.fm, a social music distribution service. Follow him on Twitter@edwardaten. The views expressed here are personal and do not necessarily reflect those of any Company with which he is or has been affiliated.

Image courtesy of Flickr user ingridtaylar.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet Q4: Platform mania and social commerce shakeout
• Millennials in the enterprise, part 2: benchmarking IT’s readiness for the new digital workforce
• Mobile Q2: Smartphone growth surges; iPad’s rule continues

Dropbox, Box and other cloud-based backup tools for desktop and mobile devices have already been wildly successful. Similar tools, such as Riverbed, StorSimple and Ctera, are available for the server market, but they take a hardware-centric approach that has failed to garner a large market.

When Amazon Web Services launched AWS Storage Gateway last month, the move seemed entirely logical, almost expected, given the enormous appetite that desktops have shown for cloud-based data sharing and backup.

But is this enough? Most companies have become pretty good at copying data. The real challenge is not restoring data, but going a level higher and recovering services when disaster strikes. That’s particularly important in the era of IT consumerization, where end users expect to access data through applications of their choice on the device of their choice. In other words, it is useless to protect data and not protect the applications that exploit it.

This is where current backup tools fall short. End users are concerned about how long the outage is going to last, not whether the data is safe (which they assume is a given). Disaster recovery is about minimizing downtime. The cloud has huge potential to make an impact here. By managing and scheduling all of the components involved in service delivery, the cloud could turn recovery time objectives into guarantees.

We are at a fork in the road: backup and disaster recovery are going to be two separate processes.

Backup will be used to quickly restore data — including files, mailboxes, attachments and database tables — and to keep auditors happy about long-term data availability.

Disaster recovery will focus on continuity and service recovery, not data restoration. It will orchestrate all of the components involved in service delivery — from storage to hypervisors, operating systems, databases, middleware and applications — across collaborating clouds. This means that when a cloud disappears, another cloud will be ready to take over at the push of a button. And we’ll be 100 percent certain about the maximum outage time. There will be no need to worry about doing disaster recovery exercises, because clouds will do them on their own, continuously and accurately.

Carlos Escapa is the CEO of VirtualSharp Software. Previously, he was a senior executive at VMware in Europe, where he managed VMware’s field operations in France, Portugal, Spain, Italy and Greece. 

Image courtesy of Flickr user Kevin Burkett.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• A field guide to cloud computing: current trends, future opportunities
• Smartphones help us to understand the cloud
• Why Dropbox shouldn’t move to the enterprise space

What CrowdControl does, essentially, is partner with crowdsourcing services like Amazon Mechanical Turk (a very close partner, in fact) to provide a layer of quality assurance between the remote workers and the client needing work done. CrowdControl provides a worker-management interface, determines pricing and — most critically — uses its software to determine whether the work product is accurate.

Here’s how I described CrowdControl’s theory and methodology back in November:

Sentiment analysis already is becoming big business for companies such as IBM and SAS that are turning their predictive analytics engines on social media streams. But CrowdControl Founder and CEO Max Yankelevich says there are two big problems in the space right now. One is that current natural-language-processing technologies are better suited to identifying keywords than they are to deciphering true sentiment. The other is that humans, whose brains are inherently better at looking at text in context and working around abbreviations and poor grammar, have a tendency to underperform.

…

To cure that problem, CrowdControl contains more than 15,000 rules to determine how accurate workers are in completing their tasks. Those rules comprise much of the company’s secret sauce, but Yankelevich explained the methods for “adjudication,” the process of judging accuracy, at a high level. A big one is called “plurality,” which entails either assessing a worker’s answer in relation to everyone else’s answer on the same question, or giving the same question multiple times and looking for the same response. Another is “gold answers:” The tester continuously inserts questions to which it knows the answer and calculates how often the worker gets it right.

The company recently completed a survey of half of Amazon Mechanical Turk workers, which should help it further refine its algorithms and processes. As Yankelevich told me in November, where workers live and what they do for a living can have a big effect on how they perform. Presently, CrowdControl says its ideal use cases are sentiment analysis, data cleansing and data normalizing, which is the process of adding consistent structure to unruly data sets.

Image courtesy of CrowdControl.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Defining work in the digital age: an analysis by GigaOM Pro
• Dissecting the data: 5 issues for our digital future
• Siri: Say hello to the coming “invisible interface”

The company is now building its own storage hardware to keep up with the exploding demand of its more than 840 million users, according to a Wired report. Facebook users put millions of their own photos and other digital paraphernalia on the site.

Frank Frankovsky, the former Dell hardware guy that who now spearheads Facebook’s data center hardware effort told Wired:

We’re taking the same approach we took with servers: Eliminate anything that’s not directly adding value. The really valuable part of storage is the disk drive itself and the software that controls how the data gets distributed to and recovered from those drives. We want to eliminate any ancillary components around the drive — and make it more serviceable.

In October, Frankovsky told GigaOM that Facebook was also looking at storage as part of the overall Open Compute Project it launched to standardize the energy-efficient data center gear. At that time he said: “Storing data at this scale has some unique challenges. We’ll work on those contributions and with the rest of the community on this.”

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Quality of the cloud: best practices for ISVs
• Migrating media applications to the private cloud: best practices for businesses
• Dissecting the data: 5 issues for our digital future

Lightwire's optics transceiver

Cisco said Friday it is buying Lightwire, a company that makes optics chips using traditional chip manufacturing techniques in a deal valued at $271 million. This is a significant acquisition for Cisco because it stays true to its core networking business and advances it with some pretty deep technology, which is just the thing to keep Cisco ahead of the growing pack of lower-cost competition on the switching side and the looming threat of open, software-defined networks.

Lightwire, which was formed in 2002, makes an optical transceiver using traditional CMOS manufacturing technologies. Competitors include Luxtera, Intel and IBM. Such transceivers are becoming more important for two reasons. One, thanks to the ability to use the CMOS manufacturing process ,the chips are priced for mass use in the data center, unlike traditional optical chips. Two, the use of optics means the chip can deliver faster bandwidth using less power, speeding up I/O between servers. Both are good in today’s power-constrained and highly demanding data centers, running applications such as Hadoop clusters or data analytics. Here’s the money quote form the release:

The acquisition of Lightwire will support our data center and service provider customers as they manage the continuing deluge of network traffic alongside tight capital and operating budgets,” said Surya Panditi, senior vice president, Cisco Service Provider Networking Group. “With the combined know-how from Cisco in silicon design and Lightwire in CMOS photonics, we will transform Cisco’s optical connectivity business to an integrated technology platform that supports our customers’ burgeoning need for cost-effective high-speed networks.”

The deal plays to Cisco’s networking strength and also advances it with some hard-core chip technology — a key element of Cisco’s plan to differentiate itself from competitors using merchant silicon, such as Arista. Sources in Silicon Valley in the networking space have been telling me that Cisco needs to counter the threat of merchant silicon by either adopting it and driving prices (and Cisco’s margins down), or by investing in “physics problems” — basically the type of R&D that got Lightwire to its product.

Earlier this month, Cisco CEO John Chambers said Cisco was ready to make more acquisitions . This one looks worth watching.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q4: Big data gets bigger and SaaS startups shine
• Dissecting the data: 5 issues for our digital future
• What Amazon’s new Kindle line means for Apple, Netflix and online media

Much has been written about the “big data” phenomenon — the petabytes of machine data from computers, sensors and other equipment; social networking data; scientific data — is a rich but unwieldy trove that is available for the taking. The big data problem is that the sheer amount and diversity of this data outmatches the abilities of traditional relational databases like Oracle, SQL Server and DB2 to handle effectively. With the Hadoop distributed data file system and MapReduce processing power, that data can be aggregated. The next step is finding tools to analyze it further.

It’s that analytics problem that has Andy Palmer excited. Palmer is a serial database entrepreneur who co-founded Vertica Systems (now part of Hewlett-Packard) and VoltDB and was a founding board member of Bluefin Lab, CloudSwitch (now part of Verizon), and Recorded Future.

“The real purpose of big data is to enable big analytics. The most compelling companies out there, I think, are those that attack that problem,” Palmer told me this week.”I really do believe that big data is, in and of itself, a tool. The real story is more about big analytics. Once you aggregate the data you then have to ask really hard questions.”

Andy Palmer

The surging interest in data analytics and visualization tools supports his take. Splunk last month filed for its IPO, and Tableau is well on its way.  Another analytics player, QlikView went public last summer, and its stock has doubled since launch, as Derrick Harris reported in GigaOM. All of these companies aim to help users make sense of all that data.

Palmer, who often works with database pioneer Michael Stonebraker, shares Stonebraker’s view that the sheer variety of data formats and the types of operations to be performed on them call for a variety of specialized databases.

There is a real need for database technology that can handle multi-dimensional data arrays — data sets that often come out of astronomy and other scientific research, Palmer said.  ”When you represent data in traditional relational databases, you can compromise the inherent nature of the data. And if you integrate a lot of data together, ultimately that data looks like a large array. Representing an array in a traditional database is really an unnatural act,” he said.

He is backing yet another Stonebraker company, Paradigm4, that is attacking that problem. In the past, the big database powers were able to shoehorn new types of workloads into their relational model. For example,  a decade or so ago, there was a raft of small, innovative object database companies — Object Design, Ontos and others — that built their businesses on the premise that relational databases could not handle objects which did not fit well into the rows-and-columns world of relational databases. Over time, however, the big data base players pushed and shoved at least some object capabilities into their databases, and those smaller companies disappeared.

Palmer and others in the big data world said this won’t happen again — that big data cannot be co-opted the same way — it would be way too expensive and resource intensive for traditional databases to try to churn through all this stuff. That’s why Oracle et al. are coming out with specialized big data products.

And when it comes to big data, the data itself will be meaningless unless the right analytic tools are available to sift through it and there are people who know what questions to ask. Big data, and the big analytics used to make sense of it, will be hot topics at GigaOM’s Structure: Data conference next month in New York City.

Photo courtesy of  Flickr user Il conte di Luna.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Facebook’s IPO filing: ideas and implications
• NewNet Q4: Platform mania and social commerce shakeout
• Why the big data startup boom will likely be short-lived

The big picture is that as the network becomes the pipeline running through modern society, his company stands to gain — if he can mange the transitions.

Lucky for Cisco, it’s not like the transition to an all-IP world connected by networks is a sudden shift. Change is happening in waves, and Chambers hopes to take advantage of as many waves as he can. Already, Cisco has managed to successfully predict and deliver products for the transition to video on consumer and corporate networks. In four years, Chambers believes video will dominate corporate communications — and even IT.

I’ve whittled down the 90-minute discussion into three key takeaways (and reported earlier on why Cisco didn’t buy Skype) that our readers will find most interesting. The topics range from the failure of the Flip buy to Cisco’s plans for the data center. Let’s begin in the home.

The changing set-top-box market.

Cisco is not, I repeat, not selling the set-top box business it purchased in 2005 when it paid $7 billion for Scientific Atlanta. Chambers denied it at least three times during the call, but he also said that in five years, he’s not sure content from service providers will be delivered from a box. Instead, it will be all done in the cloud.

Cisco's home videoscape product.

On it’s face, that seems like an admission that the set-top box is headed for the scrap heap, but Chambers is right that service providers are not going to give up on boxes inside their subscriber’s homes. None of the executives inside telcos or cable companies are eager to provide a pipe into the home, they all want some kind of hardware to ensure a device where they can deliver software, services and any other value add that will help them stay above the commodity pipe business. The only question is will that box be a modem, set-top-box or a residential gateway. Whatever it is, Cisco will be happy to provide it.

Speaking of the cloud, Cisco has a (secret) plan there too.

Chambers was mum on his plans to address the dual threat of an entirely new data center customer that cares mostly about price and scaling out, as opposed to the nifty software and plug-and-play aspects of Cisco’s UCS gear. He was also pretty quiet on the looming threat of software-defined networks, although he pointed out that Cisco is a member of the Open Networking Foundation and will support Open Flow. However, the real threat of Open Flow and SDNs is that they will undermine Cisco’s architecture play.

Cisco's UCS server.

In the data center, Chambers emphasized the customer wins at service providers (about 45 percent of the customers) and Fortune 500 companies. Cisco is betting that a system that goes from chip-based to software and services will win out. So far it’s doing well, but I also think it’s missing out on the next big wave of growth in the server market — companies like Amazon, Facebook and Yahoo that buy their servers by the racks (and even by the data center) as opposed to by the box.

Those companies are trying out merchant silicon in switches such as those sold by Arista, or even lower-level gear from the likes of companies like Huawei or Quanta. Chambers’ response to the rise of merchant silicon and commodity boxes is to bring on the ASICs. That’s right, Cisco plans to keep designing its own chips as part of its architecture play — but it also plans to design them faster. When I asked Chambers to go deeper into this, he declined.

It’s possible a custom ASIC could bring value to Cisco, even in the webscale world, if used a little bit more strategically. For example, SeaMicro is a hot startup in the server space because it’s building a box that could replace 500 machines from five years ago and run at 96 percent of the power. SeaMicro can do this because it has developed its own ASIC to handle the networking inside that box.

When I pointed this out, and asked if Cisco could deliver a fundamental rethink of its ASIC to deliver that type of functionality, Chambers said he wasn’t willing to discuss this further, so as not “to let all his competitors know where Cisco was going.”

But M&A is in the air.

Chambers seemed frustrated that no one asked about Cisco’s recent announcement that it planned to start buying other companies. He didn’t really offer any insights about who or what he wanted to buy, but on Friday Cisco announced the planned acquisition of LightWire, a company that makes an optical transceiver.

He also acknowledged Cisco’s mistaken efforts to get into the consumer market with the Pure Digital acquisition (otherwise known as the Flip camera buy.) He said when his team realized the deal wasn’t working, they quickly decided to shut it down, and did so, which is possibly the best way to spin a $590-million mistake.

Indeed, Chambers appeared to be the master of being able to admit his mistakes, and then spinning them quickly to an asset. Whether it was his ability to recognize quickly that the $590 million buy of Flip wasn’t going to work or his realization that an effort to change the pricing and product mix on switches led to a sudden drop in margins that he has since fixed, Chambers hasn’t met a market transition he couldn’t either see coming or fix in retrospect with quick action or an acquisition.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• NewNet 2012: companies and technologies set to disrupt
• 12 tech leaders’ resolutions for 2012
• Cloud computing 2012: a pessimist’s guide

QlikView — Tableau’s biggest competitor in the next-generation business intelligence space –went public in July 2010 and is currently trading at about $30 a share after starting at around $15. In 2011, it brought in $320.6 million in revenue from more than 24,000 customers, almost a third of that in the fourth quarter alone. In January, Splunk, a software company that makes it easy to parse through, analyze and visualize server logs and machine-generated data, filed for its IPO and hopes to raise $125 million. Splunk closed its fiscal year 2011 with $66.2 million in revenue, nearly doubling annual sales over each of the past two years.

Christian Chabot

Tableau, for its part, earned almost $72 million in revenue in 2011, co-founder and CEO Christian Chabot told me during a call on Thursday, almost double what it earned in 2010. The company nearly doubled its head count in 2011 to about 350 employees, and Chabot said it is looking to add another 300 or so employees in 2012. “It has always been Tableau’s intent to build a big and international and publicly held company,” he said, and the revenue, employee and management growth are all part of that plan. It has more than 50,000 users.

These companies are growing like gangbusters and surviving the siren song of M&A because they make products that make it easy to analyze and visualize data. The advent of big data highlighted the importance of both of those tasks, but it also introduced the world to data scientists — which are in high demand and low supply — as the way to do them. However, Tableau, QlikView and Splunk all connect to big data platforms such as Hadoop and Teradata and vastly simplify the process of performing certain analyses and creating compelling charts.

More importantly, though, these companies also cater to everyday users who don’t have any need for big data — or even skills in legacy BI tools — but who understand the importance of analyzing the data they have. Tableau even has a free version, Tableau Public (requires registration), that is hosted in the cloud and caters to bloggers and other knowledge workers who understand the importance of data analysis but don’t have the budget to pay $999 for Tableau’s entry-level desktop software.

QlikView screenshot

QlikView Founder and CEO Lars Bjork told me during a November call that just as Google and Apple have made consumers the masters of their personal technology, his company as well wants to make business users the masters of their own tools.

QlikView CTO Anthony Deighton expanded on this, noting that end users in business are often pushed data from analysts rather than being able to derive their own insights. “We want to be the end of the end user,” he said.

Splunk is already letting customers click and search their way through the data in its system similar to how they surf the Web, but founder and CTO Erik Swan — who I will be interviewing on stage at our Structure: Data conference next month — told me in June that the company is hard at work on new capabilities that will let developers actually write MapReduce-like applications to access Splunk data using popular languages like Ruby and Java.

Sample from the Tableau Public Gallery

Tableau’s Chabot explained his vision to me during the company’s user conference in October. Tableau has invested heavily in a product that understands users’ data and can guide them to the best ways to analyze and visualize it — all while cloaking analytical functions behind a drag-and-drop interface. Right down to the psychological rules and Edward Tufte presentation principles built into the product, he said, it’s all about creating the most intuitive and yet amazing product possible, an iPhone in a world of legacy brick phones.

“In any field of human endeavor . . . there are a hundred to a thousand more people who understand the data of that field more than they understand reporting and analytics,” said Chabot. When they are all using your product, it turns out, that’s great for business.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Dissecting the data: 5 issues for our digital future
• Defining Hadoop: the Players, Technologies and Challenges of 2011
• Amazon’s DynamoDB: rattling the cloud market

Box, formerly known as Box.net, is one of several companies — including fan-favorite Dropbox as well as Microsoft, and Google — competing to store your digital paraphernalia — Word documents, presentations, photographs, whatever — in the cloud and make it accessible from your devices of choice.

A big part of the update is the facelift Box gave virtually all Android clients. “We worked closely with Google to build a modern interface using Google’s Ice Cream Sandwich principles. That Android 4 UI will run cross all Android clients,” said Matthew Self, VP of platform engineering for Box. (Ice Cream Sandwich is Google’s name for the latest Android smartphone OS.)

Given the number of Android flavors in the field — every carrier has its own — and different versions of each, that single UI claim is no small feat. Self estimates there are easily hundreds of device-Android combinations out there.

Also new to Android: Users will be able to upload multiple files in batch mode and in background and can invite collaborators to work on  a document from their device. And, they can comment on these documents from their phones, Self said.

While Box supports a wide variety of non-Android devices–including Apple  iPhones and iPads, RIM’s Blackberry and Blackberry Playbook, even HP’s defunct TouchPad — Android appears to be the favorite.

“Android has eclipsed iPhone on the phone side and it’s growing fast in tablets.  There are a lot of Android phones coming into the enterprise,” Self said. He cited Gartner numbers showing Android with 50 percent of the smartphone market, Apple iOS with 25 percent and Microsoft with just 2 percent — a number he does not think will improve much.

That explains why Box offers no native Windows Phone support at all, although an HTML5 browser-based client runs on most devices. Self discounted the ability of Microsoft to gain significant market share in smartphones even though the new Windows Phone Mango OS has been well reviewed.

Update: A Box spokesman wrote in to clarify the company’s position on Windows Phone. He said:

At Box, we want to make it dead simple for our users to share and collaborate on business content from any device. We’ve invested aggressively in building amazing experiences on iOS and Android because those are the platforms our customers are using, but we’re always tracking adoption and demand, and will support Windows when we see it get traction in the organizations we serve.

With the updated Android clients comes support for four new languages — French, Italian, German, and Spanish– so Box is no longer an English-only experience.  The company will show off its new offerings next week at the Global World Congress in Barcelona.

It’s becoming clear that Box is banking on Android at the expense of Windows Phone and even Apple’s popular devices. The fact that Microsoft’s SkyDrive and  Apple’s iCloud cloud storage services could be considered Box competitors might be a motivating factor.

Still, given RIM’s Blackberry woes, the continued fragmentation of the Android market and the worry that Google’s acquisition of Motorola is causing rival handset makers –all of which could help Microsoft’s smartphone efforts– this looks like a risky move.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• 2012: Data, spectrum and the race to LTE
• NewNet Q4: Platform mania and social commerce shakeout
• NewNet 2012: companies and technologies set to disrupt