In case you didn’t see this: IBM created a simple cluster of just 12 antiferromagnetic atoms (you can look up antiferromagnetic on Wikipedia, but not after midnight tonight). John Markoff at the New York Times did a nice write up on IBM’s work. IBM’s work is impressive and could eventually be the foundation of a set of radical new storage products in the future that can hold massive amounts of data with amazing breakthroughs in both speed and power requirements.

As promising as this is, we’re already seeing an explosion in storage technology based upon incremental but substantive progress in processors, drives, files systems, and a variety of software components. In the near future (if not already) even a modest organization will be able to cobble together robust and relatively inexpensive storage systems with 16, 32, 64, or even 128 terabytes without breaking a sweat. Ok, I admit I’ve been in the industry long enough that I still get a bit lightheaded that my iPhone has a 64GB flash drive and that even a small workgroup has terabytes of storage on hand. Creating value in the storage industry will become less about capacity and much more how to coherently deal with dozens of terabytes and even petabytes of data, the bulk of which will be unstructured. If the unstructured data mess is bad today, just hang on. It is going to get much worse before it gets better.

As impressive as the advances in hardware have been and promise to be, it is our firm belief that value creation in storage is going to be dominated by advances in software breakthroughs over the coming years. This will partly come through new generations of applications in the field of metadata, search, analytics, and management. But there will also be significant gains from the “under-the-covers” software. Components like scale-out file systems and advanced protocols will make big advancements in the coming years. Just look at the work being done on SMB 2.2 or NFS 4.1 to get a sense of the significant work being done at this level. I think the winning companies of the future will be those that marry together these software elements with next generation applications in disruptive ways.

Speaking of disruptive, wish us luck with the Seattle weather tomorrow. I’m personally hoping we get pounded, I love winter and all that it entails.

And lastly, while Likewise won’t be taking an official position on SOPA and PIPA, I will post a link if you’d like to make a donation to Wikipedia. I did.

http://wikimediafoundation.org/w/index.php?title=WMFJA085/en/US&utm_source=donate.

Barry

Software for storage management should quickly bring to the attention of data managers those events and patterns of events that suggest problems and opportunities.

Here are some examples of file usage that may be of interest to data managers:

1. If a user attempts to read a file containing sensitive salary data, is the access attempt indicative of a worrisome security violation?

2. Suppose spreadsheet files with financial information reside unprotected in a directory. Should the data manager consider changing the access rights for the directory?

3. If a directory containing a large amount of data has had few accesses over the past month, should the data manager consider moving those files to offline storage?

4. Given a high level of usage of files residing on a CIFS server, should the data manager consider, for performance reasons, redistributing the files across other servers?

5. If a user often writes large files to disk, is this suggestive of inappropriate behavior — such as misuse of company resources for a personal business? (Note that information about the content of the files would be useful in evaluating the inappropriateness of the behavior.)

6. If a user reads a large number of files at an unusual time (e.g., in the middle of the night, or just before leaving the company), is this suggestive of undesired behavior?

How can data management software help data managers zero in on such usage patterns of interest?

First, the software should display current and historical data in tabular or graphical reports, with results ordered by likely significance. Second, the software should implement alerts that notify data managers, by email or by graphical display elements, of situations of particular interest. Third, the software should let data managers explore the data along different dimensions and with various restrictions. Finally, the software should allow data managers to customize reports and alerts to omit useless information and focus on interesting patterns.

These requirements can place a significant burden on data managers, both for customizing reports and alerts and for examining the data. As much as possible, the software should help data managers by automatically predicting which events are of interest. Furthermore, the software should allow data managers to provide feedback that the software can use to customize reports and alerts.

In other words, storage management software needs to perform predictive analytics.

Some examples of predictive analytics in other domains are:

1. Given query terms, predict which documents will be of interest (search engine querying).

2. Given a customer, their history of purchases, and the purchase histories of other customers, predict which products the customer might be interested in buying  (recommendation engine, collaborative filtering).

3. Given a loan application, predict whether the applicant is a worthy credit risk (credit evaluation).

4. Given the submission of a product order on a web store, predict whether the order is fraudulent (fraud detection).

5. Given data about corporations, along with historical stock market performance data, predict the price of a stock at some time in the future (time series analysis, regression).

6. Given an email message, along with a history of messages, predict whether the email message is spam (spam filtering).

Predictive analytics software typically builds a statistical model of the domain and applies machine-learning techniques to categorize or cluster items into interesting classes. Such techniques are broadly categorized into two types: supervised learning and unsupervised clustering algorithms.

With supervised learning, the software categorizes data into two or more classes (e.g., “spam” and “not spam”), based on a training set of labeled historical data. For example, given a large number of emails labeled as “spam” and a large number of emails labeled as “not spam,” spam filtering software categorizes a newly arrived email by comparing it to the two sets of training examples and deciding, based on statistical or geometrical measures, which class (“spam” or “not spam”) is the likely correct label.

With unsupervised algorithms, on the other hand, the aim is to automatically organize items into similar sets by discovering patterns in the data. Unsupervised clustering is often useful as a preliminary step for exploring data and for optimizing supervised learning.

In the context of data management software, the feedback from data managers provides the training data with which the supervised algorithms can learn to improve their predictions. Unsupervised learning is useful for modeling typical usage patterns of files by users.

The key to successful predictions is threefold: (1) identify the correct predictive attributes; (2) model the data using an appropriate formalism (e.g., Bayesian networks, nearest neighbor search, or logistic regression); and (3) settle on a mechanism for feedback so that the software can learn over time.

The surge of unstructured data is unprecedented and dealing with this challenge requires innovative technologies to revolutionize the way companies think about, and use their data to enhance their business.  Right now, we are the only provider offering storage products that approach the big data problem from an integrated perspective, and we will continue to invest in developing technology that breaks down data silos and further unifies identity, security, and storage. It’s been a busy and exciting time for Likewise.  Thank you all for your support of Likewise!  I, and the entire team at Likewise, would like to wish you and your loved ones the very best over the Holidays!

Best wishes,

Barry

I think what struck me first and foremost was the number of people – from very large companies – who told me what a problem unstructured data was becoming. Now, we’ve been talking about this problem for quite some time, but to hear it from these high level executives really confirmed our concerns.

So, how big a problem? Here are some real numbers I heard. These IT executives told me that their data was growing on average between 400 to 800% per year – and the growth was expected in many cases to be exponential for the foreseeable future. The cause of the growth is due in part to Cloud and virtualization technologies as well as the proliferation, perhaps surprising, of new ways of working (like the usage of social media in the enterprise) and new access devices. This proliferation is causing significant redundancy of the data (not to mention potential security issues). Big data is another cause of growth as enterprises number crunch large data sets resulting in numerous large files. Also add to it the fears of throwing away data and the continuing digitization of the workplace.

The other number that stood out to me was the estimate that more than 50% of IT spending is going to storage. Think aboutthat. Even as storage hardware plummets in cost, and tools like compression, deduplication and data-tiering become more advanced, storage costs are still consuming a disproportional amount of an organizations IT budget.

The fact is, we all have to get much smarter about the data we have – just throwing more disks at the problem won’t solve it. We have to get smarter about the data itself – truly understand its value and risks. Currently, we use fairly rudimentary methods for understanding our data – frequency of access, application type and document type – but until businesses look at their data from the top down – AND bottom up and understand everything about it – they will just continue to throw disks into the wind.

To begin with, I spent time talking to – but especially listening to — several devops people about cloud deployments.  The conversations were insightful because these are the very practitioners who were trying to implement a cloud solution rather than just discuss cloud strategy. The conversations covered private, public, and hybrid cloud solutions offered from vendors and from service providers alike. The takeaway for me was that while cloud strategy seems to be maturing as people are thinking more and more about utility computing, the private cloud deployments are still struggling with the interfaces, the infrastructure, and how to support their internal customers.

The one statement that stuck with me during my introductory tutorial on cloud storage was, “We’re being asked to provide departments with a private cloud. This in itself is nothing new, we’ve been providing IT resources on demand for years, but the new requirement is for defined interfaces on how to access the resources.”  Enabling end-users to take control of their IT destiny is the definition of cloud (based on today’s terms) – Utility computing on demand for end-users.

A second takeaway found mostly in conversations during the “Hallway track” was how several researchers that I have high respect for, were moving from their traditional system view of computing to big data problems such as environmental sciences, chemistry, and inter-mural medical diagnostics. The focus on developing new models for managing, processing, and making sense of petabytes of unstructured data is just a sign that our work here at Likewise is on the right path: to help researchers and administrators understand their respective worlds in ways that were not possible until recently.

LISA had enough content, ideas, and conversations to keep me running hard until the next chance I have to take a pulse of our industry.

We’ve been saying for a while that unstructured data challenges require innovative technologies to address how data is secured, stored, and managed in order for organizations to drive value from their unstructured data in an expedient way. The Likewise Data Analytics and Governance application provides a consolidated, scalable infrastructure that expedites data analysis for improved business intelligence that in turn, feeds data driven decision making. Ultimately, unlocking the hidden value of information contained within unstructured data can fuel the creation of new data applications, resulting in improved competitive advantage. Read the datasheet for in-depth technical information.

The customer beta program offers exceptional value to enterprise organizations, including:

• Significant cost savings
• Access to the application now
• Ability to become a strategic design partner
• Receive the production version of the application at no additional charge upon full product release
• Included deployment services and support

To learn more about the beta program and begin improving management of your unstructured data visit http://likewise.com/products/beta.

Read the entire write up here: www.ssg-now.com/likewise-storage-services-provides-oem-nas-developers-with-file-services-and-security