Mark Chmarny

Big Data Twitter Monitor - Satisfying My Geeky Cravings for Development

2012-02-22T17:21:00.002-08:00

Development

Most of my posts on this blog are technical; often focused on the challenges of Cloud Computing, IT operations, and most recently the opportunities around Big Data. Some of you may know, however, that I used to be much more hands-on in the software development process, most recently during my time at Windsor Solutions.

As part of my work at EMC, I've recently had the chance to focus on the architecture of VMware's new PaaS solution called Cloud Foundry. In the process, I developed somewhat of a nostalgia for application development.

So, in part to satisfy my geeky cravings, and in part to prove to myself that I still can do it, I have embarked on a small extra-curriculum development project: Big Data Twitter Monitor.

To make it even more interesting, I picked a whole new framework (Node.js). I wanted to create an app that would use a NoSql storage (MongoDB), some appealing UI (jQuery) and provide the simplest migration path to the Cloud (Cloud Foundry).

Application

The application (bigdatamon.cloudfoundry.com) is supposed to identify the most common authors of Big Data related tweets. It makes periodic search queries to Twitter for a set of pre-configured Big Data keywords, and, store the resulting data, along with authors' metadata, in a local DB. Subsequently, the user can make a period selection, upon which, the app performs a MapReduce query in the DB and displays the most frequent authors along with some rudimentary summary information and their tweets for that period. All the information obtained from Twitter is linkable back to the original source on the UI.

There is more information about the underlining architecture of the application, as well as the entire source code in a public project on GitHub. This is by no means a production application. There is a lot of missing (test?), but, it did what it was supposed to....fulfill my development nostalgia.

Three Killer Big Data App Opportunities

2012-02-17T09:47:00.000-08:00

Unless you’ve been living under a rock for the past year, it is hard to imagine that you’ve been oblivious to the hype of Big Data. Almost synonymous with this concept, and perhaps the main reason for its current buzz, is Hadoop. This open-source, scalable, distributed data storage framework from Apache did not create Big Data, it existed many years before it, but what it did do is super-size it on a scale unachievable by its predecessors.

But here within lies the problem…the Hadoop framework, as powerful as it is, does not deliver any real value—it is an empty shell. Its Big Data success is an ability to leverage the Big Data framework in a form of game-changing applications.

While many startups managed to cash-in on the Big Data hype by raising large amounts of money to make the Hadoop framework easier to deploy and manage, we have not seen many truly innovative solutions that deliver the raw power of Hadoop in readily available solutions.

I am not oblivious that in the hands of talented data scientists, Hadoop can deliver great business insight. But, until we address the very apparent shortage of this kind of talent, we must capitalize on delivering new game changing applications "Killer Big Data apps," if you wish. These are applications that will leverage the unrealized potential and deliver a next-generation Big Data solution that are both a necessary to today's business, and, are desirable by end users.

The opportunities of these game-changing applications, in my mind, fall into three groups:

Leverage Unused Data

This is data that is already collected but not harnessed to deliver any business value. One example of data that is currently unused is the area of DevOps. The two main products in this field that I am familiar with are Chef and Puppet. With the recent popularity of these tools, and the increasing number of devices (nodes) they can manage, these products are in a very unique position of knowing a lot about the environments in which they reside. Here is the kicker though, in today's version of these products, much of that visibility is lost, discarded as soon as it goes out of scope. Can you imagine the opportunities to leverage this data to drive new value through pattern recognition or usage modeling/forecasting?

Drive New Value From Summary Data

This one is close to my heart, as I spent close to nine years in developing environmental, health and transportation data management systems. Often in this area, large data sets are extensively summarized before any meaningful analysis at the county or state level. By the time this data gets available on the national level, we already lost much of its meaningful insight. The opportunity here is to not only leverage the detail, but, augment it with related data-sets to provide fuller view.

Harness Heterogeneous Data Sensors

Whether we like it or not, we, and many things around us, are "nodes" in elaborate sensor networks collecting Big Data. I know this sounds scary, so, let me explain. The devices we carry, the vending machines we frequent, the thermostat monitoring the temperature in our homes, all these have sensors; and increasingly numbers of these devices collect actionable data. Traditionally, we think of these data sources as silos: "If vending machine indicates it ran out of soda, refill it.” The opportunity in this area is to develop the capability to augment already existing points of potential data collections, mobile sensors if you wish. High velocity sensor devices that are capable of acquiring multi-media input (visual, air & temp sample, motion indicator, etc.) and relaying this data to predefined aggregation points. These devices will deliver new data points from existent infrastructure allowing for new pattern recognition and the development of new delivery models.

I realize that the level of effort to develop such applications is considerable, but, what would you rather work on—game changing solutions or yet another Zynga game?

Analytics for the Masses – How to mainstream Big Data and educate much needed waive of new data scientists

2012-01-13T06:02:00.000-08:00

Big Data is a hot topic, who knows, may be even hotter than the Cloud itself. But, there is a notion of voodoo magic there that prevents wider adaptation and limits potential flow of talent.

Now, many companies have done great strides towards increasing the accessibility of Bid Data. For example, the recent EMC Greenplum UAP announcement, demonstrating Chorus as a human interface to data (structured or not), was a perfect example of how to make the Big Data analytics more accessible. But, all these efforts are targeting a very small subset of the potential users. I believe we are now in a unique position to bring the social aspects of data analytics for even wider audience.

Challenge
When I talk to people about the value of Big Data, especially those outside of the large enterprise circles, I often am faced with the same two arguments:

Too complex entry point - General inability to evaluate Big Data application platforms without a substantial upfront investment in development resources and/or required infrastructure
Really big hammer, but, where are the nails? - Even those who do understand the value of Big Data analytics enough to invest, struggle with practical application and lack of general vision for valuable use-cases

Forester estimates that today businesses, on average, utilize less than 5 percent of available data. Why so little? It is not ignorance, resulting from lack of visibility of that data; the rest is simply too expensive/complex to analyze. Businesses will need to be educated in seeing value in that data, and, have affordable platforms to harness the other 95 percent.

Solution

With the growing popularity of Big Data appliance and the increasing availability of Hadoop and Pig implementations like the one released by LinkedIn representing their homegrown PageRank solution based on DataFu library and some Pig's User Defined Functions, we are now in the unique position to address both of these Big Data misperceptions.

Recently, the US government has begun to open source pieces of the Data.gov platform. The raw content aggregated from the lowest levels of state, county and municipal governmental agencies across the country represents a unique opportunity for Big Data evangelism, a chance to bring analytics for the masses.

So far, these PBs of good quality data have only been understood on a summary level. Robust, user-friendly, publicly accessible platforms enabling immediate unadulterated access to that data at detail would:

Demonstrate the power capabilities of the underling Hadoop-based platform
Provide a common test-bed for new products and features
Create breeding ground for next-generation of data scientists
Contribute to Big Data liberation story

The educational aspect of such platform can’t really be overstated. This is where Big Data exploration and socialization will occur -- by allowing data scientists to manipulate data, collaborate with colleagues, and readily share their findings. I do not believe there are analytical platforms out there today that meet these requirements. Could perhaps this platform serve as training ground for the Big Data practitioners of tomorrow?

Content
While the bulk of the initial data-set would have to be sourced from Data.gov and Census data, additional self-propagating sources could easily be preconfigured in the system to make the solution more appealing to data scientists day one:

User-defined custom sources (XML or JSON Web Service & static content like CSV)
Website tools such as WHOIS, bit.ly, and Compete
Services that use email addresses as search terms, including Github
Finding information from just a name, with APIs such as WhitePages
Services, such as Klout, for locating people with Facebook and Twitter accounts
Search APIs, including BOSS and Wikipedia
Geographical data sources, including SimpleGeo and U.S. Census
Company information APIs, such as CrunchBase and ZoomInfo
APIs that list IP addresses, such as MaxMind

These data sets, when combined with features like Real-Time Collaboration, Data Search & Exploration and On-Demand Visualization, would make this platform an immediate hit and assure ongoing value of this solution over time.

Conclusion

This social analytics platform will not only make the Big Data more accessible to the masses, but also educate data scientists and other experts about the value of Big Data solutions. By creating a publicly accessible environment where the general public can analyze the large and current data-sets, we will further the adaptation of Big Data concepts and gain access to large numbers of data scientists who are currently outside of the reach of the still evolving market.

The pitch is simple: "Go ahead, scrape, parse, analyze and integrate open and proprietary data. We’ve got huge collections of scientific, social and geo data ready for you to harness. We are excited to see the new insights you will develop."

Big Data more than just size, true value in business application and real-time analytics

2011-11-08T02:30:00.000-08:00

When I talk to people about Big Data, especially those not exposed to the constant onslaught of NoSQL hype, I often get asked the same question: "Is it really that much better than database?" Assuming this a comparison question to the traditional RDBMS, I want to put down some thoughts on the subject.

The major reason why Big Data, and by extension NoSQL products like Hadoop, are so popular right now, is because there simply is no other option to deal with large, unstructured, data sets. Now, here is where many fail to grasp the significance -- it is the size (petabytes) and the unstructured format that is the core problem. Sure, given enough time and money RDBMS could maybe handle it, but, it would be expensive, time consuming and not agile enough to adopt when the business questions change.

That's why a whole new breed of products are emerging that build on top of Hadoop-like solutions that address the increasing proliferation of data. There are some like Cloudera that manage to deliver to market pretty impressive offerings. The business opportunities however reside more in visualization of the on-going stream of data. It is the decision-making impact that is missing from the Big Data offerings of today. Businesses today must gain new efficiencies in the midst of a very competitive market; the need to drive knowledge from their data to make smarter business decisions.

Those successful in capitalizing on this opportunity deliver applications that use big data technologies as a means to improving business processes, not using "Big Data" as a core value proposition of their solution.

Mobile space is a section of the big data market that many believe to develop faster than others. It will certainly require innovation because of the sheer number of devices and the bi-directional content they generate/require. There are more phones than people in the United States! Now add to this the exploding tablet....

A couple of net-new, real-world examples of companies that do understand the difference in the market today: 33Across in advertising and ipTrust in application security. Both build on Big Data as a tool to address real business problems.

On a bigger scale, very soon Big Data will stop being "big" and become a fundamental part of doing business. Accelerating that transition will allow companies to gain advantage. I am talking about solutions that will be able to absorb the ever-increasing stream of data and delivering a real-time analytics to process this information, and thus, enable instantaneous decision-making and smarter way of doing business.

These ideas are not too far in the future. Various big-data hardware optimization techniques and advances in scalability will make true, real-time analytics real.

Corporate IT Oblivious To New Role, But, Their Renaissance is Coming

2011-11-01T16:51:00.000-07:00

Enterprise IT organizations of today are largely unaware of the different ways their space is being impacted. IT personnel is increasingly unprepared to handle their shifting roles, and as a result, will become increasingly less relevant to the overall value of their companies.

Gross overgeneralization, yes, but so true. This post is inspired entirely by my recent work experiences, so, let me vent out a little here while I try to avoid being specific as to not get myself into trouble.

IT managers are latching onto the promise of virtualization as purely a means of lowering the cost of doing business as usual. At the same time, Gartner estimates that while business units strive for more agile processes and technology delivery solutions, IT will have increasingly less effective control of their IT spending which is already down to 25%. As IT organizations have larger amounts of their resources locked into assuring the "reliability of mature technologies", as much as 70% of their overall capacity, they are simply unable to address the new technical challenges facing their companies.

Now, I do realize there are many organizational and market challenges that IT faces every day. However, I am convinced that it is their unwillingness to challenge status quo that prevents them from becoming the responsive thought-leaders of enterprise that they once were.

Peter Sondergaard, senior vice president at Gartner and global head of research recently said: "Mature technologies are code for obsolete. You must dare to employ creative destruction to eliminate legacy, and selectively destroy low impact systems." Now, with predictions like "Mac have no place in the enterprise" and "Private Cloud is the last resource," I wouldn’t placed too much value on Gartner’s advice, but, Mr. Sondergaard is onto something here.

It is the "reliability of mature technologies" that suck the life out of IT on daily basis. I recently spoke with an IT manager whose team was almost entirely consumed by the maintenance of two systems: Exchange and SharePoint, while at the same time playing an endless game of cat-and-mouse preventing users’ access to SaaS solutions like Springpad or document collaboration sites like Box.net. Here is the kicker -- their project for next year is to virtualize these "mature" applications! I can think of no better illustration than the Titanic's orchestra making sure their music sounds just right while sinking to their demise.

Much has been written about the Consumerization of Enterprise and the Post-document Era. These events are breaking-up the internal IT monopoly and shifting their role from provider to broker. This can be illustrated by the increased use of single purpose applications which are simple to use, easy to develop, and really good at one thing. Successful enterprise IT will develop their own Cloud strategy and evaluate their every initiative against it, instead of being endlessly driven by their products or technology. Watch, the renaissance of IT is coming.

Federated not Balkanized - The Future of Data and Its Current Cloud Challenges

2011-10-27T06:34:00.000-07:00

As a long-term Cloud storage user I recently wanted to re-evaluate my options. New content management providers became available and I wanted to make sure I wasn’t missing on the new shinny tech out there.

As I was considering the pros and cons of each option, I realized the apparent shift in my personal attitude towards cloud data storage over last few years. My concerns used to be solely with security. Now, while the data security is still critical, I am much more interested in data access, ownership, integration and its control.

Many people talk about how the recent consumerization of the enterprise, where the lines between our personal and work data are being increasingly blurred. But nothing brought it home for me as much as what I saw during the recent VMworld in Las Vegas where Steve Herrod, CTO from VMware, was talking about the new content storage solution code-name Project Octopus. It provides Dropbox-like experience to corporate users while preserving the IT control over the company content. Before he gave a demo, Steve has asked how many of the 20,000+ attendants currently use a consumer cloud storage solution like Dropbox at work. About a half of the audience raised their hands. These are some of the most network-savvy, security-conscious users of the Cloud industry!

Should we be surprised? How many of us currently use our personal devices at work? More importantly, how many businesses are actually OK with that? So, how did we get here? More importantly, how must we deal with this exponential growth of data while preserving the necessary level of control?

For starters, we need to realize that with the proliferation of SaaS-based solutions, we are giving up more and more control over our own data. When was the last time you read the agreements for which we so nonchalantly check the “I Agree” box when signing up for new Web-based app?

But SaaS is not the problem here, as we are moving to the post-document era and increasingly larger amounts of our content is managed in the public cloud, we do not necessary need to give up control. Rather, we need to start thinking about a more federated storage model. Now, I know the concept of “storage federation” get some people really excited, but, what I am talking about here is a model focused not on private vs. public storage but a fabric that is intimately aware of the data content, its origin, as well as its access and retention policies in context of user’s current identity across all providers.

Tim Berners-Lee, at his recent keynote address at RSA Europe, talked about the demand for control of storage. Not as something we will need in the future, but rather, as a clear and present danger of impacting the cloud adaptation and balkanization of our data resulting in our inability to leverage its real value.

Analyst firm IDC now claim the growing volumes of cloud storage providers will lead to combined storage spending of $22.6 billion by 2015. We must figure out a scalable and secure way of controlling all that storage really soon, otherwise the promise of cloud value will be overshadowed by increased loss of control and eventually lesser security.

The Shifting Sands of IT Infrastructure Management

2011-10-21T08:15:00.000-07:00

Ever since his keynote at VMworld in Vegas, I'd been following Steve Herrod's posts on the subject of IT management, specifically in the area of Infrastructure. The need for higher levels of automation is obvious, however, the need for embedding management in the infrastructure in order to achieve that level of automation represents a huge shift in my opinion from how we are currently viewing cloud infrastructure.

Take away for me from Steve's latest post on the subject:

Automate everything in sight as this is the only way to achieve the efficiency and economics of cloud computing
Change IT mentality from provider to service broker
Get up to speed on vCenter and vFabric Application Management suites

Joe Tucci, CEO, EMC Talks Information Processing in Seattle

2011-10-20T10:21:00.000-07:00

Joe Tucci spoke in Seattle last night. I did not go, but, thanks to a pretty good TV feed (45 min long) from the Distinguished Lecturer Series at University of Washington Huskies I was able to take some notes on his view on information processing over the next decade and where EMC will focus:

Recent IDC report puts info growth at 44x (0.9 zettabytes to 35.2) & 90% unstructured
Most companies spend 3/4 on maintaining existing infrastructure (will be true for 10 years)
3D movie is about a petabyte with all camera angles and footage included
The average company is attacked 300 times per week
IT staffing will increase less than 50% in next 10 years but the data under management will grow much faster.
EMCs Mission: To lead customers towards a hybrid cloud
x86 based private clouds and hybrid clouds
EMC still owns 80% of VMWare (was that in question?)
There are now more than virtual machines shipped than physical machines
Killer app: Real time data analytics
EMC 5 year M&A plan: roughly 50% of investments in R&D (10.5B) and 50% in M&A (14.0B)
EMC has 14,000 sales people
EMC is now 152 in fortune 500

Revenue is $17B
Free cash flow: $3.4b

EMC Modular Greenplum Appliance Supports Multiple Data Types

2011-10-20T06:22:00.000-07:00

EMC acquired Greenplum in 2010, and it wasted no time in developing an EMC Greenplum Data Computing Appliance (DCA) combining EMC storage hardware and replication and recovery options with Greenplum's massively parallel processing (MPP) database. EMC's Data Computing Division is expanding on Greenplum's deep support for in-database analytics with partners including SAS and MapR.

EMC introduced its own distribution of Hadoop software in May, and a Modular DCA set for release this fall promises to support the Greenplum SQL/relational database as well as Hadoop deployments on the same appliance. With Hadoop, EMC addresses analysis of truly big data like clickstreams and unstructured data such as social-network comments. The Modular DCA will also support high-capacity storage modules on the same appliance for long-term retention of records to meet regulatory mandates.

Sources:
EMC Tailors Storage Systems For Big Data
Big Data: Informatica Tackles The High-Velocity Problem
Big Data A Big Backup Challenge
12 Top Big Data Analytics Players

Hilton listens to its customers feedback

2011-09-28T16:43:00.000-07:00

Upon late arrival in DoubleTree hotel in MA this week I was surprise to find out virtually no vegetarian choices on their menu. When I posted about it on Twitter, I was surprised with an almost immediate apology from @HiltonOnline informing me they are goanna contact the hotel to address my concern directly with the hotel manager. Subsequently, upon arrival from work the next day, I found a personal letter from the manager with some Trail Mix and an invitation to call their chef for “off the menu specials” any time. I can only wish more businesses responded to on-line complains that way. Well-done Hilton. You have my future lodging business, for sure!

Is that time of the year again; get your polka on

2011-09-23T10:35:00.000-07:00

Polish music, dance, food and drink abound this weekend at the Polish festival on Failing Street between the two historic buildings of the Polish Library and St. Stanislaus Church (3900 N. Interstate Ave), Saturday 11:00 AM-10:00 PM and Sunday from noon to 6:00 PM.

Highlights include non-stop stage performances, a polka contest on Saturday and Sunday, and obviously polish food and imported beer.

More info at portlandpolonia.org

What really makes storage "cloud storage"?

2011-07-11T06:42:00.000-07:00

With the recent announcements from Amazon and Apple about their cloud storage offerings and the increasing proliferation of smaller offerings from companies like Dropbox, Box.net and SugarSync, it gets increasingly harder to distinguish a basic remote storage offering from a true cloud-based one. Not surprisingly, every one of the providers has their own definitions of cloud storage.

The National Institute of Standards and Technology (NIST) defines cloud storage as:

Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.

Well, not much help there. So, in pursue of a more descriptive and definitive list of attributes of cloud storage I reviewed the recent report from ‘Storage Strategies NOW’. The following list seem to best describe the characteristics of cloud storage:

Resource pooling and multi-tenancy. Multiple customers can use the same storage infrastructure. Resources are pooled and multiple cloud resources (servers, storage and applications) are assigned and unassigned to customers as needed. Resources are location-independent and transparent to the user — in most cases the customer does not control or need know about the actual location of a particular resource. A customer‘s data is protected from access by other customers. In the case of private clouds, multiple departments or business units within an organization share infrastructure and applications.
Scalable and elastic. Virtualized storage that is for all practical purposes infinitely and immediately expandable. Storage can be expanded or contracted as needed easily and efficiently.
Accessible via standard Internet APIs and communications protocols including HTTP, FTP, XML, SOAP and REST.
Service-based. Customers typically have no capital costs (CAPEX) and pay for storage as a service (OPEX).
Pricing is normally based on usage. Customers typically pay a per-gigabyte rate for upload and download and a per-gigabyte fee for monthly storage. In addition, some providers charge for each data access request based on reads, writes, etc.
Shared and collaborative. Because cloud storage is usually accessed via the Internet it allows data access from multiple locations and multiple users.
On-demand self-service. Customers can typically manage their storage service using some sort of management console. Customer service agents are usually only needed for problem resolution and similar tasks.

In short: data is deployed to cloud storage either through Web-based applications or through Web services application programming interfaces (APIs). Web-based applications are often used for manual access to data or management functions, while APIs are used for more automated or transparent approaches. Since standard APIs and communications protocols are used, the physical location of the data becomes irrelevant, since it can be made available virtually anywhere via the Internet or private network. This also means that cloud data can be easily replicated to multiple locations for fault tolerance, high availability and other purposes, often without involvement of the customer.

Source: cloudtweaks.com

Granularity of Multitenancy - The New True Differentiator of Cloud Service Providers

2011-07-04T07:18:00.000-07:00

In today’s competitive service provider environment, multitenancy can no longer be an empty promise; it is an absolute requirement for any true efficient cloud offering. A multitenant provider can host multiple organizational units and multiple logical groupings of computing and storage capabilities assigned to various combinations of users and organizations. That means that each discrete computing environment must be invisible to users not associated with that environment and must not be able to impact other computing environments.

Let me be clear, this is not an easy thing to do in general, and even harder at scale!

The three areas where multitenancy is still rarely demonstrated in any granularity are:

Provisioning orchestration
Resource management
Operational monitoring and reporting

In the marketplace where more and more providers must compete on cost, the ones that deliver higher granularity of multitenancy are able to maximize their infrastructure and thus drive higher profits. Furthermore, those who can provide a reliable multitenancy will subsequently attract more prestigious group of clients who will pay extra to manage their architecture in such environments. Why? security and compliance! But, that a subject for another post.

Disclaimer
While I work for EMC, the opinions expressed in this post are my personal. They are not approved in advance by EMC and do not necessarily reflect the views and opinions of EMC.

Chasing the Cloud: Why I joined EMC

2011-06-20T05:11:00.000-07:00

After nine years and almost 50 projects, last Friday was my last day at Windsor Solutions, Inc.

As a technical leader, I’ve always been passionate about innovation. At Windsor I’d been able to express this passion by architecting custom software for government agencies. Whether it was a Facility Warehouse, Exchange Network Services or Emergency Responder Client, I’ve always aimed to deliver an innovative and user-friendly solution.

While I am proud of the work I have done at Windsor, over the last year, I have started enviously observing the increasing momentum of Cloud Computing. To me, the cloud represents a fundamental shift in the way we deliver IT. It also represents an enormous opportunity. To pursue that opportunity, I am joining EMC Corporation; an undisputed leader in both public and private cloud services.

I'm excited to be joining the newly created Solutions Group and I look forward to making significant contributions at EMC as it continues to provide leadership in Cloud space. With the product portfolio (VMware, SpringSource, RSA etc.), and multiple partnerships (Intel, Cisco, VCE etc.), EMC is right at the epicenter of the Cloud transformation. This is where the solutions of tomorrow are being developed today.

About EMC
EMC provides information infrastructure technology and solutions such as unified storage, content management, security, virtualization, backup and recovery, private clouds, virtual desktop infrastructure, efficiency, automation, and archiving.

EMC is headquartered in Hopkinton, MA, and has a global presence with approximately 43 thousand employees. The company’s profits for 2010 were $1.97 billion on revenue of $17 billion.

EMC was recently ranked 3rd on Fortune’s list of the World’s Most Admired Computer Companies and was recognized in 2010 for its IT Proven Project with a CIO 100 Award, which awards business that are creating new business value through innovation of technology. As a testimony to employee satisfaction, EMC was also ranked 33rd in the Top 50 Best Places to Work of 2010 by Glassdoor.com.

VPLEX Geo Is Out, Now if only someone knew how to solve the geopolitical problem it creates

2011-05-09T13:29:00.000-07:00

With the VPLEX GeoSynchronicity 5.0 announcement today (yes, it is available today) you can finally have that active-active geographically dispersed storage setup across large geographical area. Did I say it is ~50ms?

At first looks it appears to be nothing more than a distributed, coherent, transactional cache of VPLEX which virtualizes external storage and makes it active/active across geographic boundaries. However, when combined with an apparent extensive API (still reading the details) quickly brings all kind of applications to mind.

I'll let you come up with your own sky-is-the-limit scenarios, but, here is one I have been faced with in the past while working with persisted data at government agencies: Federated, Geopolitically-aware storage provider that enabled capturing system state from many VMs across multiple countries with exact synchronization which presents the storage to the VMs as they are running in the cloud – without the dataset ever actually really existing at the service provider. Still formalizing this idea in my mind, but, the privacy regulations of EU is one place where I can see this quickly becoming a game changer.

Video: MOV | WMV

Why Narwhal is not my type of a whale. Reasons for going back to Meerkat after a day with Ubuntu 11.04

2011-04-30T10:33:00.000-07:00

UPDATE: I have figured out how to deal with the overheating issue. A few new settings on the /etc/thinkfan.conf. Since I was able to work around my dislike for Unity and these new scroll-bars, I decided to upgrade after all.

Ever since 8.04 I have always run the latest version of Ubuntu. So, when 11.04 was officially released this Thursday, I managed to carve out some time this weekend for what I expected to be a quick upgrade. To be honest, right from the start I was not very excited about Unity, but, I was committed to giving it fair shake.

I decided to try an upgrade as my not-even-six-month-old machine (Lenovo x201 i7 solid state and a ton of RAM, very nice) was set up just right and I really did not wan to go through all the configuration and special touches again.

That's where the issues started. Even though my system was up to date, I kept getting dependency problems when running the upgrade tool. Apparently the tools assumed I was still running Open Office. Since I have replaced it with LibreOffice long before upgrade there were some language dependences the upgrade tool simply could not deal with. That bug has now been classified as critical so I suspect will be available any time now.

At that point I was kind of committed, and since I had a good backup of the entire system, I decided to do a fresh install.

Well here is where the fun really started. Long story short, there were a few things why I decided to go back to Maverick, but the three main ones are:

Processor Overheating – there appears to be an issue with the Lenovo machines overheating if not configured with an additional thinfan utility. I found a bunch of people experiencing the same issue on x201s where the fan does not kicks in time and the laptop reaches 100C when it shuts itself down to prevent total meltdown. I installed thinfan and configured with with the exact same settings I was using on Maverick. Good news that prevented the overheating, but, it kept the fan running even when the temps where below 40C. I never had this issue on any other version of Ubuntu. There are scripts to work around this issue, but, for me this was strike #1.
UI – During normal day I use 8-9 apps at most. I have shortcuts assigned to all of them (to lunch chrome for example I hit CTRL+ALT+B (“B” is for browser). Unity, while nice for the screen-shots, is kind of useless to me. I really wanted to like it. But, all my files are in in the cloud, apps I launch with shortcuts and configuration is a kind of one-time thing. Also, if you have to re-engineer how scrollbars work, you'd better have a really good reason. The current implementation simply falls short. Don't even get me going on the huge shadow around everything. Strike #2.
Incompatibility - So, even though I am not the biggest fan of Unity, I though, no problem, I just use the traditional Gnome UI, reset back to the “normal” scrollbars and everything will be OK. That's when I started installing some apps and was faced with interesting errors. VirtualBox debian package for example would get an error indicating that the it was not “well designed”. Yes, you can skip these errors, but for me, this was strike #3.

Now, I am sure all these issues will be resolved over time. I also realize that many of my complains can be attributed to personal preference. On the end of the day though, this was the worst upgrade experience I had since I started using Ubuntu on 8.04.

Now, any other OS and I would be stuck with rebuilding my machine. On Linux, I launched Deja Dup, connected my USB drive and went for a 40 min run. When I came back, I was back to usability, performance and most importantly the productivity of the Ubuntu I love. Long term, a distro switch may be coming up I am afraid.

Five simple steps to a perfect cup of French press coffee

2011-04-26T07:36:00.000-07:00

There is a lot of confusion as to the proper technique for making French press coffee. Thats why I was pleased to read somewhat of a confirmation of my own approach on a recent blog entry from James Hoffmann, a former World Barista Champion. You can see a video of the technique here.

The one thing I did not do is to "skim" the grounds from the top of the coffee before plunging. I heard it described as a "weirdly satisfying thing to do", especially when you let the grounds "bloom" in the press pot.

Here are the basic steps of his technique which supposedly originated from Chris Young the acclaimed Fat Duck chef:

The brewing ratio is critical. So using a scale to weigh the grounds and the water will make a remarkable difference to the consistency of your coffee. I like 70g of grounds per liter of water. My press pot will hold about 700g if water, so usage 49/50g of grounds to the hot water.
Grind size matters. For French press, the coffee should be ground coarse and have a uniform particle distribution (actually the distribution is trimodal, but that’s a tangent). Only burr grinders can achieve this. If you don’t have a decent burr grinder, have your beans ground for French Press by a reputable coffee shop.
Keep the brewing time consistent. I use 4 minutes for the grind size I use and will adjust the brewing ratio to find the ideal strength for my cup of coffee. Grind size, brewing ratio, and brewing time all interact, so adjusting only the brewing ratio helps me from getting confused when I’m dialing things in for a new batch of beans.
I will usually not cover the press during the steeping. I like to allow the grounds to "bloom" as much as possible. Anything that prevents this tends to yield an uneven extraction from the cake of coffee.
Skimming makes an amazing difference. I was shocked just how big this difference was when I first tried it, but it makes sense. The basic French Press design allows a lot of the "fines" from the coffee to pass through the plunged screen. These fines continue to steep in the coffee, resulting in very over extracted coffee with a bitter taste and a muddy mouthfeel. By skimming the cake of swollen grounds before plunging, you’re throwing out a lot of these fines, so you end up with less overextraction and a cleaner mouthfeel.

Source: FREAKONOMICS

VMware announces Cloud Foundry, Spring developers rejoice!

2011-04-12T13:16:00.000-07:00

Watching Cloud Foundry webinar earlier today I could not help but be impressed with the innovation VMware brought to their Spring Framework.

If you have ever worked with Spring before you are already familiar with the basic principles of that framework: simplicity, consistency and portability across platforms.

Building on these goals, VMWare today delivered further enhancements in their Cloud Foundry offering to make the lives of Spring developers even easier. But don't trust my word, consider these three features:

Deploying a Spring application to the cloud is as simple as dragging and dropping within SpringSource Tool Suite, and even when building an application to run in the cloud, developers can take advantage of the productivity gains enabled by Roo and Grails exactly as they normally would.
Projects like Spring Social and Spring Data embrace innovative technologies such as Twitter and non-relational data stores that are increasingly popular for cloud-based applications, and they do so in ways that are consistent with the existing Spring platform. Cloud Foundry provides services to support such applications. RabbitMQ will be available as a cloud service soon, so the same applies to applications that rely on RabbitMQ for messaging via Spring AMQP and Spring Integration.
The cloud is first and foremost a new deployment environment, and yet it's easy to create an application that can run in and out of the cloud without even swapping configuration files.

Today’s announcement is certainly good news for Spring, but Cloud Foundry goes beyond Spring or Java for that matter. VMware wants to serve developers, whatever their chosen language and framework. So Cloud Foundry is inherently cross platform. It supports Rails, as well as Spring, applications, on the same underlying infrastructure. Did I mentioned Node.js? Ah yes, but that is a subject for a different post.

Source: Springsource

Enhance your cloud-based solution's security by encrypting data on the client with the latest AWS Java SDK

2011-04-10T17:20:00.000-07:00

While there cloud providers continue to enhance their compliance and security offerings, the fact is that if your solution persisting any data to a cloud-based storage system you are at the mercy of the storage provider.

The newest version of the AWS SDK for Java alleviate the data security on the persistence layer somewhat by providing an optional client-side encryption. Once enabled, the SDK will automatically encrypt data before sending it to Amazon S3, and decrypt it before returning it to your application. You have full control of the keys used to encrypt and decrypt your data and the keys are never transmitted over the wire.

This feature is implemented using a technique known as envelope encryption. Here's a diagram that should help to illustrate the concept:

Your calls to the AWS SDK for Java include a Master Key. This key is used to encrypt an envelope key that is generated within the SDK. The envelope key is used to encrypt the master key and the data before it leaves the client. The encrypted envelope key is stored alongside the encrypted data along with a description of the master key.

On retrieval, the encrypted envelope key is compared to the description of the master key. If they do not match, the client application is asked to supply the original master key. You can use this feature to integrate your application with an existing private key management system.

This functionality is implemented within the AmazonS3EncryptionClient class. This class is a subclass of the original AmazonS3Client with additional parameters and methods to control the encryption and decryption process.

The SDK provides a number of configurable options, including the ability to use either asymmetric or symmetric encryption. You can also choose to store the encrypted envelope key as S3 object meta-data or as a separate S3 object.

Source: Amazon Web Services Blog

The proliferation of virtualized devices creates new risks that can only be addressed at the root (hypervisor) level

2011-03-30T05:00:00.000-07:00

Virtualization is becoming increasingly ubiquitous in the data-center thanks to its numerous benefits and rapid return on investment. That's a good thing, right? Hardware consolidation, greatly increased flexibility with provisioning, better scalability etc.

Unfortunately the increased flexibility of virtualization, which enables so many of its benefits, can simultaneously increase security risk and complicate compliance initiatives. Failure to address these concerns can hamper efforts to virtualize critical applications.

To rain in the virtualization wild-west, Intel recently announced the TXT (Trusted Execution Technology) as a response to the Trusted Platform Module (TPM) specification published by the Trusted Computing Group and accepted as an ISO standard in 2009.

Building on top of that capability, software vendors like HyTrust offer now virtual appliances that can access the TXT status through the vSphere vCenter API and make decisions on controlling guest operations from the host server. In addition to that, HyTrust also offers network-based policy management for your virtual infrastructure that provides administrative access control, hypervisor hardening, and audit-quality logging to protect you from malicious, or sometimes just careless, insiders. And there are others now who also build new offerings or extend their current feature set to proof of hypervisor protection on top of TXT including: hardware root of trust, trusted virtualization environment, event management tools.

The proliferation of virtualized devices is only going to get worse (eh? better). Think in terms of evolving your security profile. Evaluate where you are today and educate yourself on the new hypervisor hardening options.

Sources:
“Hypervisor Security: Don't Trust, Verify” by Schalk Theron
http://www.hytrust.com

Tripwire: Compliance and Security-as-a-Service for MSPs

2011-03-24T10:16:00.000-07:00

Security-as-a-service provider Tripwire dropped me a line a few weeks ago regarding a new offering for MSPs and cloud service providers. Intrigued, I had a chat with Tripwire Chief Business Development Officer Dan Schoenbaum about the Secured By Tripwire offering, and heard about how the company is helping service providers help their customers clamp down on cloud data loss and meet compliance needs.

Unlike other aspects of IT, security and compliance aren’t “nice-to-have” as Schoenbaum puts it — especially when it comes to cloud services. And where many other vendors are developing identity and access management solutions, Tripwire is zeroing in on protecting against data breaches and making sure compliance needs are not only met, but that they continue to be met as time goes on. As you might guess, Tripwire’s biggest customers are financial institutions, retail organizations, and federal agencies.

Now, here’s where the IT channel comes in. IaaS offerings like Amazon Web Services’ can boast about the high security and compliance of its data centers, the end customer doesn’t get a whole lot of benefit when information’s flying back and forth between their offices and the cloud, according to Schoenbaum. But Tripwire’s MSP and cloud service provider partners can use Secured by Tripwire to lock those transactions down tight.

The added benefit for the MSP, beyond just that security, is the addition of upsells like ongoing compliance reporting, log management, permissions and ownership management, and so on. And it’s cheap, too, with Tripwire providing partners with the hostable software for free and only charging based on what the customers use. MSPs are free to set their own price, and thus, their own margins.

by Matthew Weinberger
Originally published at: talkincloud.com

Trusted Cloud: the trend to watch this year

2011-03-17T07:15:00.000-07:00

Trusted Cloud played a big part in in the Intel's Day in the Cloud presentation a couple of weeks ago here in Portland, OR. Almost all cloud providers are currently striving to enable more secure multi-tenancy without sacrificing efficiency, cost advantages or the overall usability.

As to prove my point, large IT solution provider, Harris Corporation along with RSA (EMC) have partnered today with Virtual Computing Environment (VCE) to co-develop and bring to market trusted cloud solutions. Together, the companies hope to accelerate adoption of more efficient and more secure IaaS solutions among commercial and governmental organizations by leveraging VCE’s Vblock cloud infrastructure and RSA-powered security.

What can we expect as part of this relationship? Apparently this strategic partnership hopes to develop best practices on how to migrate customer applications to the Harris Trusted Enterprise Cloud Platform, with Vblock infrastructure at its core.

Amazon Announces Virtual Private Cloud (Amazon VPC)

2011-03-15T04:55:00.000-07:00

Imagine being able to provision virtual network topology in the Cloud that closely resembles a traditional network in your own datacenter. You control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways.

Well, imagine no more.

With Amazon Announces Virtual Private Cloud (Amazon VPC) you can easily customize the network configuration. For example, you can create a public-facing subnet for your Web servers that has access to the Internet, and place your backend systems such as databases or application servers in a private-facing subnet with no Internet access. You can leverage multiple layers of security, including security groups and network access control lists, to help control access to Amazon EC2 instances in each subnet.

Additionally, you can create a Hardware Virtual Private Network (VPN) connection between your corporate datacenter and your VPC and leverage the AWS cloud as an extension of your corporate data center.

Other things you can do with Amazon VPC:

Create an Amazon Virtual Private Cloud on AWS’s scalable infrastructure, and specify its private IP address range from any range you choose.
Divide your VPC’s private IP address range into one or more public or private subnets to facilitate running applications and services in your VPC.
Control inbound and outbound access to and from individual subnets using network access control lists.
Store data in Amazon S3 and set permissions such that the data can only be accessed from within your Amazon VPC.
Attach an Amazon Elastic IP address to any instance in your VPC so it can be reached directly from the Internet.
Bridge your VPC and your onsite IT infrastructure with an encrypted VPN connection, extending your existing security and management policies to your VPC instances as if they were running within your infrastructure.

Source: Amazon AWS Website

Lack of interoperability number one reason for vendor lock-in

2011-03-10T20:09:00.000-08:00

In a recent post on cloudtweaks, Simon Ellis (Labslice), posed some interesting questions regarding vendor cloud lock-in and whether or not it represents a greater issue than cloud security.

Considering how much has been written on the subject of cloud security, I would like to focus on the apparent unease everyone is experiencing about the biggest cloud providers and their balkanization of approaches to virtualization and its management.

Now, I am convinced that the issue of cloud security will eventually diminish to level equal with other aspects of cloud infrastructure, platform and services. At the same time, sooner or later, the next generation of IT professionals will start to realize the potential for an even larger concern: vendor lock-in.

So, what is the main contributor to the risk of vendor lock-in today? In my mind the answer is clear: interoperability! More precisely, its lack amongst the cloud hosters. Until efforts like Open Data Center Alliance by Intel are successful, there simply will not be any cloud agnostic consoles and APIs.

Until then, it will be difficult to extract your implementations from one vendor and migrate to another. Simply put, as long as there is no interoperability amongst the hosters, we will not see any serious price competition. Amazon, Microsoft or Google are all too happy to have you stay right where you are as the price of migration is simply too high, or even impossible without a large re-engineering effort.

Yes, I know, there are some companies that do provide either migration utilities for Hybrid Cloud architecture (like CloudOps which I saw during the UP2011 conference) or some sort of middle-ware whereby the reliance on any specific hosting platform is replaced with lock-in to that middle-ware provider. Also, there is always an option where the entire infrastructure may be delivered in a form of a service, although even then some of those platform decisions are made by the hosters.

In short, let's hope these interoperability efforts work. Otherwise, in the few years we will find ourselves all dressed up with noplace to go.

Intel's Cloud Builders Program

2011-03-05T07:17:00.000-08:00

Intel, at their cloud conference here in Portland, Ore., has just described how and why cloud service providers (CSPs) need to make their data centers far more efficient.

During the opening, Intel's General Manager of High Density Servers, Jason Waxman, outlined the basics of Intel's Cloud Builders Program.

Intel, in conjunction with many major cloud industry players like Microsoft, Parallels, NetApp and Canonical, is developing reference architectures, essentially giving guidelines on how to deploy technologies from different vendors to create cloud solutions.

Right now, Intel has apparently some 50 different reference architectures based on the company’s own experiences with testing and deploying cloud partner solutions.

It can't be an easy change for a chip manufacturer to think in terms of services, not technology, but at the same time, Intel stands to make a lot of money in the data center if they can help their technology partners figure out better, faster, cheaper, and interoperable cloud services.