On Monday, October 3, the San Francisco Cloud Computing Club and Silicon Valley Cloud Computing Club hosted a joint session that was notable for any number of reasons. Someone described it as being involved in a Twitter / Clouderati twitterstorm, but face-to-face. Whatever it felt like, it was a great source of good thought and numerous, mutually respected points of view.
James Watters, of Silcon Angle acted as the MC and moderator for the session, and took it upon himself to capture the spirit of the session. He kindly invited me to add in my take on the meetup and we found ourselves with a jointly authored recollection of the conversation.
Here's a snippet. For the full version, take a look at Silicon Angle's site:
...
Q: What is the impact of internal private clouds on both enterprises and external cloud service providers? (Question submitted by Randy Bias)
James Watters: I got the ’scrunch face’ from Randy Bias, and James Urquhart when I suggested that private clouds need to adhere to public cloud standards to be really useful. I believe this is important because it keeps both the economics and usability innovations of the public cloud proximal to how users evaluate their internal private clouds–or as /Hoff said once, allows public cloud to be the forcing function for change.
If Private or internal clouds get really exotic, with proprietary in-house created management, deployment and consumption functions they won’t play as easily with the coming wealth of interesting solutions created on top of public cloud standards.
The other point is simple: this is what really smart companies already have today. If you sit down with the top investment banking firms in the country many of them have highly sophisticated JeOS optimized application deployment, scaling, patching, and management functions for autonomic computing–but its expensive to create this kind of in-house IP.
Amazon sources tell me that over 40% of their revenues are driven by third party applications built directly atop their API. If you build an internal cloud not compliant to public standards you may be left without access to this increasingly important ecosystem of innovation.
Rich Miller: For better or worse, the adoption of cloud-oriented computing by the Enterprise and Small-Medium Business (SMB) will start as a transition from ‘the way things are done now’ to in-house, on-premise clouds. IT organizations will get religion … in part through the widespread adoption of server virtualization … and start operating their in-house IT organizations like utilities: lots of self-service, pay-as-you-go, multi-tenancy. (Remember: cloud is an operating model, not just a technology model.)
But, in order to get there in an orderly fashion, the path will be evolutionary. And, in order to get there, some of the internal clouds will be mixed-bags of infrastructure-cloud offerings (especially in-house data clouds), platform-cloud offerings and application-cloud offerings.
To your point, James, one way in which coordination and compatibility with public cloud offerings may come about is if the management systems that the enterprise uses for their in-house operations are built to recognized ’standards’… those offered by the most powerful service providers (e.g. Amazon AWS) or technology providers (e.g. VMware). Over a reasonable period of time, the management of an in-house, on-premise cloud will morph easily into managing hybrids (both on- and off-prem). ...
The competition to claim the IaaS crown for enterprise computing has seen a major set of interesting events this week. As mentioned in this post (and its update) on Citrix and Xen.org, their preemptive announcement has a lot of people paying attention. Clearly, the industry expected something major to come from VMware's VMworld2009 conference this week, and they were not disappointed.
On Monday, VMware detailed the vCloud API in an open session. It's come as a mostly pleasant surprise. The API is closely associated with the DMTF's OVF/OVA standards, relying heavily on OVF/OVA as the means of representation and description of the virtualized solutions. Some initial reactions, based on an admittedly superficial review: (I reserve the right to add to this, and reverse myself on any of these pronouncements.)
[Geek alert] Apparently, when a network resource request is expressed by the (vApp) template, the IaaS provider can return an existing network or dynamically generate a new one that meets the requirements of the request. However, the network resource name that's returned implies provider-specific semantics. This effectively diminishes the out-of-the-box portability of solutions that rely on vCloud API. [End geek alert]
From first reading and reviews by people I respect (such as William Vambenepe and Rich Pelavin, co-founder of Replicate Technologies) the vCloud API 'does the right things,' on most of the points where it is definitive. There are clearly aspects of the specification that had to be left for further refinement. There's a slightly 'hurried' aspect to the spec, but I believe we should be thankful that they did not feel compelled to do everying in the first go-round, thereby making a mess that has to be cleaned up later.
What's interesting to me is the approach they've now taken with respect to making vCloud API 'the standard.' Coincident with it's release to the public, VMware has submitted the vCloud API Specification to the DMTF for the purpose of making it the Cloud API standard. VMware has found real value in its participation in DMTF over the past few years. Both the quality and general support of OVF/OVA as a DMTF standard proves out the upside to this standardization strategy. Immediate submission of the spec to DMTF is in stark contrast to the approach adopted by Amazon Web Services, which has published the details, but has kept it in a somewhat nebulous (if you'll pardon the expression) legal status. As Sam Johnston (@samj) points out in this post to the CCIF group, the intellectual property issues related to EC2's API are pointedly going in another direction, with specific restrictions to use of the API only with Amazon's own services and patents pending that relate to the API.
So, what about the commercial impact on the services industry? At a special event held yesterday at VMworld 2009, Paul Maritz made the claim that over a thousand service providers are now in position to offer VMware-ready cloud services, based on the vCloud API. Quick to reiterate that message, companies like AT&T, Savvis, Verizon Business, Terremark, Bluelock, Hosting.com, Logica and Melbourne IT announced immediate (or real-soon-now) launches of services under the VMware vCloud (TM) Express program. Software companies building on the vCloud API that announced include Aptana, Cloudera, CollabNet, CohesiveFT, EngineYard, ParAccel, RightScale, rPath, SpringSource, Terracotta, TIBCO, and Zend. Whew.
So, does having this kind of support mean that VMware is the winner as the basis for enterprise use of cloud services? Not yet. It's way too early to tell. At this point, we need to watch the players take the field and start the competition. Let's just see what enterprise users actually use, and for what purposes. To my mind, the place to focus is on the software companies. To the degree that cloud applications get built using the vCloud API as the approach of choice, that will be the measure of real leadership in this market.
The Reg's Timothy Prickett Morgan has posted Xen packages build-your-own-cloud kit and it adds some needed clarity to the upcoming Xen/Citrix announcement.
Simon Crosby's quoted as stating the goal of delivering to market a standardized stack for a cloud-based deployment of the Xen hypervisor. Prickett Morgan makes the point that it's an open source vertical platform, but clearly should not be considered a turn-key solution.
The Xen Cloud Platform does not include tools for creating, provisioning, monitoring, or managing a cloud. Rather, it is a complete infrastructure virtualization stack that companies building clouds can standardize upon.
The article goes on to point out that it's not only 'free' but hackable open source. It brings up the issue that GPL-based open source licensed code has an interesting loophole ... Since a service vendor's revised/enhanced/hacked code is not being re-distributed (but used solely for the provision of services), the license does not require the licensee to return the enhancements to the open source community. What isn't clear from the article is the licensing regime under which XCP will be offered.
Another point of clarification is the distinction Citrix is making between XCP and the Citrix Cloud Center (C3).
And this future product will be distinct from the Citrix Cloud Center (C3), formerly known as XenServer Cloud Edition, that Citrix pitched last year and tweaked when it started giving away XenServer for free this past February. ...
The Xen Cloud Platform is not C3, but it will include some storage management, chargeback, and other features that Citrix created for C3 or the Essentials for XenServer tools that are necessary for cloud providers. The cloud stack includes the Xen hypervisor, with support for either Linux or Windows instances inside of its virtual machines. The stack also includes a domain 0 Linux installer for the Xen hypervisor that is pulled right from the kernel.org site where the Linux kernel lives. ... Citrix will open source storage features it has created to link into disk arrays to do volume management, snapshotting, cloning, and such, and chargeback and other features to cope with usage tracking will be added to the stack as well by Citrix.
Of major interest is the approach to a distributed virtual switch infrastructure. The article notes that XCP will include the Open vSwitch that's available under the Apache 2 license., and is supported by the Citrix XenServer 5.5 hypervisor. This is not yet a full OSS solution, and Citrix will be under pressure to either release to open source some of their virtual switch technology, or Open vSwitch will need to light a fire.
Finally, the support of DMTF's OVF is reiterated.
According to eWeek and Shannon Snowden, Citrix Systems and Xen.org will be developing a 'full-blown cloud computing platform that will rival VMware's vCloud offering.' I'm not yet sure what this means, because there are still piece parts of VMware's vCloud (particularly the details of the vCloud API) that are yet to be revealed. That said, the interview on Snowden's Virtualization Information seemed to have the right elements: portability through support of OVF (YAY!!), commitment to DMTF standards, XenMotion workload migration between datacenters and clouds, extended virtual networking infrastructure, and cloud-scale virtual storage infrastructure.
We discussed the the expected impact to Citrix and XenServer. Both Simon and Ian (Pratt) think that having a bigger footprint of XenServer is good for Citrix and ISVs in general because the (Xen Cloud Platform) XCP won’t necessarily be focused on the management layer, but the foundational components to having a stable, functioning cloud platform. After all, Citrix is already providing XenServer for free.
In fact, the orchestration and management capabilities of open source projects Eucalyptus and OpenNebula.org as well as commercial offerings from vendors and cloud providers will integrate with XCP since these projects are Xen-based already.
Simon said the plan is for Citrix Essentials to work with XCP, so this makes business sense to me. Citrix gets more XenServer in organizations that already are running Xen to power their clouds and have an opportunity to sell more Citrix Essentials.
For me, the early prize in the contest between VMware and Citrix is the cloudbursting title. This would incorporate three elements:
555 KUBIK: HOW IT WOULD BE, IF A HOUSE WAS DREAMING
3-D projection on German building created by UrbanScreen
Relax, Bloggers. The AP Isn’t Out to Get You
The level of nervous blogging and twittering following the New York Times article , including Jeff Jarvis' How (and why) to replace the AP , has been pretty extreme. I was reassured to see Ryan Chittum's post in the CJR , which he wrote after actually going to the source... the AP. Now, I'm sure there are those in the blogosphere about to accuse the AP of spinning and damage control, but I'm tempted to take to heart this quote from a Senior VP of product development at the AP:
“Are we going to worry about individuals using our stories here and there? That isn’t our intent. That’s being fueled by people who want to make us look silly. But we’re not silly.”
Russian Programmers Petition Russian Government for a Holiday
Russian programmers have turned to the Administration of President (of the Russian Federation) Medvedev with a request for the establishment of professional holiday... Day of the Programmer. September 13 (the 256th day of the Gregorian calendar) is being promoted as the official Programmers Day. (Translated article from C-News )
In May, Christofer Hoff wrote a very interesting post about one of the unintended consequences of enterprise use of cloud services ... the serious and costly impact of right to audit clauses in the service contracts and their impact on *aaS. He posits that the enterprise customer now does make much heavier use of the RTA. So distinctly different is the level of use, that it's generating exceptionally high costs for the *aaS provider.
The sense I get (from Hoff, the comments, and from speaking directly to cloud service providers) is that the RTA clauses have been just that... clauses in a contract, with very little actual planning and preparation by the individual service provider. Thus, every invocation of the right ends up being a one-off project, distracting the provider, and diverting precious engineering and operations resources. (At the time, I commented that it sounds like a commercial opportunity. )
Yesterday, Hoff posted an extension to the concept, suggesting a security API for Cloud Stacks. I was 'off the grid' most of the day, but discovered when I reconnected that it generated a huge volume of twitter traffic. The post posits a sensible consideration: compliance structure commonality suggests that they could be 'built in' to a security control model. The result would be open API(s), which would be (voluntarily) implemented by the providers of *aaS stacks. This would provide for an open architecture for scanning a provider's offering for network vulnerabilities, as well as configuration management, asset management, ...
This way you win two ways: automated audit and security management capability for the customer/consumer and a a streamlined, cost effective, and responsive way of automating the validation of said controls in relation to compliance, SLA and legal requirements for service providers.
I haven't yet waded through the resulting twitterstorm, but it's formidable. It also gets into some strongly held, loudly presented 'religious' positions from a number of the security twitterati, which I don't pretend to follow comprehend.
This is a brilliant suggestion that's operationally difficult to get implemented over the broad range of *aaS providers. The pragmatics of getting this kind of adoption starts with getting agreement on a security automation regimen (such as SCAP). Having once figured which flavor of security automation, it seems that there will be a requirement for a trusted (open) third party -- a community effort or a vetted commercial ecosystem -- that prosecutes this through the 'cloud community.'
I expect some substantive discussions within the CSA community, and look forward to it as an interested observer and (potential) beneficiary. I'm really interested to see who picks up the mantle. Given the importance of BOTH the necessity of improved cloud audit and an open security management regimen, I hope that the Feds jump into this as a constituency with a real, economic need. If the government's production use of *aaS is as important as they seem to indicate, this should be given big-time attention by the country's CIO.
A number of people much smarter about data security than I have often made the point that one has to distinguish between passing a compliance audit and actually being secure. It reminds me of an education system that places so much emphasis on passing a competency test that the material being "learned" is completely secondary.
So, when I see reports of presentations like this, it makes me sad. It also makes me concerned for those who have their personal or corporate data protected by organizations focused on 'passing the test' as opposed to 'absorbing the material and putting it into action.' The point that Pironti makes in the presentation SHOULD be obvious.
If organizations continue to focus on security by compliance, he argues, the adversaries will continue to win as their attacks become more effective and more damaging. “Compliance can be a good starting point for securing information infrastructure and data if an organization has not put anything in place previously, but it cannot be the end point of the conversation.”
However, I'm not even sure what he means when he goes further to state that "(w)e need to stop thinking about information security and start thinking about information risk management.” Then there's
“The technology is just a vessel for the data and has little value by itself. By focusing on the data, enterprises will be better prepared for the challenges that they may face from any adversary”
We should always be sure to consider that the 'map' is not the same as the 'territory.'
I enjoy reading Joe Weinman's posts. And today's post at GigaOM is no exception. He does a great job organizing the problem of data when considering the architecture of cloudbursting. Joe's post has prompted me to break my 'radio silence' of a few months.
In 4 1/2 Ways to Deal With Data During Cloudbursts, Joe calls out a number of architectures and discusses some of the considerations that impact the choice of a relevant scenario. I won't try to recreate the architectural strategies, but I was taken by how relevant these same constellations are when addressing some of the more advanced considerations of data clouds, peripatetic workloads and data governance.
1) Independent Clusters: This one is pretty straight forward and Joe's characterization of "minimal communication and data-sharing requirements between the application instances running in the enterprise and cloud data centers" makes sense. The data-specific considerations in using the cloud service resources tend mostly to center about providing the user with a uniform (or at least acceptable) standard of data security.
2) Remote Access to Consolidated Data: This strategy is called out for those situations in which application instances running in the cloud require access to a single-instance data store, or data store(s) which must for various reasons remain within the confines of the enterprise data center.
Notice my 'or' in the last sentence. Besides architectural requirements that require a single-instance data store, the reality of enterprise IT is that data stewardship requirements often require the authoritative datum to remain within the enterprise data center.
3) On-Demand Data Placement: Weinman points out that
...if I/O intensity and/or network latency are too high for remote access, then any needed data that isn’t already in the cloud must be placed there at the beginning of the cloudburst, and any changes must be consolidated in the enterprise store at the end of the cloudburst. The question is: “How much data needs to get where, and how quickly?”
This is clearly the right question to ask first. If a large data set is required to be in close proximity to the cloud service application instances, this may require enterprise IT to rely on a number of tactics to reduce delay in commencing cloud-based operation: large bandwidth networking services, possibly made available on-demand; advanced WAN optimization technologies (e.g. data deduplication).
As in my consideration of the remote access to consolidated data, on-demand data placement may imply a requirement for additional measures to deal with compliance and data stewardship, therefore calling on the purveyors of fast file transfer or on-demand, adjustable data transport services to offer a form of 'in-flight' data mediation services. Alternatively, the enterprise data center may be called on to implement dataset virtualization approaches or data masking systems in order to remain in compliance.
4) Pre-positioned Data Placement: He makes the point that pre-positioning "... adds additional costs as a full secondary storage environment and a metro or wide-area network must be deployed.'
4.5) BC/DR Plus Cloudbursting: This was the point at which I chortled with recognition.
Thanks, Joe! I've been looking for the context in which to make this point for years. This has been a soapbox of mine for a long time ... almost since the notion of utility computing (now 'cloud computing') started circulating as a meme.
In addition to using cloudbursting as the premise on which to incorporate business continuity and disaster recovery costs into calculation, I'd like to throw in at least one more, in hopes of getting this to 4 3/4 ways to deal with data + cloudbursts. Please bear with me... this is work in progress.
Data Governance, Data Stewardship and Data Residency:
Many of the issues relating to data in conjunction with cloudbursting are not new. When you stop to think about it, the 4 1/2 ways that Weinman outlines are variants of a generic data sharing problem across organizational boundaries. If we add any form of data sharing to the real cost of the enterprise data center, the issue we must address is that of Data Stewardship. It's been defined in various places, but here's one of my favorites since it places it in context with Data Governance.
Data Governance: The execution and enforcement of authority over the management of data assets and the performance of data functions.
Data Stewardship: The formalization of accountability for the management of data resources.
Data governance in the enterprise data center may require a 'complete' record to always be under the stewardship of the enterprise, and never at risk of being located in a different legal jurisdiction (e.g., the details of a financial transaction must remain in the immediate and direct control of the responsible financial institution). Examples abound, but one can point to financial and personal information which must, for compliance reasons, never leave the geographical borders of a country with stringent data protection regulation n (e.g., not in that cloud-resident datastore in India or Switzerland).
In these cases, the implications of cloud bursting on data may require the addition of data masking/data obfuscation, or applications which are demonstrably proven to operate on meta-data of other kinds without jeopardizing data stewardship compliance. This particular aspect of Data Stewardship is sometimes called the Data Residency Dilemma.
Getting to 4.75 - Data Governance Plus Cloudbursting: Even if, in addition to taking responsibility for data mirroring or replication to provide Business Continuity / Disaster Recovery, the enterprise is constrained from using Data + Cloudbursting because of the costs and constraints of data governance, the question arises: Are there services / technologies that can be provided by the *aaS supplier which can be brought to bear? To me, this appears to be a question of data center pragmatics rather than strictly an issue of recalculating the breakeven point.
There are many technologies for data sharing, some of which come into play for Data + Cloudbursting. When the solution requires extending the 'boundaries' of the enterprise in both the application and data domains (as we do with cloudbursting), the first question has usually been constructed as: Should the shared data reside inside or outside the firewall?
Elastic perimeter technologies: For cloudbursting with data 'leaving the building', elastic virtual private networks (such as CohesiveFT's VPN-Cubed , particularly their Data Center to EC2 version) address the underlying, network-oriented issues of wandering data.
Data masking & obfuscation: Conventional data encryption of "data at rest" does not satisfy the safety requirements of most enterprises when data is placed outside the corporate data center. Because the data must be decrypted when “in use” by a cloud-resident application image, conventional disk or file encryption does not protect against a compromise of or misuse by any systems processing the data. Suitably transformed portions (i.e. fields) can be used that provide integrity of the source data required for the application by means of data masking or obfuscation.
Meta-data & data virtualization: We're now starting to see, usually in conjunction with specific SaaS offers, data 'proxy' servers and other means that allow the enterprise to retain specific data elements 'locally resident' within the data center rather than residing 'in the clear' within a data cloud. What we can expect to see within the next year are solutions that provide this type of offer associated with Master Data Management technologies, or enhanced data-in-motion services provided by cloud service providers at all levels -- Iaas, PaaS and Saas. The most immediate utility of these offers will be for enterprises wishing to make real use of cloudbursting.
---
Joe Weinman broadened the definition of the real costs of an enterprise data center and has shown clearly how cloudbursting + pre-positioned data can contribute to addressing the BC/DR costs. Like BC/DR, the enterprise data center has to consider data governance in the context of interorganizational data sharing. Cloudbursting is just one form of data sharing, and presents the innovative cloud service an opportunity to provide generic solutions to data sharing governance for the enterprise.
Truth in advertising: In two of my recent entrepreneurial adventures (Univa and Safe Data Sharing) as well as two for whom I've acted as an advisor (Perspecsys and Replicus ), the problems of data stewardship and anticipatory data transport (e.g. moving/replicating the dataset well in advance) all come into play.