StorageRap

EMC, the Misanthropic Media Company?

2010-07-29T03:40:57-07:00

It's getting very hard to keep up with all the crazy social media stunts coming out of Hopkington, but they seem to have done it to themselves again. First was the questionable spamming for viewers so they could claim they had a viral video, then today they just "leaked" a 3PAR sales "kill sheet" - and also apparently established a "secret" site with the URL Notapp.com, where they compared their own guarantee program to Netapp's. According to Simon Sharwood at Search Storage Australia, the site was removed and accessing the URL directed browsers to EMC's site.

Perhaps it is all part of a new marketing strategy by newcomer Jeremy Burton, who joined EMC as Chief Marketing Officer back in March. As best I can tell, Burton's new marketing strategy for the company is that people will believe anything. Maybe he doesn't think there are enough new products coming out of EMC - or that the delays in getting their ballyhooed FAST out the door are too embarrassing - but instead of trying to promote EMC on its own merits, it looks like he is doing his utmost to mud wrestle. Is that what EMC is paying him the big bucks for?

EMC suddenly is taking a bigger interest in 3PAR. That's good. Search Storage Australia just published parts of a competitive document that EMC was circulating to it's partners about 3PAR. It certainly wasn't a surprise because we'd seen it previously, but I was sorry to see it published because it made EMC look ridiculous, which was working pretty well for us. But now that it's been outed, here is what we have to say about it (in the guise of Ineption's lead character, the CRO)

The messaging is not built in, but our zero detection technology for optimizing capacity is. The host SW commands to do this are short and do not require "careful coordination". Veritas, Oracle, Windows Server and Linux software all work with minimal operator effort. For instance, this document from Oracle, describes the whole process, with the sole operator command being this: #bash ASRU LDATA.

Can EMC provide online reclamation of zeroed space without risking capacity overruns and with tolerable performance? 3PAR can. Does EMC have these capabilities in both mid-range and enterprise storage arrays? 3PAR does.

3PAR has both Flash and 1 TB SATA drives. We also have Adaptive Optimization software that uses Flash SSDs for storage tiering. EMC still doesn't have it after they made such a big deal about it last year. They like to tell customers that their size gives them development advantages, but their track record doesn't support their claim.

3PAR arrays allow users to create many tiers, but without the need for disk pools. Tiers are constructed from the combination of drive type plus RAID level. For instance, you can have separate tiers for SATA, FC and Flash SSD drives with the RAID level you select. Our Dynamic Optimization software allows admins to move data from one tier to another. You can "dial in" the performance and protection you want.

All systems have a peak output , ours just happens to have a lot more throughput than theirs - and at higher disk utilizations. We have published benchmarks that show how our systems perform. They don't. Adding disk drives to a system and utilizing those drives is far easier with a 3PAR system than either VMAX or Clariion where you have to wrestle with putting drives in the pools you want to use them for.

There are no disk pools in 3PAR storage. Pools trap resources so you can't use them. Work isolation in pools leads to hot spots and storage admin nightmares. Wide striping does not mean you can't have tiers. That is an idiotic statement.

VMAX can configure large pools - and all the drives in them have to be at the same RAID level meaning you can't create multiple tiers within those pools. If you want multiple tiers, you need multiple pools and all the headaches that involves. Change management in an environment with multiple pools is complicated. You also need to consider the pools needed for snapshots and remote replication. Are those easy to provision and change on EMC storage. Most would say "no".

3PAR uses all disk spindles all the time for delivering IOPS and pro-active sparing is done using reserved space on those drives. Rebuilds do process quickly. Would EMC have you believe they never have to perform drive rebuilds? Really?

The RAID6 thing really makes me sad. They look so stupid when they say it. We're all sorry to say goodbye to that piece of FUD.

Our front end archiecture was designed for large-scale parallel connectivity to match the massive bandwidth capabilities of our wide striped back end. Our benchmarks and the cost per IOPs in those benchmarks speak for themselves. Our customers also tend to run 3PAR systems at much higher disk utilizations than they run other vendor's arrays.

We support a huge number of ports on our systems w/full active/active data access across all controllers. All controller nodes can be used to access all data volumes. We have a number of customers that run fairly sizable SANS without switches because they have enough ports on their arrays so they don't need to consolidate access through switches.

5- 9s? We're there. Our systems get pounded on every day in some of the largest private and public data centers in the world. They are designed with complete redundancy in all components and have advanced capabilities such as Persistent Cache to maintain high levels of performance even after the loss of a controller.

The delays in bringing their FAST tiering software - a product they were hyping in April of 2009 - to market have shown that size doesn't matter much when it comes to delivering technology on time. I'm not saying 3PAR always delivers on time, but EMC is far from immune to these problems. In fact, the need for them to coordinate across multiple product lines creates certain disadvantages for them.

As to their comments on our support; they are pure FUD and grasping for straws. We would not be able to maintain the customers we have if it were not for our efforts at supporting them.

* * * * * *

The following content was added on July 30th by Rusty Walther, 3PAR's Vice President of Customer Services & Support.

Stating that 3PAR “outsources support” is just plain silly, especially coming from a company that keeps most of the worlds’ largest offshore outsourcing companies in business. Like EMC, 3PAR uses Third Party Maintenance suppliers (TPM’s) for break-fix field activities. In some geographies, EMC and 3PAR even use the “same” TPM. But EMC also outsources most of their volume call center and Level-1 Technical Support to offshore suppliers. Not so at 3PAR. Everyone that touches a 3PAR support case is a 3PAR-badged employee. I challenge EMC to identify a single outsourcing company that handles 3PAR technical support. EMC’s outsourced technical support sub-contractors could be listed alphabetically, by geography, or by technology category … but you’d need a couple of sheets of paper to do it.

How do you measure virality in a video?

2010-07-27T14:07:17-07:00

The whole idea of a storage company making a video that reaches "viral status" is very interesting to me since I like making videos and I work for a storage company. There are several problems with making a viral storage video including the relative size of the storage community, which is not large enough to generate large numbers of viewers and then there is the matter that competitors tend to not promote each others work very much, which creates a dampening effect on the whole word of mouth thing that I have associated with viral media.

So, I get the idea of trying to promote a video, because you need a big lift beyond the industry to do it and you need to take advantage of the dynamics of the online video world.

If you follow this link and read it, you'll get an idea of how corporation can make viral videos. But not all viral videos are corporate - there are some that make it on creativity, execution and timing. Although its against very steep odds, a non-corporate video video can succeed if its interesting and compelling enough for people to want to share it with others. Being outrageous helps a lot. The question remains - if most viral videos result from corporate marketing programs, what does that say about the definition of viral - is it only the most numbers of viewers, or is it something else?

As far as I can tell, the most "viral" video produced in the storage industry was probably the Hitler social media video, which quietly made the rounds last summer and now has over 18,000 views. To me, this seems like the right amount of views to qualify for virality in the storage industry. I'm fairly certain that no company was promoting it with a marketing campaign - although there were many industry employees that passed the link through email. As far as I know, the producer of the video is a mystery (at least to me) and I don't think it was done to achieve any business goal.

I've made a few videos that had nothing to do with storage that were simply attempts at being funny. They were fairly well received, but were not promoted by marketing and have not generated nearly the same number of views as the "corporate viral" videos or the Hitler video have. Admittedly, I get a little jealous when I see somebody else's corporate production reaching getting a lot of hits, but I know what to expect without a campaign.

But it's also giving me the opportunity to re-post what I think was my best and funniest video - a video cartoon that I made the day that Oracle announced that it was going to acquire Sun. One question, - neglecting the numbers and the promotion necessary to generate "viral status", does this video have viral qualities? Why or why not?

From viral spam to virulent sham

2010-07-26T13:18:22-07:00

The twitterverse is busy again today with discussions surrounding EMC's us of spambots to generate views of videos they are trying to make viral. If you are interested in seeing what is being said, check out these people's tweets and you'll be off on a trip down a dark hole.

@johnful, @dvellante, @sfoskett, @valb00, @furrier

Here are a couple cartoons I made about it last week from my new cartoon, Ineption:

Netapp's Val Bercovici suggest this viral spamming as the end of innocence in social media, but innocence exited the social media stage long ago.

I'm much more concerned about how large companies like EMC can use social media to suggest product and customer relationships that stretch the truth well beyond the impressions that a reader might take away from reading suggestive blog posts from respected corporate voices. As "unofficial company statements" that are more influential than press releases, social media pieces can distort things in a way that more-accountable corporate marketing are not allowed to.

Last week, Chad Sakac and Chuck Hollis published blog posts that pointed to an EMC white paper about details of a VMAX implementation at Terremark, an excellent 3PAR customer. Readers of these posts would probably think that VMAX was being used as the storage behind Terremark's multi-tenant, Enterprise Cloud service offering. That would be stretching things more than just a little bit. I commented on both blogs and the responses to my comments were interesting. I guess I feel a little kinder towards Chad as a result.

It is possible that somewhere in the world, a VMAX is being used by Terremark. One would expect Terremark to be looking at various storage platforms as a matter of course, it only makes sense for them. After all, VMware made a significant investment in Terremark last year and we all know who owns VMware. There are certain favors that EMC can ask that vendors such as 3PAR can't. But Terremark also has to operate Enterprise Cloud in their US major data centers every day and the storage they use for that is not in a test lab - it's production - and it is 3PAR storage.

And its not for lack of trying on EMC's part. Last November when VCE was announced, Terremark was discussed as a featured customer in both Chad's and Chuck's blogs. That was OK, I understand the excitement that surrounds a big announcement. But nine months later, to suggest that this announcement had given birth to a major production environment for a service that it is not supporting sort of stuck in my craw.

Here is a video I made at VMworld with Jason Lochhead, CTO of Hosting Solutions at Terremark last year where he talks about vCloud Express and Enterprise Cloud. Very cool offerings and definitely on the leading edge of VMware-based service offerings.

It's not a viral video, but it has a lot more to say about what people care about than the videos EMC has been chasing with spambots.

The Netapp vSphere 4.1 decoder ring

2010-07-21T19:31:34-07:00

There was a lot written last week surrounding VMware's release of vSphere 4.1. Netapp appeared to have a lot to say, but it was confusing to figure out what they were really talking about. I think I've got it now.

It's unusual for a company to be invited as a centerpiece of high-visibility festivities and then mysteriously decide not to follow through. It would be like getting complimentary tickets and backstage passes from Lady Ga Ga herself, telling all your friends about it and then not going. It it does make one wonder. Why wouldn't you do whatever it takes to be included in VMware's big summer announcement party? Well, if you're Netapp, the answer appears to be - "Being there is over-rated. Just make sure everyone thinks you were." Call it Photoshop for PR or call it keeping your poker face, it's a mash up of a blown opportunity and opportunistic courage.

The excitement for VMware's storage partners was concentrated in two areas: VAAI (vStorage API for array integration) and SIOC (Storage I/O Control). The initial release of VAAI includes new SCSI block storage commands that allow the arrays to offload host systems from redundant, resource-consuming tasks. SIOC is a method for managing I/O queues to create more fairness in accessing storage resources. Netapp issued a press release last week in conjunction with the vSphere 4.1 release, but it was for their Virtual Storage Console, not for the support of the storage enhancements in vSPhere 4.1. There was a flag waving mention of VAAI:

"Additionally, NetApp is supporting the new VMware vStorage APIs for Array Integration (VAAI) capabilities that offload data management tasks from the host server to the storage system. This can free up host CPU cycles for better performance and increased virtual machine density."

That's not exactly saying anything, but its more than they had to say about SIOC, which was zilch.

The bottom of the release directs readers to Vaughn Stewart's blog for more info. Apparently, Netapp's PR department left the rest of the innuendo up to Vaughn - a diligent and loyal Netapp employee who understands that sometimes a vendor blogger doubles as a PR bagman. It looks like I need to add a new chapter to Vendor Blogging with Dummies.

You have to dig into the comments to get some of the details, but Vaughn's blog does a decent job explaining that Netapp is working on delivering VAAI functionality in Q4 2010. Now, that's not all that late considering its only 6 months or so away, but as a privileged insider to VAAI development, it's not a great showing either. In fact, it wouldn't surprise me if some of the companies who were not in the program, such as Compellent, HP, IBM and Xiotech come out with VAAI plug-ins before Netapp does. As for 3PAR, we will have our VAAI plug-in available in September as part of a maintenance release. We didn't have a lot of time to develop VAAI functionality after gaining access to the APIs in early 2010, but we fast-tracked the development of it in order to make the announcement.

As much as I admire Vaughn's hutzpah for stepping in to carry the load that others at Netapp should have, there were a few problems with what he said. First was the absurd statement that " SAN is attempting to be more NAS-like". There is so much wrong with that statement that it's difficult to find a place to start. Who or what is SAN? Is VMWare SAN? Is the T10 SCSI standards committee SAN? Is SAN the being an embodiment of SAN the block protocol? Is there a virtual reality thing going on here? And what is NAS-like anyway? Does it have anything to do with the size of one's beak or the way particular vowels resonate in the sinus cavities? Or is it like racing the back roads in a used chevy? Whatever Vaughn meant, I tend to dislike the imprecision of technology anthropomorphism.

The second thing Vaughn said was "As for the first release of VAAI... These features ALREADY EXIST in NFS." Really, block zeroing? That is a function developed for EagerZeroThick volumes, which are only supported on VMFS datastores, not NFS datastores. Perhaps we will see that change in the future, but for now its SAN only.

Hardware assisted locking is a way to allow smaller granular locking for VMFS and addresses an issue with VMDK-level operations in a shared datastore. Because NFS puts VMDKs in separate datastores, which are locked independently, hardware assisted locking is unnecessary for NFS. In other words, its a SAN only function because the current NFS datastore architecture doesn't need it.

The other API in VAAI is Full Copy. This VAAI API appears to be functionally equivalent to a Netapp utility called RCU (Rapid Cloning Utility) that was included as a function in their Virtual Storage Console. It is not, however, something that exists in NFS, unless Netapp wants to give that feature to all it's NAS competitors. As a vSphere function, Full Copy will be available to all vendors that implement the VAAI APIs. It will be interesting to see what differences there are as far as programmatic control using the VAAI plug-ins, vendor-specific consoles and Powershell.

Robert Cockerill from Thames River Capital: A man with broad responsibilities and 3PAR storage

2010-07-20T13:46:23-07:00

3PAR customers like that fact that 3PAR arrays are so easy and fast to manage. In this video, Robert Cockerill from Thames River Capital in London talks about all the various things he does, his Windows-based infrastructure, how 3PAR's thin provisioning helps him manage it all and how simple it was to protect it with 3PAR Remote Copy.

Ineption No 1: The CRO gives The Storage Arrogance his dream job

2010-07-19T03:34:28-07:00

Today is the grand opening of a new cartoon strip, Ineption

(Click the cartoon to see a larger image)

Clarifying VAAI capabilities and implementations

2010-07-14T14:15:55-07:00

Virtual Geek and I had a discussion on his post yesterday about vSphere's VAAI capabilities announced yesterday.

I wrote about the fact that we already had zero detect technology in our product, which is useful for the new Full Copy command because it allows customers to remove zeroed data from clones when they are created and return them to array free space.

The discussion became a bit confused when Chad interpreted what I was saying as pertaining to Block Zeroing.

Block Zeroing and Full Copy are different aspect of the VAAI API. The intent of block zeroing is to reduce the amount of CPU effort and storage traffic required to write zeroes across an entire EagerZeroThick (EZT) VMDK when it is created. The intent of Full Copy is to make clones of VMs quickly without consuming I/O bandwidth. Things get interesting when you start thinking about making a full copy of an EZT VMDK that was created using VAAI with block zeroing - but I'll discuss that later.

I also want to clarify what zero detection technology is. 3PAR T and F class arrays have zero detection technology, which is enabled by Thin Persistence software, that recognizes zeroed blocks as they are read by the array and returns them to the array's free pool. Any read requests made to these block addresses will return a zero value. In essence it is dedupe for zeroes.

However, Zero detection is not needed when an EZT VMDK is created using the VAAI plug-in because the array will recognize the intent of the command and not write the zeroes. In other words, the VMDK will only contain a very small amount of reserved space when it is created. Again, any attempts to read blocks in those ranges will return zero values. Zero detection is effectively bypassed during the creation of the EZT VMDK.

The exception to this behavior is when the EZT VMDK being created is written to a thick volume - in that case the array will write zeroes across the entire VMDK.

The remaining cases for the creation of EZT VMDKs on 3PAR arrays occur when the VAAI is not used. For a thick volume, the entire VMDK has zeroes written to it. Thin volumes not using zero detect also have zeroes written over the entire VMDK. Thin volumes with zero detect will not have zeroes written to them and will contain only a small amount of reserved space.

FWIW, the reserved space is used as instantly-available capacity that can be allocated on-demand when writes start coming into the volume. 3PAR arrays always "read ahead" free space to improve the performance of thin provisioning.

The next bit here could be a bit thorny, so clear your head. The matter of making a Full Copy of an EZT VMDK to a thinly provisioned volume was something Chad said was not allowed. My assumption here is that the type of thin provisioning used makes a big difference.

For instance, if you are using TP from VMware, I could see where they would not allow a full copy to be made. The problem is that the full copy will return all the zero values for the source VMDK, whether or not those zeroes were ever actually written - and write them to the target TP volume. In other words, the target could be much larger than the source. In the VMware TP scheme, this could make for problems in a hurry if you were making a bunch of clones this way.

In contrast, if you were using a 3PAR array with zero detection, the Full Copy of the source VMDK would return zeroes for the entire VMDK, but the zero detection would strip them out again as the target was being written. You could make as many clones as you wanted this way, knowing that the physical capacity they consume would be a multiple of the physical capacity consumed by the source VMDK. In other words, you wouldn't have to worry about virtual zero bloat making a mess of your VMFS volume.

One of the big differences between 3PAR's zero detection technology and other vendors zero-reclaim technology is that 3PAR's process is real-time-on-ingestion as data comes into the array, whereas zero-reclaim works in a post processing fashion after the zeroes have already consumed disk space. This could be a significant difference in many cases because the post-processing method has the potential to create unexpected capacity-full conditions before the zero-reclamation process even has a chance to start.

vSphere 4.1 Array Integration on steroids

2010-07-13T10:57:24-07:00

We've been anxiously waiting for VMware's announcement of vSphere 4.1 for weeks. There are many big things in this release, including significantly scaling the management capabilities of vCenter and increasing the number of simultaneous vMotions that are supported. The door is open for ESX deployments to achieve much greater densities than they could previously and that's a big deal to large enterprises who want to get more resources under the control of fewer points of management. There are still great gains to be made in consolidation - more on that later.

In the storage world, there are a couple big things, SIOC and array integration through the VAAI API. Technodrone has put together an excellent post on SIOC and I highly recommend that anyone wondering how to make this functionality works should go to this post and read it. Array integration has been advanced in three ways:

Hardware assisted locking
Full copy
Block zeroing

Array integration through the VAAI API is already at a very advanced status at 3PAR with some of the most important functions implemented through our I/O co-processor ASIC. While some companies want to write off the importance of hardware, 3PAR believes there are many things that need to be done in hardware to get the performance needed to truly scale storage for virtual environments. Our co-processors are key to getting much greater storage utilization and higher VM ratios and are one of the 3PAR innovations that separate our best of breed products from everybody else. The capabilities discussed below are available in the hardware today, and will be enabled with a software upgrade in September.

OK, lets talk about hardware assisted locking first. For customers that have experienced locking problems, this is a big deal. The problem has been well-documented online - but in a nutshell, customers have run into problems where an operation that locked the LUN for a VMFS did not complete, thereby freezing all I/Os for all systems using that LUN. That was certainly a nasty problem - not a bug necessarily, but certainly an incredible pain in the rear to all involved.

VMware's response in vSphere 4.1 was to include a command in the VAAI API using an atomic test and set instruction for implementing fine grained locks for small block sizes. There will still be locking in VMware, but on a much smaller scale.

Unique to 3PAR is the fact that this new locking mechanism is implemented in our I/O co-processors where it completes very quickly, as opposed to implementing it in code in the controller. If you consider an environment with high VM ratios and multiple vMotions going on you want this granular locking mechanism to as quickly as possible. Nobody else comes close to the speed that 3PAR processes them.

Next is the new Full Copy capability - also with co-processor assistance to reduce the capacity of the copy that is made. 3PAR has zero detect and reclaim technology integrated into the co-processor. With zero detection running in an array, as new writes are made, strings of zeros are detected by the co-processor and those blocks are returned to free space inside the array. If future reads are made to those blocks, zero values are returned, but not from disk. The result is that copies of VMDKs with lots of zeros in it, will be much smaller after the copy is made - and the copy will proceed much faster.

This sort of functionality works amazingly well with EagerZeroThick (EZT) volumes in vSPhere. VMware requires EZT for Fault Tolerance (FT) and MSCS clusters and also recommends EZT for high performance. The main complaints about EZT are that it takes extra time to write all those zeroes when the VMDK is created and that it doesn't work well with thin provisioning. With 3PAR's zero detection, the time it takes to writes all those zeros and the space they consume is a non-issue, but more on that later. Virtual Geek at EMC wrote about VAAI today and in his discussion of what does not work for full copy he mentioned copying from an EZT volume to a thin provisioned one. Actually, he's wrong about that where 3PAR is concerned, because EZT to Thin works very well on a 3PAR array with zero detect.

The image below illustrates the advantages of using EZT on a 3PAR array with zero detection:

The last API element to discuss is Block Zeroing. The idea is that the host communicates to the array to write a string of zeros when it is provisioning storage or overwriting blocks to a non-EZT VMDK. vSphere writes a lot of zeroes in order to maintain data integrity with multi-tenancy. The hypervisor zeroes zero out blocks prior to writing them in order to ensure that a virtual data imprint from an old VM does not occur for a new VM.

But writing all those zeroes consumes CPU and I/O bandwidth that could actually be used productively, so VMware included a new command to offload the host from writing zeroes, effectively shunting that workload to the array. Voila - problem solved with 3PAR!! The zero detection and reclamation technology in a 3PAR array not only offloads the host from writing zeroes, but it also gives customers instantaneous reclamation of capacity with a smaller digital footprint (less capacity consumed) and faster performance. That's pretty cool and it's a trifecta that only 3PAR has.

What is amazingly cool about today's vSphere announcement for 3PAR customers is that all three API elements, hardware assisted locking, full copy and block zeroing are already implemented in 3PAR's T and F series hardware platforms, and will be usable by the end of September with a firmware upgrade.

Our co-processor architecture really delivered for us this time. But it's been delivering the goods for our customers for a long time already. In virtualized environments our customers tell us they double their VM density, while cutting their storage capacity in half - all while reducing the amount of storage administration necessary by 90%. Those stats can be hard to believe, but when you look at what we delivered on the first day vSphere 4.1 was announced - when most people didn't even know we were working on it - it might make it easier for people to understand why.

We take virtualized environments very seriously. People that don't know about 3PAR don't consider us to be a leader in virtualization, but when they find out the depth of technology we have and how well it works across our entire product line they understand we are leading in ways that really pay off for them. And the bigger they are, the bigger the rewards can be - especially after today.

Microsoft's mediocre cloud appliance

2010-07-12T11:44:07-07:00

A short while ago, Microsoft announced their plans to create a stack platform appliance for Azure. This was done, presumably, to deal with the competitive threat from VMware, which is part of the V-Block-centered Acadia business venture. Sorry Microsoft, but IMHO, this is a clear signal that the cloud stack has been catapulted to the stratosphere of hype.

There are a couple assumptions about clouds in a can that show the ready-fire-aim nature of these types of solutions.

The first is that clouds in a can will provide an on-ramp for private cloud infrastructures to to use public cloud infrastructures in the future. Oh yes, and how exactly will that work? Vaporware drawn on white boards with clouds in the middle? The allure of an easier future through stacks rivals any of the malarkey that our industry has ever produced.

The second assumption is that customers will save money with stacks and appliances. There are too many vendors involved, all of them hoping to increase revenues. Most of these integrated solutions will have surprisingly high price tags. Of course, there will be loss-leader infrastructure in a can products too. THAT should be interesting to watch.

The idea of a turn key infrastructure is appealing as a concept, but the real cost of operating an infrastructure is where the advantages lie. Some customers will be happy with expensive mediocrity that traps them.

In contrast, Best of breed products today are those that are powerful, flexible and easy to use. These are the products to build a private cloud with. Take your infrastructure where it needs to go instead of driving somebody else's vehicle down their highway.

David Scott on cloud dynamics in the IT industry

2010-07-06T11:07:07-07:00

Forbes.com recently published a video interview with our CEO, David Scott on their Intelligent Technology channel. David who talks about the shift from privately run data centers to utility, public cloud computing services. Here are some of the key points from it.

The move to industry consolidation represents the inability of large systems vendors to grow organically and they have had enter multiple product segments with a patchwork or products and services.
The evolution of enterprise IT as a utility computing service represents a threat to incumbent vendors' revenue streams because they are not always the leading vendors selling to public cloud service providers. The development of full stack products is an attempt to maintain their incumbency.
In the shift towards public cloud for enterprise IT services, service providers are looking for best of breed solutions, as opposed to integrated stacks.