However, as time has gone on, and technology has allowed for persistent storage on the cloud, organizations have started to rethink this paradigm. In fact, several companies are using Hadoop and big data today to gain competitive advantage. And while they are running it on virtualization, they are not moving the data. There are other advantages.

VMware’s Big Data product line marketing manager Joe Russell, spoke with Roberto Zicari this week in an interview on ODBMS.org that helps articulate why Hadoop not only can run on virtual infrastructure using Project Serengeti, but why companies should consider it to save time and make Hadoop more usable.

Understanding Data Locality with Hadoop Running on Serengeti

With Hadoop and virtualization, it’s important to see virtualization through the lens of data locality.

In a distributed system, data locality is about keeping the processing and data together to achieve greater performance and avoid network bottlenecks. Moving large volumes of data with vMotions is unreasonable. With petabytes of data, it would take too long and affect data availabillty. Similarly, separating the processing introduces latency that is unacceptable.

As Russell explains in the article, Serengeti is providing value by preserving data locality, but allowing organizations to deploy a enterprise-tested High Availability and Fault Tolerant Hadoop clusters in minutes—something even the most seasoned Hadoop veteran can not do. It also paves the way for advanced use cases such as mixed payload deployments and multi-tenancy.

Russell explains further:

A common misconception when virtualizing Hadoop clusters is that we decouple the data nodes from the physical infrastructure. This is not necessarily true. When users virtualize a Hadoop cluster using Project Serengeti, they separate data from compute while preserving data locality. By preserving data locality, we ensure that performance isn’t negatively impacted, or essentially making the infrastructure appear as static. Additionally, it creates true multi-tenancy within more layers of the Hadoop stack, not just the name node. 

I think there is some confusion when we say “in the cloud”. Here, Steve is talking about running it on a public cloud like Amazon. Steve is largely introducing the concept of data locality, or the notion that large amounts of data are hard to move. In this scenario, it makes sense to bring compute resources to the data to ensure performance isn’t negatively impacted by networking limitations. VMware advocates that Hadoop should be virtualized, as it introduces a level of flexibility and management that allows companies to easily deploy, manage, and scale internal Hadoop clusters.

Zicari probes further and challenges while you can keep the data together, aren’t there basic functions that depend on the data and processing happening together:

Zicari: There are concerns on the approach of decoupling Apache Hadoop nodes from the underlying physical infrastructure. Quoting Steve Loughran (HP Research): “Hadoop contains lots of assumptions about running in a static infrastructure; it’s scheduling and recovery algorithms assume this.” What is your take on this?

Joe Russell: A common misconception when virtualizing Hadoop clusters is that we decouple the data nodes from the physical infrastructure. This is not necessarily true. When users virtualize a Hadoop cluster using Project Serengeti, they separate data from compute while preserving data locality. By preserving data locality, we ensure that performance isn’t negatively impacted, or essentially making the infrastructure appear as static. Additionally, it creates true multi-tenancy within more layers of the Hadoop stack, not just the name node.

I think there is some confusion when we say “in the cloud”. Here, Steve is talking about running it on a public cloud like Amazon. Steve is largely introducing the concept of data locality, or the notion that large amounts of data are hard to move. In this scenario, it makes sense to bring compute resources to the data to ensure performance isn’t negatively impacted by networking limitations. VMware advocates that Hadoop should be virtualized, as it introduces a level of flexibility and management that allows companies to easily deploy, manage, and scale internal Hadoop clusters.

How Does It Work?

VMware created Hadoop Virtual Extensions (“HVE”) to make Hadoop distributions virtualization aware. It works by inserting a node group layer between the rack and host to make Hadoop distributions topology aware for virtualized platforms. So, while technically Serengeti has separated the compute resources from the data to allow for better management, scaling and faster deployments, the hypervisor knows to keep the processing and data on the same physical machine.

Russell also outlines how High Availability is added through using vSphere:

We ensure High Availability (HA) by leveraging vSphere’s tested solution via Project Serengeti’s integration with vCenter (management console of vSphere).

In the event of physical server failure, affected virtual machines are automatically restarted on other production servers with spare capacity. In the case of operating system failure, vSphere HA restarts the affected virtual machine on the same physical server.

In Hadoop nomenclature, this means that there is HA on more than just the name node. vSphere’s solution also allows for HA on the jobtracker node, metastores, and on the management server, which are critical pieces of any Hadoop system that require high availability.

More importantly, as Hadoop is a batch-oriented process, it is important that when a physical host does fail, that you are able to pause and then restart that job from the point in time in which it went down. VMware’s vSphere solution allows for this and has been tested amongst the biggest Enterprises for the better part of the past decade.

HVE has been donated back to Apache Hadoop. Similarly, Serengeti is also open source. While it doesn’t make sense for VMware to spend money and engineering to have Serengeti ported to work with other hypervisors, Russell does state that this is very much the desire.

Announced this morning on the new Pivotal blog, where RabbitMQ now resides, this version includes enhancements to garbage collection, consumption, requeuing, memory use, and dead lettering.

For those on Mac OS X, there is a newly packaged, standalone release of RabbitMQ that doesn’t require a separate Erlang install.

Some key, new capabilities include eager synchronisation of mirror queue slaves, automatic cluster partition healing, and improved statistics (including charts) in the management plugin. There are also many enhancements and bug fixes to the server, Java client, Erlang client, and a number of other plugins, including federation, old-federation, shovel, Web-STOMP, STOMP, and MQTT plugins, as well as the consistent hash exchange.

RabbitMQ’s blog post on the topic shares screenshots of several new features like the ones for new charts and filters below:

Read More:

• Download RabbitMQ 3.1.0
• Check out the release notes for RabbitMQ 3.1.0
• Read more from the Rabbit Team about the RabbitMQ 3.1.0 release on the Pivotal blog
• See more images from the new release

There are ways to preserve the mainframe and improve application performance, reliability and even usability.  As one of the world’s largest banks sees, big, fast data grids can provide an incremental approach to mainframe modernization and reduce risk, lower operational costs, increase data processing performance, and provide innovative analytics capabilities for the business—all based on the same types of cloud computing technologies that power internet powerhouses and financial trading markets.

The Fast Data component

One customer used used vFabric GemFire to save $75 million dollars on a mainframe modernization project, and it has been used for the past ten years as highly performant, horizontally scalable data transaction layers, or big data grid, for mission-critical applications. Both GemFire and it’s sister product, SQLFire, are known to achieve linear scale. Key use cases include credit-card transaction systems, stock trading platforms, foreign exchange systems, web-based travel reservation systems, and mainframe batch data offloading. As an in-memory data grid, its main advantages are being able to do in-memory, sub-millisecond transactions while still maintaining the highest standards for fault-tolerance, high-availability, and linear scalability on a distributed platform.

The Big Data component

With big data analysis, Greenplum has a multitude of customer case studies with companies like O’Reilly Media, Skype, and NYSE Euronext. These solutions have become well-known when it comes to analytical analysis on multiple terabyte or petabyte data sets where traditional relational databases begin to break down, stop scaling, or fail to deal well with un-structured data. Greenplum technology provides a complete big data solution for both structured and unstructured data, based on the Greenplum Database and Pivotal HD—a commercially supported distribution of Hadoop that includes HDFS, MapReduce, Hive, Pig, HBase, Zookeeper, Sqoop, and Flume. The recently announced Pivotal Advanced Database Services powered by HAWQ allow for SQL queries to run on the fastest Hadoop-based query interface on the market today—a 100X+ faster solution.

Fast Data + Big Data: Better together

Big data and fast data solutions make a lot of sense together as we’ve seen on many customer solution blueprints delivered over the past several months. This is because most business owners and administrators aren’t able to fully utilize the data being captured in their transactional systems on a daily basis. From a business value perspective, the fast data layer can bring scalability and reliability to the business while reducing the cost per transaction. Most transactional systems also benefit from predictive analytics on transacted data, and the fast data layer enables this type of real-time transaction analysis that can also incorporate big data result-sets. The big data layer provides insight on mountains of data to help with decision making and support traditional performance metrics or enable more advanced types of visualization and data science.

From Mainframe to Big Fast Data Architecture

Moving from mainframe to big, fast data is an evolution. A phased approach—step by step—is certainly the most recommended way of modernizing applications. It makes sense because it minimizes risk and better justifies investments. After working with many customers who face this problem, here is one approach we recommend.

1. Selecting the Pilot: Pick a Starting Point

As with most major initiatives, an initial use case or small scope should be used as a pilot to validate the architecture choices and prove a return for the overall project. The ideal project candidate should a) have little or no integration points with other systems on the legacy platform, b) be small but critical to existing business processes, c) consume a considerable amount of operational expenses, and/or d) represent a business risk in its current state. By screening this way, we should be able to deliver something of value to the business,  reduce OpEx, and make the improvement quickly while avoiding bad decisions.

2. Designing the Modern Data Architecture for Co-Existence

The goal of this step is determine what legacy data stays, migrates, or integrates. First, there is an analysis on the pilot’s data model. Then, we begin to design a data architecture that makes sense for a highly scalable, distributed data grid and still supports the existing business model and processes. The analysis should identify which entities are transactional, a mix of transactional and analytical (e.g. part of a real-time analytics model), or purely analytical. During this process, we make decisions regarding data model partitioning, replication, colocation, disaster recovery, transaction consistency, and more. We also decide which data to leave on the legacy platform, accessing it on the fly as needed using the GemFire integration layer capabilities.

3. Integrating Mainframe and Big Data Grid

While the data architecture is being defined, we start building the initial big fast data infrastructure. Then, the pilot migrates the first use case to the modernized architecture. By using the GemFire/SQLFire asynchronous integration layer, we can provide data consistency between the new application and legacy mainframe application. Transactions done on the modernized system are delivered simultaneously to both the legacy system and the analytics platform. Integration with legacy can be achieved using either a mainframe connector, CICS Web Services, messaging platform, or any other integration protocol.

4. First Deployment Risk Mitigation Plans

When the pilot is complete and has proven to be better performing than the legacy system with much lower maintenance costs, we are ready to partially turn off our first piece of the legacy system. The legacy system, especially if living on a mainframe, should stay there for a period of time to support ongoing business. During this time, new transactions should start happening on the new system and data can be validated against the original system to make sure it is behaving exactly as expected. This will minimize risk, assure a seamless architecture evolution, and avoid headaches from unexpected problems. While the deployment acts as an advanced, operational cache for the mainframe, the mainframe still receives the data it needs while both analytical and real-time or predictive analytics data stores are updated.

5. Evolution

Step-by-step, other applications or portions of the mainframe can be carefully migrated to the new platform in a similar manner—without risk. As this happens, we gradually reduce mainframe usage, costs, and time to market for new deployments. We gain a level of scalability proven by data grids that run the most rigorous and high-performance data environments on the planet—those that power financial transactions. We also enable new methods of analysis to unleash business insight and value.

Of course, there is an initial capital expense to make; however, the investment is justified by reduced operational expenses. Companies can also save on capex by leveraging existing, partially used infrastructure since the software runs on commodity hardware.

About the Author: Frederico Melo (a.k.a. Fred Melo) has a degree in Computer Science and has been working with Software Engineering for the last 14 years. His areas of expertise include Grid Computing, Highly Scalable Architectures, Big Data, Fast Data and Legacy Modernization. He is currently based in Sao Paulo, Brazil working as Field Engineer for Pivotal.

Pivotal is now open for business!

Pivotal, first announced in December, is a new venture started by VMware and EMC that is focused on Big Data and Cloud Application Platforms. Formally launched as a stand-alone entity today, Pivotal is led by former VMware CEO Paul Maritz, who has been working as Chief Strategy Officer at EMC since last August.

In a webinar today, Maritz not only confirmed the new initiative is now a stand-alone business with 1,250 employees from VMware and EMC, but he also surprised listeners with an announcement that General Electric is making a strategic investment of $105 million into Pivotal. GE’s Vice President and Corporate Officer Bill Ruh joined the webinar today and said GE will hold a 10% stake in the new company. CEO Jeff Immelt also joined the call to explain This brings the value of the newly launched Pivotal to $1 billion.

GE also announced this morning that their Software Center is standardizing on several of Pivotal’s technologies, essentially being the first public customer to endorse the new company.

GE and Pivotal have also entered a commercial agreement to begin a broad research and development project aimed to advance new analytic services and applications that support GE’s vision of the Industrial Internet. In a press announcement from GE earlier today, the company detailed more about what they plan to accomplish:

Over the past two decades, technology has connected people globally and created unprecedented opportunities for business and consumers. In the next decade, the Internet will also transform industries like aviation, rail, energy, oil & gas, infrastructure and healthcare, connecting human insight with machine intelligence, advanced analytics and low-cost-sensing to drive new levels of productivity and efficiency for global industries.

The Pivotal and GE have established an R&D facility in San Ramon, California that is already in operation with over 400 hundred employees developing new ways for industrial and medical products to use big data. The distribution of ownership among the companies will be EMC will own 62 percent; VMware 28 percent and GE 10 percent.

How and Why Pivotal Formed

Last August, Maritz left VMware to go lead a vision of how to seriously advance the industry to make Platform-as-a-Service widely appealing to the enterprise and dramatically accelerate adoption. Basically, Maritz wants to sell you “Google in a Box”. He wants to make it easy for you to achieve Google-style engineering, and become the next great software company that much more quickly.

Pivotal is aimed at helping companies get enormous jump-starts on development, and make it easy for applications to scale across cheap commodity servers in public and private clouds. It will make it easy for companies to be more like Google, Facebook, Twitter and Instagram without having to invent the development and production systems themselves. With Pivotal, they will have a highly-productive development platform and ultra-scalable production systems that deal with big, fast data inherently. And it will all be delivered as a service—ready to go when you are.

The appeal of this type of development environment will lower the barrier to entry for new innovative applications and right size the economics of investment for data center support based off of real usage. This idea is so appealing to companies and analysts that it is expected to be an $8 billion dollar industry this year—and will grow to $20 billion within the next 5 years.

Announcing Pivotal One: A Next Generation Enterprise Platform-as-a-Service (PaaS)

Armed with products from VMware including Cloud Foundry, Spring, and the vFabric middleware solutions, and EMC’s Greenplum and Pivotal consulting experts, Pivotal already has many solutions in place. However, today Pivotal’s Scott Yara revealed a plan for Pivotal One, the name of Pivotal’s next-generation Enterprise PaaS that will integrate new data fabrics, modern programming frameworks, cloud portability and support for legacy systems.

According to Yara, in order to achieve the next level of productivity, this solution will have to be targeted at the developer. Pivotal One will improve developer productivity, simplify and speed how applications deal with large data, and enable true cloud independence. Designed for the enterprise, it will also allow companies to still use legacy systems and even improve their performance.

Yara also outlined another key requirement placing data as the top priority. Modern applications will need to build the access and real-time analytics of data into every solution. Pivotal One will be investing heavily into big fast data solutions including Hadoop.

The first release of Pivotal One will be in Q4 of this year. Components of the Pivotal One Platform will include:

Pivotal Data Fabric

Pivotal HD provides integrated, advanced data services by integrating the industry leading massively parallel processing (MPP) Greenplum Database with enterprise-hardened Apache Hadoop. With enterprise data services like HAWQ and Pivotal in-memory data grid technology that make Hadoop more stable and usable, Pivotal HD reduces implementation costs for Hadoop in all but the most complex environments.

Pivotal Cloud and Application Platform

Based on Cloud Foundry and Spring, building applications with Pivotal builds in portability, scaling, automation, and resiliency. Pivotal Application Fabric bakes in rapid application development for messaging, application services and database services with the Spring developer ecosystem and analytic and visualization instrumentation. This platform will also feature other products acquired from VMware’s portfolio including Groovy, Grails, GemFire, SQLFire, RabbitMQ, tc Server, and Web Server.

Pivotal Expert Services

Pivotal Expert Services is the consulting arm of the operation that helps companies achieve agile development and sophisticated data analytics on a project-by-project basis.

Pivotal Labs

Pivotal Labs are the for-hire engineering team that are known for their industry-leading agile development methodology that have helped hundreds of companies to rapidly build and deploy modern mobile, web, and enterprise applications.

Data Science Labs

The Pivotal Data Science team is a specific team of consulting experts that will partner with businesses to accelerate analytics projects and exploit new value hidden in your data.

Open Source Support

Collaborative and customer-driven open source support and co-development.

Pivotal Open Source Software

Pivotal supports some of the most vibrant open source software communities in the world, including Spring, Cloud Foundry, RabbitMQ, Redis, OpenChorus, and more.

Training is a great way to speed up development, learn how to improve performance and usability for your applications and generally build confidence in your skills. This month, SpringSource is offering java developers a 15% discount code on all VMware trainings including Core Spring, Spring Web, Enterprise Integration, and Hibernate classes.

To secure your 15% discount, be sure to use the promo code springcustomerpromo during your registration process (promo is not available for partners). All of the following qualifying classes for May, 2013 can be found below:

Step 1: Core Spring

Americas

• May 07 – 10: Core Spring in Boston, MA
• May 07 – 10: Core Spring in Dallas, TX
• May 13 – 16: Core Spring in Phoenix, AZ
• May 13 – 16: Core Spring in Sacramento, CA
• May 13 – 16: Core Spring in San Francisco, CA
• May 13 – 16: Core Spring in San Jose, CA
• May 14 – 17: Core Spring in Washington, DC
• May 14 – 17: Core Spring in Washington, DC
• May 14 – 17: Core Spring in Los Angeles, CA
• May 14 – 17: Core Spring in Portland, OR
• May 21 – 24: Core Spring in Salt Lake City, UT
• May 21 – 24: Core Spring in Seattle, WA
• May 27 – 30: Core Spring in Bogota, Colombia
• May 28 – 31: Core Spring in Stamford, CT
• May 28 – 31: Core Spring in Charlotte, NC

Asia Pacific

• May 06 – 09: Core Spring in Canberra, Australia
• May 06 – 09: Core Spring in Hyderabad, India
• May 06 – 09: Core Spring in Bangalore, India
• May 14 – 17: Core Spring in Singapore, Singapore
• May 20 – 23: Core Spring in Sydney, Australia

Europe, Middle East & Africa

• May 07 – 10: Core Spring in Rome, Italy
• May 07 – 10: Core Spring in Lisbon, Portugal
• May 07 – 10: Core Spring in Madrid, Spain
• May 13 – 16: Core Spring in London, UK
• May 14 – 17: Core Spring in Kontich, Belgium
• May 14 – 17: Core Spring in Paris, France
• May 14 – 17: Core Spring in Stockholm, Sweden
• May 21 – 24: Core Spring in Zagreb, Croatia
• May 21 – 24: Core Spring in Berlin, Germany
• May 28 – 31: Core Spring in Prague, Czech Republic
• May 28 – 31: Core Spring in Hamburg, Germany

Step 2: Spring Web / Enterprise Integration with Spring / Hibernate with Spring

Americas

• May 06 – 09: Spring Web in Edmonton, AB
• May 06 – 09: Spring Web in Phoenix, AZ
• May 06 – 09: Spring Web in Sacramento, CA
• May 06 – 09: Spring Web in San Francisco, CA
• May 06 – 09: Spring Web in San Jose, CA
• May 06 – 09: Spring Web in Phoenix, AZ
• May 21 – 24: Enterprise Integration with Spring in Ottawa, ON
• May 21 – 24: Enterprise Integration with Spring in Toronto, ON
• May 21 – 24: Enterprise Integration with Spring in Montreal, QC
• May 21 – 24: Enterprise Integration with Spring in Edison, NJ
• May 21 – 24: Spring Web in Chicago, IL
• May 28 – 31: Enterprise Integration with Spring in Los Angeles, CA

Europe, Middle East & Africa

• May 06 – 08: Hibernate with Spring in Wien, Austria
• May 06 – 08: Hibernate with Spring in Berlin, Germany
• May 06 – 08: Hibernate with Spring in Zurich, Switzerland
• May 07 – 10: Enterprise Integration with Spring in Amsterdam, Netherlands
• May 14 – 17: Enterprise Integration with Spring in Munich, Germany
• May 20 – 23: Enterprise Integration with Spring in London, UK
• May 21 – 24: Enterprise Integration with Spring in Dublin, Ireland
• May 27 – 29: Hibernate with Spring in Munich, Germany
• May 28 – 31: Enterprise Integration with Spring in Brussels, Belgium
• May 21 – 24: Spring Web in Prague, Czech Republic
• May 21 – 24: Spring Web in Rome, Italy
• May 21 – 24: Spring Web in Amsterdam, Netherlands

Live Online

• May 06 – 09: Spring Web Online in Americas
• May 21 – 23: Hibernate with Spring Online in Americas
• May 21 – 24: Enterprise Integration with Spring Online in Americas

Note: If you cannot find a professional training near you, you can always request an onsite SpringSource training

Yet, legends can sometimes be surrounded by myths—these myths can lead IT executives down a path with rose-colored glasses.

Data and data usage is growing at an alarming rate.  Just look at all the numbers from analysts—IDC predicts a 53.4% growth rate for storage this year, AT&T claims 20,000% growth of their wireless data traffic over the past 5 years, and if you take at your own communications channels, its guaranteed that the internet content, emails, app notifications, social messages, and automated reports you get every day has dramatically increased.  This is why companies ranging from McKinsey to Facebook to Walmart are doing something about big data.

Just like we saw in the dot-com boom of the 90s and the web 2.0 boom of the 2000s, the big data trend will also lead companies to make some really bad assumptions and decisions.

Hadoop is certainly one major area of investment for companies to use to solve big data needs. Companies like Facebook that have famously dealt well with large data volumes have publicly touted their successes with Hadoop, so its natural that companies approaching big data first look to the successes of others.  A really smart MIT computer science grad once told me, “when all you have is a hammer, everything looks like a nail.” This functional fixedness is the cognitive bias to avoid with the hype surrounding Hadoop. Hadoop is a multi-dimensional solution that can be deployed and used in different way. Let’s look at some of the most common pre-concieved notions about Hadoop and big data that companies should know before committing to a Hadoop project:

1. Big Data is purely about volume—NOT TRUE

Besides volume, several industry leaders have also touted variety, variability, velocity, and value. Putting all arguments about alliteration aside, the point is that data is not just growing—it is moving further towards real-time analysis, coming from structured and unstructured sources, and being used to try and make better decisions. With these considerations, analyzing a large volume of data is not the only way to achieve value. For example, storing and analyzing terabytes of data over time might not add nearly as much value as analyzing 1 gigabyte of really important, impactful information in real time. From a tool-set perspective, you might want an in-memory data grid built for real-time pricing calculations instead of a way to slice and dice historical prices into a dead horse.

2. Traditional SQL doesn’t work with Hadoop—NOT TRUE

When Facebook, Twitter, Yahoo! and others bet big on Hadoop, they also knew that HDFS and MapReduce were limited in their ability to deal with expressive queries through a language like SQL. This is how Hive, Pig, and Sqoop were ultimately hatched. Given that so much data on earth is managed through SQL, many companies and projects are offering ways to address the compatibility of Hadoop and SQL. Pivotal HD’s HAWQ is one example—a parallel SQL-compliant query engine that has shown to be 10 to 100s of times faster than other Hadoop query engines in the market today—and it was built to support petabyte data sets.

3. Kill the Mainframe! Hadoop is the only the new IT data platform—NOT TRUE

There are many longstanding investments in the IT portfolio, and the mainframe is an example of one that probably should evolve along with ERP, CRM, and SCM. While the mainframe isn’t being buried by companies, it definitely needs a new strategy to grow new legs and expand on the value of it’s existing investment. For many of our customers that run into issues with mainframe speed, scale, or cost, there are incremental ways to evolve the big iron data platform and actually get more use out of it. For example, in-memory, big data grids like vFabric SQLFire can be embedded or use distributed caching approaches for dealing with problems like high-speed ingest from queues, speeding mainframe batch processes, or real-time analytical reporting.

4. Virtualized Hadoop takes a performance hit—NOT TRUE

Hadoop was designed originally to run on bare metal servers, however as adoption has grown many companies want it as a data center service running in the cloud. Why do companies want to virtualize Hadoop? First, let’s consider the ability to manage infrastructure elastically—we quickly realize that scaling compute resources, like virtual Hadoop nodes, help with performance when data and compute are separated—otherwise, you would take a Hadoop node down and lose the data with it or add a node and have no data with it. Major Hadoop distributions from MapR, Hortonworks, Cloudera, and Greenplum all support Project Serengeti and Hadoop Virtualization Extensions (HVE) for this reason. In addition, our research with partners has show that Hadoop works quite well on vSphere and can even perform better under certain conditions—running 2 or 4 smaller VMs per physical machine often resulted in better performance, up to 14% faster, than a native approach according to benchmarks we’ve done with partners.

5. Hadoop only works in your data center—NOT TRUE

First of all, there are SaaS-based, cloud solutions, like Cetas, that allow you to run Hadoop, SQL, and real-time analytics in the cloud without investing the time and money it takes do build a large project inside your data center. For a public cloud runtime, Java developers can probably benefit from Spring Data for Apache Hadoop and the related examples on GitHub or online video introduction.

6. Hadoop doesn’t make financial sense to virtualize—NOT TRUE

Hadoop is typically explained as running on a bank of commodity servers—so, one might conclude that adding a virtualization layer adds extra cost but no extra value. There is a flaw in this perspective—you are not considering the fact that data and data analysis are both dynamic. To become an organization that leverages the power of Hadoop to grow, innovate, and create efficiencies, you are going to vary the sources of data, the speed of analysis, and more. Virtualized infrastructure still reduces the physical hardware footprint to bring CAPEX in line with pure commodity hardware, and OPEX is reduced through automation and higher utilization of shared infrastructure.

7. Hadoop doesn’t work on SAN or NAS—NOT TRUE

Hadoop runs on local disks, but it can also run well in a shared SAN environment for small to medium sized clusters with different cost and performance characteristics. High bandwidth networks like 10GB Ethernet, FoE, and iSCSI can also support effective performance.

Taking Action to Overcome the Myths

While many of us are fans of big data, this list can help you take a step back and look objectively at the right approach to solving your big data problems. Just like some building projects need hammers and others need screwdrivers, hacksaws, or a welding torch, Hadoop is just one tool to help conquer big data problems. High velocity data may push you towards an in-memory, big data grid like GemFire or SQLFire. A need for massive, consumer-grade web scale may mean you need message-oriented middleware like RabbitMQ. Getting to market faster may mean you need to look at a full SaaS solution like Cetas, and Redis may meet your needs and find a home in your stack much easier than a full blown Hadoop environment.

To learn more about the products in this article:

• Read over 100 articles about GemFire or SQLFire
• Check out the case studies on RabbitMQ
• See the Pivotal HD product page or the Hadoop Virtualization pages on VMware.com
• Learn more about Hadoop in the cloud with Cetas
• Find out more about Redis

In a single laptop benchmark, Salvan ran four different scenarios in order to obtain some insight on performance of the default setups for these messaging solutions. Each test had 1 process dedicated to enqueuing and another dedicated to dequeuing. The message volume and size ranged from 200 to 20,000 to 200,000 messages and 32 to 1024 to 32768 bytes. Both persistent and transient queues and messages were used.

The main points are below, however you can also check out Salvan’s blog post to see the source code, charts, and a data sheet.  Here were the highlighted results:

• Brokers perform much better with fewer, bigger messages.
• Persistence drawbacks appear with big messages, and time on small/medium messages is spent on processing not I/O.
• ZeroMQ’s simple feature-set delivers great performance.
• QPID seems to perform the best when persistence is not used.
• AMQP seems more optimized than STOMP.
• RabbitMQ seems to outperform others by a factor of 3 except for the case of big messages.

Adina’s related presentation is titled, “Messaging Systems—How to make the right choice?” In it, she covers the following:

• What are messaging systems?
• How messaging helps with performance, complexity, scalability, quality, cost-effectiveness, high-availability, and messaging patterns
• Comparison of the Java Message Service API, AMQP, and STOMP with code examples
• Trade-offs
• Functional and operational requirements
• The benchmark results

In one slide, Adina highlights the benefits of message oriented middleware in the cloud, a SaaS model, and explains how overhead is reduced and scale is introduced with greater simplicity and lower cost when messaging takes place in the cloud. Her examples include Amazon Simple Queue Service,  CloudAMQP, and StormMQ. To add commentary to this slide, we’d like to also point out that RabbitMQ is available in the cloud via Cloud Foundry and Pivotal along with MongoDB, Redis, and there is also support for Scala and the Play Framework. With RabbitMQ and Cloud Foundry, there are some pretty cool things worth pointing out:

• You can use Node.js with the Cloud Foundry RabbitMQ service
• In Java, you can use Grails with RabbitMQ on Cloud Foundry or build Spring apps that hook up to the RabbitMQ service
• With Ruby on Rails, you can develop in Ruby or Sinatra  via the bunny gem and access RabbitMQ services
• There is a new simulator, video, and open source bits available for RabbitMQ and a how-to article for deploying the RabbitMQ simulator on Cloud Foundry.

More About RabbitMQ Performance

Less than a year ago, we began publishing some information about RabbitMQ’s performance and will summarize it here and begin to build a collection of RabbitMQ performance information.

In part one, we measured performance on a single PowerEdge R610 with dual Xeon E5530s and 40GB of RAM, RabbitMQ 2.8.1 and Erlang R15B with HiPE compilation enabled and the code is available. We showed the difference in send rate, receive rate, and average latency improvements between RabbitMQ 2.7.1 and 2.8.1 due to a capability called internal flow control and significant memory improvements.

In part 2, we outlined how different features affect performance using smaller messages. For a baseline, we showed auto-ack of 44,824 messages per second with one producer and one consumer. Looking at aspects of performance independently, we can publish at 53,710 messages per second with no consumption and consume 64,315 messages per second stand-alone. We shared performance number differences from configuring the mandatory flag and immediate flag impact rates as well as acknowledgements, publish confirms, and message persistence. The diagram below explains how message send rate and bytes rate change with message size—message rates drop with size increases, but the number of bytes sent increases. You can read more about message size and horizontal scale prefetch counts in this portion of the post.

In the article, we go on to publish information for scenarios with large queues and paging—we load and drain 500,000 messages, 10,000,000 messages to show where the bottlenecks occur. There are also other in-depth performance resources on the RabbitMQ blog and you can read over 45 case studies, tutorials, and updates on RabbitMQ at the vFabric Blog.

Of course, in the spirit of community, if you’ve run across any great performance articles on messaging and RabbitMQ, we would love to hear from you and share your stories on this blog.  To let us know, leave a comment below and we will contact you.

For more information on RabbitMQ:

• Check out the vFabric RabbitMQ product page
• Learn about the New vFabric Reference Architecture and how RabbitMQ fits
• Download a trial of RabbitMQ
• Read how Rabbit is integrated with over 100 developer tools and platforms
• Check out case studies on how Instagram, HuffingtonPost Live, Roblox, and Indeed.com scale with RabbitMQ.

Why? In a nutshell, companies like McKinsey believe the use of big data and technologies like Hadoop will allow companies to better compete and grow in the future.

Hadoop is used to support a variety of valuable business capabilities—analysis, search, machine learning, data aggregation, content generation, reporting, integration, and more. All types of industries use Hadoop—media and advertising, A/V processing, credit and fraud, security, geographic exploration, online travel, financial analysis, mobile phones, sensor networks, e-commerce, retail, energy discovery, video games, social media, and more.

At first glance, it sounds like many of the above business needs were already solved by conventional data warehouses, business intelligence, and statistical analysis programs. This is not the case—the conventional systems begin to fail when the data sets become too large, include fast-growing unstructured data formats, or face both of these issues. With size and complexity issues, traditional BI systems can become too expensive. This is why Hadoop was invented.

Simply put, Hadoop follows the MapReduce model to slice data into chunks of work, spread the work across a large number of commodity servers, and aggregate the work back into a single output. It’s parallel computing approach out-scales the old models and is more cost-effective at doing so.

Effectively Managing Hadoop from the CFO’s Eyes

In the early days of the web and enterprise apps, everyone got so enamored by the potential for growth and productivity that both business and IT teams spent money prematurely—we ended up with a massive number of underutilized servers that cost us an arm and a leg to operate. Then, we spent more to virtualize these resources, get better utilization out of our datacenters and reduce our overhead.

With the big data technology trend, we are facing the same excitement around Hadoop. It’s going to be an investment area for the next decade or two, and your CFO is going to see this coming. This time around, we can spend IT dollars much more wisely buy putting Hadoop on virtualized infrastructure from the beginning. For those of us that have learned the painful TCO lessons from the past and understand the economics of virtualization, here is a list of ten key, financially sound, cloud infrastructure requirements that should be part of any Hadoop project:

1. Initial Hadoop projects should be explored for the most pressing issues in the company and start by aligning with the CEO and CFO’s top needs and goals.
2. Hadoop investments should run with the same data center efficiency and cost effectiveness as other virtualized platforms that have high server consolidation ratios and require less CapEx and OpEx than non-virtualized environments.
3. Hadoop pilots should identify a big problem, make the scope concise, and complete quickly to prove the time-to-value and identify future costs and risks thoroughly. We all learn by doing—don’t drag out the time to value by over-engineering.
4. Hadoop must be able to co-locate with existing applications and run on existing virtualized hosts. This approach should accommodate a Hadoop pilot without new hardware or help manage shared infrastructure budgets in a cost-effective manner.
5. Hadoop nodes should use the concept of time sharing. For example, when email, database, web, or ERP applications are idle, the compute power available should be transferred to Hadoop nodes that are analyzing improvements in business performance.
6. The Hadoop infrastructure should be able to scale up or down elastically, on-demand, and across clouds for burst compute needs. This capability would allow you to expedite a big analysis on your company’s performance by temporarily adding new Hadoop nodes on a 3^rd party cloud service to increase capacity.
7. Hadoop VMs should not require significant resources to scale, provision, deploy, replicate, or move because a cloud-centric, virtual machine infrastructure can accommodate this.
8. Hadoop should be available to the company as a shared service. This is one of the most cost-effective ways to provide Hadoop as a service. In this model, it is available to all departments based on chargeback accounting. Even with shared services, virtualization still allows for enough isolation to meet independent business and security needs.
9. Hadoop should not require expensive, high availability or fault tolerance (i.e. no downtime) frameworks based on hardware. Distributed computing is meant for commodity computing in the cloud.
10. Hadoop training, at least at a high level, should be provided to every IT person who engages with various business units and departments—Hadoop attracts talent and paves careers.

To learn more about how VMware is helping virtualize Hadoop clusters, check out Project Serengeti.

The Pivotal platform will unite data, application, and cloud fabrics, helping enterprises to develop faster, understand more, and succeed at an even greater scale. It is a platform that makes the consumer grade enterprise a reality.

Pivotal brings together a prodigious set of technologies and talent from a number of EMC and VMware entities, which include Greenplum, Cloud Foundry, Spring, GemFire and other products from the VMware vFabric Suite, Cetas, and Pivotal Labs.

Paul Maritz, the Pivotal Leadership Team, and special guests will unveil this platform, and make a special announcement during a live streaming event on Wednesday, April 24th at 10:00 am Pacific/1:00 pm Eastern.

Sign up for the event at gopivotal.com and follow @gopivotal on Twitter for updates.

The world’s largest banks have historically relied on mainframes to manage all their transactions and the related cash and profit. In mainframe terms, hundreds of thousands of MIPS are used to keep the mainframe running these transactions, and the cost per MIP can make mainframes extremely expensive to operate. For example, Sears was seeing the overall cost per MIP at $3000-$7000 per year and didn’t see that as a cost-effective way to compete with Amazon. While the price of MIPS has continued to improve, mainframes can also face pure capacity issues.

In today’s world of financial regulations, risk, and compliance, the entire history of all transactions must be captured, stored, and available to report on or search both immediately and over time. This way, banks can meet audit requirements and allow for scenarios like a customer service call that results in an agent search for the transaction history leading up to a customer’s current account balance. The volume of information created across checking, savings, credit card, insurance, and other financial products is tremendous—it’s large enough to bring a mainframe to its knees.

Mainframe Jam can be Sticky

No, this is not some type of jelly you find in a mason jar at a farmer’s market in Armonk, NY.

Just like other data stores, a spike in resource requests can bring a system to a halt. When mainframes get jammed and go offline for a few minutes—it costs banks millions of dollars per minute. Mainframe jams eventually cause the executive team to drop what they are doing until they know the CIO and IT organization are all focused 100% on removing the jam and working to avoid it in the future.

What causes a mainframe jam? In the case of one of Latin America’s largest banks, it is simply the volume of data. Their mainframe runs transactions and creates logs for compliance, risk, and audit purposes. These logs are sent to a message queue and database that also runs on the mainframe hardware. If this database is overloaded, the messages queue up, and the system begins to slow down until it begins to deny transactions.

Making Big, Fast Data Scale for Financial Services Mainframes

What the bank needed is a way to ingest a large number of transaction log messages and never deny transactions. Valued at over $60 billion, their client’s transactions generate a message volume of around 200-300 million per day with peaks of up to 15,000 messages per second. With each message being about 4KB of data, this totals up to be about 1.2 terabytes per day of financial transactions.

This is big data for sure, but it was not yet fast data.

Architecture Approaches with Pivotal’s vFabric GemFire and Greenplum

The VMware vFabric team partnered with the bank to show them how this data jam could be avoided in the future by building out a proof of concept.

Below you’ll find an architecture diagram of the proposed solution. We used vFabric GemFire as the fast ingest layer to consume MQSeries messages right as they hit the server. This prevents messages from queueing up and also allows them to be asynchronously persisted to a Greenplum appliance for analytics. For the customer, this deployment worked like a black box because GemFire is embedded inside the Greenplum hardware appliance, so setup was minimal and easy.  vFabric tc Server, Spring Integration, and RabbitMQ could also be used to host various web services. .

Basically, this solution allowed the high volume set of log events and messages need to be stored on a persistence layer in GemFire outside the mainframe that could scale elastically as needed by simply adding more nodes.  Since the data is placed in-memory with GemFire, it also allows the data to available for search or reporting as soon as it leaves the MQSeries queue.

The bank saw several additional advantages to running a big, fast data grid outside the mainframe:

• Running the messages outside of the mainframe would save MIPS and money for a set of data that was not considered core to the business operations.
• The mainframe would be able to handle a much higher transaction throughput, avoid jams and have a greater ability to scale on it’s current hardware.
• Business intelligence analyses could be run on the transaction logs. Once this data could be placed on a big data grid outside the mainframe, it would allow various departments to do analysis on customer profiles, usage characteristics, fraud, security, customer service, and marketing.

For more information on vFabric GemFire and Greenplum, see:

• GemFire
• Greenplum

About the Author: Frederico Melo (a.k.a. Fred Melo) has a degree in Computer Science and has been working with Software Engineering for the last 14 years. His areas of expertise include Grid Computing, Highly Scalable Architectures, Big Data, Fast Data and Legacy Modernization. He is currently based in Sao Paulo, Brazil working as Field Engineer for Pivotal.