Blogging about web services and cloud computing may not be as much fun as writing a commercial for a tasty treat, but today’s announcement that Solace has partnered with Layer7 Technologies reminded me of that same idea: two things that are independently useful, and even better together.

When companies extend to the web using either web services or cloud computing there is generally a division between the management of security and policies for these outward-facing services and the interactions with in-house application components that comprise the service. It’s not a new problem, but many firms are still struggling with the implementation of large projects, and throwing racks of servers and armies of people at the problem to achieve necessary performance, reliability and scalability.

Appliances have taken off in the last few years because they make some pretty complex problems do-it-yourself friendly. Layer 7 has built a successful business providing turnkey solutions for web services, SOA, and cloud security, while Solace has made enterprise service buses rack-and-run simple with unrivaled performance and scalability. Before today, each company had a compelling appliance solution for one side of the equation.

Now Solace and Layer 7 have connected the peanut butter with the chocolate, so to speak, to provide an easy-to-use and truly end-to-end platform for enabling communications between internal and external systems and users. Two great tastes that taste great together!

This architecture is applicable to many applications and industries, for example:

• Government — Secure, inter-agency communications are a top priority for the U.S. government and many others around the world. National security and emergency response both rely on connecting the dots between information sources in real time so people can identify and respond to risks and emergencies as they take shape.
• Retail banking — Automating consumer services with a secure, high-performance and reliable web experience is essential for modern retail banking.
• Telecommunications — Mobile services and highly personalized provisioning and billing are the future for telecommunications providers worldwide.

Each of these industries faces the same fundamental challenge of securely delivering real-time information and interactive services to their (often huge) audience. We may not have groovy headphones or cowboy hats, but just like in the videos, the feedback from our customers suggests we’re onto something with this combination.

The press headlines are (mostly) celebrating passage of the financial reform act. I even got a bulk email from the president saying it was a great day for the little guy and a bad day to be a banking special interest guy. Then he asked me for 5 bucks and encouraged me to send the note on to 6 other people in my town. Wait, did the president just send me a chain letter?

But the reality is that financial reform is not at the finish line; the practical side of reform hasn’t even glimpsed the starting line. Financial reform is an agreement that more regulation and consumer protections need to be in place, but does little to define what those regulations or protections might be. There are some broad stroke intents related to making derivatives trading more transparent and changes in capital requirements that are fairly straight forward. But what information regulators will require from banks or specifics on how risk management will change are not yet understood. We don’t even know which bodies will do the regulating, or what their mandates will be.

The key for the banks will be in becoming more nimble and adjusting as the rules and regulations change and morph over the next several years. About a decade ago, corporate agility was at the forefront of business requirements, primarily because business opportunities were changing so quickly that firms with inflexible systems were being left in the dust. More recently cutting costs for higher profits has been in vogue, but we will almost certainly see corporate agility (probably disguised as some new buzzword) move back up the priority charts as a result of the Dodd-Frank act. Little else of the specifics are known, aside from the assurance that, as with all new regulations, lawyers and consultants will benefit.

To find out more about how to be ready for financial reform, please send me $5 and forward this blog post to 6 people inside your firm

A lot has changed, though, and CEP offerings have evolved into the equivalent of next generation app servers. Those early customer deployments of many years ago, along with years of refinement and optimization, have made CEP an essential piece of the financial application puzzle.

There are a lot of CEP engines out there, because each industry has unique types and patterns of events, and it takes considerable industry expertise to provide a solution that can make sense of them. So where one CEP provider may excel at government use cases, another may hire experts from health care and focus on that industry. In capital markets, the clear leader is StreamBase. They have deep knowledge around, and solutions for, a wide range of financial use cases including FX data integration, profit and loss analysis, risk management, smart order routing, algorithmic trading, and compliance.

Today we announced a partnership with StreamBase to integrate their CEP solution with our hardware messaging offerings. A publish/subscribe messaging bus like ours is the best place to collect and aggregate events, and the natural way to distribute them for processing. So by pairing their CEP expertise with our best-of-breed messaging platform, we’ll accelerate time to production and reduce TCO for our mutual customers.

CEP engines work best when they get high value data, and hardware messaging is exceptional at filtering millions messages of raw input per second down to just the stream of information relevant to that application. This makes a very elegant scaling architecture, because both assets are used for what they are best at. Solace for filtering and distributing events, and StreamBase for real time analysis, correlation and decision making.

This alliance brings value to two key sets of customers:

• Solace customers that are tired of custom coding their complicated financial applications. StreamBase makes application development faster and easier, without sacrificing performance.
• Streambase customers that are having problems with the middleware (performance, scalability, TCO) feeding the event machine.

We have many mutual customers on the sell side and the buy side, and one commonality is that they’re always looking to improve their ability to reacting to changes to markets, data rates and regulations without continually reworking their applications.

Today’s CEP is sexy technology that can do a lot to make any business better, and with a massively scalable event backbone underneath it, the sky’s the limit. We look forward to working with StreamBase to bring many innovative new applications to capital markets.

In fact, the handling of this crisis is giving us a glimpse of the future of safety in open water drilling, now that money is less of an object. Here are a couple of recent articles that caught my eye:

• BP oil spill update: sensors measure spill — BP has installed sensors near the wellsite to improve their ability to estimate how much oil is spilling into the gulf. Some of the harshest criticisms have been around their inability to accurately determine how bad the situation is at “ground zero”. Sensors will help them more accurately understand oil flows and estimate what percentage they are capturing, as well as what degree of response is needed for the remainder.
• ‘Gliding’ robots patrol Gulf oil spill — BP has deployed water drones to swim around the gulf and measure data such as temperature, salinity and organic materials. By providing a much more accurate and real-time picture of the oil plume, that data will aid the planning and execution of containment and cleanup efforts.

If you just consider the financial and political cost of a single event like this, it would surely be much cheaper to outfit the areas subject to offshore drilling with a permanent set of sensors that can report on ocean quality and contaminants minute by minute, instead of two months after the next spill. The nice thing about sensors is they work cheap — once deployed they continuously collect data without interruption. And that data can be mined for a wide range of uses beyond the petroleum industry, helping with hurricane predictions and general environmental study.

This is not at all far fetched. As the cost of sensors and networks have come down, there’s been an explosion in fixed and mobile sensors, and the systems that are used to process the data. Also, there are already large scale efforts to monitor the oceans for environmental changes over time.

Every sensor network drives two complimentary needs: to identify long term trends using after-the-fact data mining  and to identify situations of immediate concern by analyzing real-time event streams. Solace is currently deployed in large scale sensor networks monitoring seismic and temperature events as well as aiding in efforts to protect major US cities from security threats.

There are many more projects where sensors are generating massive amounts of data that’s fed into a federated information bus for distribution to real-time analytics and data warehouses. Applications related to RFID-based inventory tracking,  smart energy grids, video surveillance, transportation flow & scheduling, logistics tracking and more. Location is an important factor in every one of these use cases, because the “where” and the “who cares?” aspect of sensor exceptions are at least as important as the raw data captured. In the consumer space we’re just beginning to tap the most abundantly deployed sensors in the world: our GPS-enabled phones. There is a staggering amount of possibility sitting in your pocket right now that will continue to change how we live, work and play.

That this trend is a harbinger of the future world doesn’t exactly take a lot of vision, it’s already happening. When we look back in 15 years at the sensors in the physical world, it will be like taking inventory of the Internet today vs its fledgling days of 1995. People were aware of the Internet, mostly as an email gateway, but few could envision the scale of information growth and accessibility that would give rise so services like Google Earth, Slacker, Wikipedia and YouTube.

The gulf oil spill has put the government between a rock and a hard place when it comes to offshore drilling. From the public’s perspective, this is like another 3 mile island, yet they won’t volunteer to give up their cars or turn off their heaters in the winter, which means demand for oil will continue to increase. Proactive monitoring of the offshore drilling zones with ocean sensors may be the best approach to regulating the offshore drilling industry while providing increased protection for the areas affected.

So when it comes to keeping track of the behavior of their trading systems, looking in the rear view mirror doesn’t cut it. Firms need to be on top of any problems that affect their trading operations, whether they’re in the acquisition and flow of market data, the automated execution of trades, or the pre and post-trade risk management that keeps things in balance.

For Solace customers engaged in high-speed trading, one of our major advantages is the highly-granular real-time operational visibility we give system administrators. Our solution uses takes advantage of the parallel nature of hardware to provide real-time per-client statistics that software-based systems can’t without impacting performance, and many of which can’t be collected at all in multicast environments.

For example, at the TCP layer our solution reports round trip time, bytes sent and received, queue depths, number of retransmits, and number of packets received out of order. And at the messaging layer, we provide instantaneous and high water mark data about queue depths, messages transmitted and received per second, and messages discarded due to queue depth problems. This information can help administrators quickly and efficiently identify and address the root cause of a wide variety of situations regardless of whether they are due to IP network problems or client application problems. This is all good stuff for middleware operations teams, but we fully realize that the visibility we provide stops at the edges of the messaging bus. The nature of middleware is that it always plugs into some other set of applications that comprise the end-to-end operational picture.

One of the more popular end-to-end monitoring solutions available is Geneos from ITRS Group. In their own words, their Geneos platform lets financial institutions “Gain control of every aspect of your environment: servers, applications, network hardware and critical network interfaces, as well as application behaviours and workflow characteristics — in a single solution.”

We’ve partnered with ITRS to let mutual customers monitor the behavior and performance of their Solace-based messaging environment as a seamless part of their environment. Together, we developed a NetProbe polling agent that directs operational statistics and monitoring data to the ITRS Geneos Active Console. The addition and integration of ITRS software to the Solace platform gives middleware operations teams as-it-happens alerting, and some seriously robust statistics for capacity planning.

A big thanks to the banking customers that originally suggested we integrate with ITRS. Geneos as a monitoring platform has a lot going for it: it’s innovative, widely adopted and well regarded within its installed base. With ITRS in our corner, high-speed trading firms have a powerful weapon to avoid becoming trading roadkill.

A bit of history will help to put V5 in context. In our V3 release, the majority of features were aimed at creating hardware versions of features previously available in software products. There is a core set of capabilities that you just must have to be in the game, and our innovation then was focused on making existing messaging features faster or easier to deploy and manage. In V4 about half of the features introduced were rounding out legacy messaging capabilities, and the other half was introducing new capabilities.

With version 5 the majority of our major new features are innovations that have not previously been available in any messaging platform. At this point, we’ve met or surpassed the important features equivalence of legacy software platforms and we’re taking advantage of what’s unique about hardware to give customers capabilities that haven’t been available in commercial messaging offerings.

There are literally dozens of little features in V5.0 but below is a summary of the major ones we’re highlighting:

• Message eliding: this is essentially rate control for market data over slow or bandwidth constrained links. For example, if a market data instruments is changing 100 times per second and you only want to see 10 updates per second, only the most recent tick at each 10th of a second will be sent. Common use cases include streaming market data to human traders at a personalized rate or rate limiting traffic over a WAN. To date, applications above the messaging system have had to each provide this functionality, now it is available to any application that needs it with a simple configuration option in the router.
• Solace router virtualization: this enables administrators to split one physical Solace deployment into as many as 500 virtual instances, each with separate security and operational characteristics. Virtualization is central to the Unified Messaging Platform since it allows multiple applications and groups to share equipment without compromising security or performance of each. Once the first applications are deployed, additional applications can be virtualized on the same equipment using a simple configuration change. This also allows messaging to be easily deployed as a secure, shared network service within a cloud or software-as-a-service environment.
• Plug-and-play routing: for larger Solace deployments, we have made a number of efficiencies in how Solace routers learn about each other and synchronize the global network-of-Solace-routers state than in prior versions. This is unique to Solace’s message “router” concept, and part of our heritage, with key engineers and designers from the world of network equipment. While most software brokers are bespoke and have no knowledge of the world beyond the function they perform, Solace message routers have always been designed to be aware of their own role and the other message routers in the network.
• WAN optimizations:  We have made a wide range of improvements to messaging behavior over WAN links to compress messages in hardware, and optimize latency over using multiple parallel network connections.

We’re very pleased to bring these advances to our customers and prospects. In truth, you all helped bring many of them to us with your excellent feedback, requirements and suggestions.

Oh, and one more thing… Unlike Apple who made you wait 2 weeks from announcement to availability and requires you to buy a new iPhone to get the cool new features, Solace V5.0 is available in GA now and is free to anyone with current maintenance.

The first day of the event was open to the public, and over 50 attendees came to see what’s happened with AMQP over the past year.  The agenda for the day included technical tutorials on the AMQP1.0 wireline specification, some sharing of user experiences, and a discussion of plans for 2010.

In summary, after six years in development the AMQP1.0 wireline spec is now in “recommended” state. That means it is frozen and the only technical changes will be errata. That’s important because any required changes must be addressed in a backward compatible manner if at all possible, so people can now implement the wireline protocol without worrying so much about the need to redevelop things.  The spec will move to “final” when there are two independent and interoperable implementations.

AMQP1.0 is a significant change from previous 0.x versions where things like “exchanges” existed.  The new architecture has “nodes” and “links”, with “sessions” and “connections” being used to provide a framework for message exchange.  It includes capabilities such as the definition and encoding of data types (which can be used for things like JMS data types), flow control, flow (de)multiplexing on a single connection, persistent/durable delivery, security, and local as well as global transactions.   This is all spelled out in great detail, and in some cases the functions have even prototyped by group members, which represents a ton of effort put forth over the past year.

The next step is for the group to build upon the foundation of 1.0:

• The group will specify how the AMQP1.0 wireline protocol can be used to have a JMS API from one vendor talk AMQP1.0 to a broker from a second vendor, and then to an API from a third vendor which could be JMS or WCF.  Both API and wireline interoperability through a broker — now that level of interoperability in the messaging world is seriously worth watching!
• The group will also focus on broker management, i.e. standardizing how brokers can be managed by users by — you guessed it — sending AMQP messages to a special “management” entity within the broker.

All in all, the three day meeting was upbeat and positive, demonstrating a lot of progress and bringing more clarity to the impressive potential of AMQP. And as if the weather gods were paying attention to our sessions, the clouds parted and the sun came out as the discussion moved on to plans for the future of AMQP in 2010 — surely a sign of good things to come.

• Performance — for high-volume, highly repetitive tasks, special-purpose chips such as FPGAs, network processors and GPU’s have consistently been shown to outperform software running on general purpose CPUs. For many use cases, especially relating to market data and trading, performance alone is enough of a justification to choose hardware.
• Simplicity — the turnkey nature of many appliances such as messaging middleware, ticker plants, monitoring tools and security enforcement is appealing to many firms. They are so much easier and less costly to procure, deploy and configure that firms can focus on what they do best instead of getting bogged down with implementation details.
• Low TCO — appliances can often do the work of many equivalent servers running software, which significantly reduces the overall cost of operating an application or infrastructure.

But let’s be honest: simplicity and low cost don’t grab headlines like performance. If you have the choice to read a story about “12% cost reduction” or one about “performance measured in nanoseconds“, which one are you going to read? This is the equivalent of a World Cup commentator trying to convince the casual fan that Ronaldo is a special soccer player because of his mid-field play, while all we want is to see him crush the ball into the top left corner. Performance is where the sex appeal is, and it’s where most appliances make their name.

With that background, I am pleased to welcome NovaSparks as a new Solace partner. NovaSparks makes a hardware ticker plant appliance that accelerates and simplifies the processing of data feeds, and a trade order book for distribution to a wide range of trading applications. Sure, like other kinds of appliances NovaSparks is easier than software ticket plants and features lower TCO, but it is their eye-popping performance that will get your attention.

Ticker plants usually plug into a messaging layer for distribution, so Solace and NovaSparks have defined an architecture for integrating our two products to help customers move one step closer to a uber-performant end-to-end solution. To extend my prior metaphor, connecting our two high-speed appliances is like England’s Defoe slicing the ball from the corner to a deftly redirected header from Rooney for a game winning gooooooooooaaaaaalllllll!!!

I found the following self-description on the web that sums them up nicely:

Two Sigma Investments is a leading technology and finance firm founded in 2001. At Two Sigma, we work each day in small teams to develop and apply disciplined, process-driven investment trading strategies. Our casual culture celebrates rigorous thought and encourages true innovation. While we’re focused on the financial markets, technology is the driving force behind our business. In fact, many say that our unique SoHo offices in Manhattan have the look and feel of a software firm. Sounds good to us!

We are pleased to welcome Two Sigma to our growing list of hedge fund customers, and we thank them for allowing us to publicly share the details of their selection of Solace.

In economics and finance, arbitrage is the practice of taking advantage of a price difference between two or more markets: striking a combination of matching deals that capitalize upon the imbalance, the profit being the difference between the market prices.

For example, if an algorithmic trading system takes in currency feeds from New York and London and spots a disparity between the two, it can buy on one market and short on the other until the disparity disappears. Traders have been doing this for decades: finding, and taking advantage of, market inefficiencies. Nobody seems to flip their lid over this kind of activity. All perfectly legal and accepted.

With latency arbitrage, an algorithmic engine again takes in two different feeds, this time a raw data feed from somewhere like the NYSE and a consolidated feed representing the National Best Bid and Offer (NBBO), and buys and sells differences between the two. Instead of inefficiencies across markets, it finds inefficiencies across time. Mechanically, it is just two available sources of information exposing an inefficiency. All perfectly legal, but not quite as accepted.

To me this is similar to saying that someone who takes advantage of an obscure IRS tax credit is somehow cheating the system. The tax code is there to define the rules, and it’s our job to play within them. Similarly, when hedge fund strategists sit down to research profit opportunities, they do so using rules laid out by the SEC. If latency arbitrage is deemed uncool and currency arbitrage is okay, the regulating bodies will change the rules. And the algorithmic hedge funds will change their practices to find low risk money elsewhere. Until then, the outrage over latency arbitrage is little more than populist grandstanding.