OpenGamma Blog

Some Insight into the OpenGamma Recruitment Process

2012-05-10T00:00:00

When OpenGamma was started by Kirk Wylie, Elaine McLeod and me, our approach to recruitment was primarily driven by our own recruitment experiences. We were all recruited into the expectations of the Hedge Fund environment: operating in a high-pressure environment with an extremely high expected level of professionalism and technical excellence. Later on, we were also involved in interviewing many other candidates for roles operating alongside us.

In this post, I'm going to describe the evolution of the recruitment process at OpenGamma and what it looks like today, specifically focusing on Platform Development roles.

How our recruitment process has evolved over time

The first interviews at OpenGamma were based upon the previous ‘Hedge Fund’ model. To some degree, with hindsight, our interviews tended to be rather too close to a process of attrition. In a desire to both put the candidate under pressure to see how they operated in a high-stress situation, and to maximise the use of a candidate's time (taking time off work can be difficult to repeat for a candidate), we tended to do ‘marathon’ interviews of 5-6 hours, with staff members each taking an hour-long slot. We also early on had a policy of every team member interviewing new candidates.

This process was not sustainable. We first introduced phone interviews, so we didn’t waste our time and the time of candidates who we didn’t believe were a good fit. After the team had expanded to a certain point, we could also no longer sustain the policy of every team member interviewing new candidates. There were a couple of reasons for this. Firstly, the sheer number of people involved became unmanageable. We began to double up staff in interviews, with typically one person leading the interview and another more passive. This meant we could get more people in the process. But every time you add a person to the process, it becomes harder to accept a candidate (an important part of our interview methodology is that every interviewer can give a “No” to any candidate; we always ask the interviewer to justify this decision, but at the end of the day the people who are interviewing will have to work with the candidate and they need to feel comfortable with working with him or her for a long time). Even a really excellent candidate has a small probability of being rejected by a seemingly rational interviewer. When you multiply these probabilities by 6, 8 or even 10, you increase the risk of rejecting perfectly reasonable candidates.

This did in fact happen on at least one occasion. After rejecting one candidate, they kept coming to mind and no one could give a good reason why they were rejected - so six months later, we offered them the job (they’ve now been with us for over a year).

Lastly, we stopped the really long interviews in most cases. Part of our reason for trying to perform interviews in one day was because we had ourselves been irritated by the two, three or even more interviews typically required in the financial industry’s recruitment process. It can be tough for a candidate to get that much time off work without arousing suspicion. We therefore still give some candidates the option of packing the interviews together, particularly if they have travelled from further afield to attend.

We also weren’t sure that screening out people who weren’t capable of operating in a super high-pressure environment was necessarily always a great idea. OpenGamma is not a hedge fund after all, and we pride ourselves on giving employees a pleasant working environment, with flexible working hours and so on. So while we do need people who can think fast on their feet and work to fix customer problems quickly, we also have room for those who are excellent, but perhaps look to longer term issues like system design and evolution rather than quick fixes.

...And what it looks like today

So, after that process of evolution, we’ve fallen into a predictable pattern:

CVs come to us via either:
- One of the few recruitment agents on our approved list (which is very short, and we are not looking to add to this list at the moment).
- Directly emailed to jobs at opengamma dot com.
- Events such as Silicon Milkroundabout.
- Personal introductions and recommendations.
The appropriate head of the team will review CVs to provide a short list. Often this is sporadic, and based on a specific requirement to hire in the short-to-medium term. In the meantime, we aim to let prospective candidates know the situation, although sometimes the volume of applications (particularly where they are wholly inappropriate “scatter gun”) means we can’t - apologies to anyone reading this who didn’t get a response. At this point we lose around 90-95% of candidates.
This shortlist is then invited to a telephone interview. This is generally a short (20-60 minute) phone call with the candidate, just to get an idea of their personality, to relate more information about the role on offer and to judge their expectations. This is often surprisingly light on technical or job-specific questions. At this point we probably lose around 20-50% of the remaining candidates.
We then ask the remaining candidates to do a code sample. This takes the form of a standard programming task (typically in Java), and we evaluate the candidate on the basis of their interpretation. We get quite a wide range of ability shown at this stage, but generally the standard is very high. Over the years we’ve come to take certain characteristics of this test as good pointers of a candidate’s way of thinking and level of ability. Even more interestingly, we provide absolutely no guidance on what we’re looking for to the candidate or (even more importantly), to the recruiter (if appropriate). Even with that lack of guidance, we find that successful candidates all tend to include the same things we’d expect, without us having to tell them. This gives us a feel for how candidates are able to code in a self-directed manner without the stress of the whiteboard-programming interview.
We then ask the candidate in for a first round of interviews. This typically lasts around two hours, with one or two interviewers in each hour-long slot. They typically focus on domain-specific problem solving (“How would you do X?”). For development that is generally algorithmic although may involve knowledge about specific hardware or software techniques. We also often ask very difficult questions on the edge of the candidate's area of expertise to see how they react to solving problems they’re not necessarily comfortable with. One thing we’re always careful to point out is that given the strength of the technical team here, for anything on your CV, chances are there will be an OpenGamma employee who’s at world-expert level in that thing. We always advise people to either exclude things they’re not comfortable with, or at least indicate their relative skill level (so that we don’t assume someone who has done a little bit of SQL can handle a technical interview with Kirk, who started his career doing nothing but database internals development).
After the interviews, the head of department will ask each interviewer what they thought separately. Discussion between interviewers is discouraged before this point as it tends to introduce a sort of ‘group-think’ confirmation bias where people who privately would endorse a candidate would be influenced by other group members to change their mind.
We then decide whether to bring the candidate in for a second round of interviews. This typically includes another one to two hours of technical interviews and a final interview with someone senior such as myself. At this point Kirk always interviews every candidate to make sure people have time to ask questions about the business side of the equation, as well as to make sure that all developers will be a good fit. After all, we’re not just choosing candidates, they’re choosing us!
At this point we decide whether to continue and extend an offer to the candidate or to pass.

In conclusion, I think the changes we have made over the lifetime of the company have improved the process of finding great people to join us at OpenGamma. It now takes us less time to find suitable candidates, puts less time pressure on candidates, and wastes less time of people who aren’t a good fit for us. Be under no illusions, we still have very high standards, and we’ve been told that we’re the hardest interviewing process in London at the moment. But if you think you can make the grade, and relish the opportunity to work with some of the top technologists in the UK, apply today!

We’ll be attending Silicon Milkroundabout in London at the end of the month. It’s a great opportunity to meet a number of exciting start-ups, all in recruitment mode. If you are interested in learning more about working at OpenGamma, find us on stand 48.

OpenGamma in a Minute - Animated

2012-04-26T00:00:00

Because OpenGamma's approach is quite different from a typical vendor in the financial technology space, we've sometimes found it challenging to get our philosophy across to the wider audience. If a picture is worth a thousand words, we thought a video must be even better - and decided to put together a quick animation explaining our approach to modern risk and trading analytics.

So, without further ado, may I present:

OpenGamma OpenHouse - May 2012

2012-04-23T00:00:00

April has been a hectic month at the OpenGamma HQ thanks to our 1.0 Platform release, but now that it's been released, we thought it's about time we organise another OpenGamma OpenHouse. We'll be opening our doors, serving up some food and drinks, and firing up the demo servers to showcase the new OpenGamma Platform on Thursday, May 3rd, 2012.

Now that the 1.0 release is finally out, there’s lots of exciting stuff to demo. If you’d like to see our R module, the Excel integration, some of our other non-Open Source GUI tools, or the open source Bloomberg module in action, this is your chance to have a chat with the developers behind those features (and enjoy a few beers while you’re at it). Even better, we've got more stuff that isn't in 1.0, so you get a sneak peak at what's likely to be in 1.1 (and beyond!).

Our previous OpenHouse was in late 2010, so there have been a lot of changes in the company since then: we’ve moved offices and doubled the size of the team here in London. (So please don't show up at the old warehouse above the Kirkaldy Test Museum; we've moved to Park Street next to the Tate Modern).

So come along to meet the team, see the Platform in action, and ask any questions you may have.

You can just come by our offices in Southwark after work on Thursday, 3rd May, but we'd like you to register on Eventbrite (Password: ValueAtRisk) so that we have a rough idea of who's coming, how much food to order, and, perhaps most importantly, how many beers to have ice cold!

We hope to see as many of you from the London area as possible there! If you can't make it next week, we'll be organising another OpenHouse some time in June (follow us on Twitter to be among the first to hear about it).

Countdown to R/Finance

2012-04-17T00:00:00

The fourth annual R/Finance conference will take place in the Windy City next month. Aimed at users of R, the open source programming language for statistical computation and graphics, the event focuses on using R as a primary tool for financial risk management, analysis and trading.

(To learn more about R and its exponential growth in the past few years, check out this excellent white paper by Revolution Analytics.)

Two members of the OpenGamma team attended last year’s conference, and talked about what early adopters of the OpenGamma Platform wanted us to do with our R Integration Module. The response was so positive that we pushed forward on the implementation (now available as part of the 1.0 release), and decided to join as one of the sponsors of the R/Finance conference. We’ll be showcasing the R integration on the first day of the conference. Additionally, Ana Nelson, a documentation consultant for OpenGamma, will be speaking on financial reporting and documentation using R and Dexy on the second day.

Working with some of the foremost quants in the industry (like Andrew Rennie, the former global head of analytics at Merrill Lynch), we’ve learned that some of the advanced portfolio analytics required by modern quantitative trading strategies are ideally suited to the R programming environment. Using the OpenGamma R Integration Module analysts can create custom stresses and scenarios, using R’s rich ability to perform statistical perturbations on market data and security terms, all fully integrated with the rest of the OpenGamma Platform. Even better, these calculations all happen on the same server-side infrastructure used by the rest of your installation: keeping your workstation free for the work that has to be done on it.

We believe in supporting the tools that quantitative finance practitioners want to use. That explains our deep commitment to Excel, now R, and forthcoming integrations such as Python and MATLAB. We don’t believe that you should have to use our GUIs to do your job.

You’ll no doubt find many of us hanging out in the conference lobby and demoing the Platform - come and say hi.

Converting Javadocs from LaTeXlet to MathJax: A git-sed-vim Case Study

2012-04-12T00:00:00

Financial analytics are all about the math, and documenting financial analytics means making the math pretty, which on the internet can be an ugly thing to do. LaTeX has been the standard way to make math pretty in print since the 1980s, but making math pretty online has always been contentious and problematic.

Projects like MathML have tried to help, but haven't really taken off because in order to display:

a x^{2} + b x + c

You need to type:

<math xmlns="http://www.w3.org/1998/Math/MathML">
    <mrow>
        <mi>a</mi>
        <mo>&#x2062;<!-- &InvisibleTimes; --></mo>
        <msup>
            <mi>x</mi>
            <mn>2</mn>
        </msup>
        <mo>+</mo>
        <mi>b</mi>
        <mo>&#x2062;<!-- &InvisibleTimes; --></mo>
        <mi>x</mi>
        <mo>+</mo>
        <mi>c</mi>
    </mrow>
</math>

Example from MathML wikipedia page. By contrast the LaTeX equivalent for this expression is:

$ax^2+bx+c$

Apart from the verbosity of the MathML markup, until very recently it wasn't safe to assume that most browsers would fully support HTML5, which includes MathML support.

So, when OpenGamma began to look for a way to document the mathematics they were implementing, they looked for a LaTeX-based option and found it in LaTeXlet, a custom Taglet for javadocs. This taglet looks for specially formatted @latex tags in Javadoc comments, renders them as LaTeX, and inserts the rendered math into the generated Javadocs as images.
The use of images is problematic for several reasons: they take extra time and processing power during the Javadoc build phase, latex must be installed on each machine that needs to generate Javadocs (this is not a trivial requirement), the generated images can be on the ugly side and, more importantly, they are very unaccessable. LaTeXlet further requires that you add an extra slash to each of the latex commands that you call. Here is an example of a Javadoc comment marked up for LaTeXlet:

/**
 * Calculates the historical covariance of two return series. The covariance is given by:
 * {@latex.ilb %preamble{\\usepackage{amsmath}}
 * \\begin{eqnarray*}
 * \\frac{1}{n(n-1)}\\sum\\limits_{i=1}^n (x_i - \\overline{x})(y_i - \\overline{y})
 * \\end{eqnarray*}}
 * where {@latex.inline $x$} is the first return series, {@latex.inline $y$} is the
 * second return series and {@latex.inline $n$} is the number of data points.
 */
public class HistoricalCovarianceCalculator extends CovarianceCalculator {}

There are a lot of tags in there. And while LaTeXlet will interpret "\\frac" as "\frac", any other LaTeX compiler will be unable to make sense of this. It's also a huge amount of extra typing to apply LaTeX-style formatting to the individual variables, i.e. {@latex.inline $x$}. And because they are replaced by images, it renders the whole sentence unaccessible.

While LaTeXlet had the advantage of being slightly less cumbersome than MathML, and allowed quants already familiar with LaTeX to use a tool they were proficient in, it was far from ideal, but was probably the best option available at the time. Now, happily, there is an alternative which isn't a least-bad option, and which is making it virtually painless to put beautiful math on the web. The MathJax javascript library allows you to insert plain old LaTeX in your HTML pages and it will render this LaTeX unobtrusively in a beautiful, accessible and standards-compliant way. Because you are writing standard LaTeX without any extra characters, you can also easily use this LaTeX in other PDF documentation, or paste it into tools like mathb.in to get a quick preview.

Cleaning up the documentation was one of the goals for the 1.0 release of the OpenGamma Platform. After we verified that we could apply MathJax to our generated Javadocs, I then had to solve the problem of converting almost 200 existing files with LaTeXlet-style markup into the simpler format we would use going forward. I wanted to find a way to automate the process, not only for the obvious drudgery it would otherwise involve, but because the probability of making mistakes in manually editing such content was very high.

I started working on a sed script which would convert the double slashes to single slashes, get rid of the unnecessary @latex tags and their associated brackets and replace them with math mode declarations like $$:

s/\s*%preamble{.*}}//g
s/{@latex.inline \(\$[^$]*\$\)}/\1/g
s/{@latex.ilb/$$/
s/end{eqnarray\*}[\s\*]*}\s*/end{eqnarray*}\n * $$/
s/end{align\*}[\s\*]*}\s*/end{align*}\n * $$/
s/\\\(\\\w\)/\1/g

This will give output like:

/**
 * Calculates the historical covariance of two return series. The covariance is given by:
 * $$
 * \begin{eqnarray*}
 * \frac{1}{n(n-1)}\sum\limits_{i=1}^n (x_i - \overline{x})(y_i - \overline{y})
 * \end{eqnarray*}
 * $$
 * where $x$ is the first return series, $y$ is the
 * second return series and $n$ is the number of data points.
 */
public class HistoricalCovarianceCalculator extends CovarianceCalculator {}

I developed this script by first committing all my other changes so I was starting with a clean git repository, and then iterating the sed script by running:

cd /mnt/work/OG-Platform/projects/
git checkout HEAD .
find . -name "*.java" -exec sed -f /mnt/work/blog/remove-latexlet.sed -i {} \;
git diff . | less

I would review the diff and then modify the sed script to improve its recognition. I repeated this several times until I was happy enough. This script isn't perfect, it sometimes misses the closing } of the @latex.il environments, and it also was a little overzealous in converting all instances of \\ to \ rather than just those in math mode chunks.

Here's another Javadoc source example:

/**
 * A simple chooser option gives the holder the right to choose whether the
 * option is to be a standard call or put (both with the same expiry) after a
 * certain time. The exercise style of the option, once the choice has been
 * made, is European.
 * <p>
 * The payoff of this option is:
 * {@latex.ilb %preamble{\\usepackage{amsmath}}
 * \\begin{align*}
 * \\mathrm{payoff} = \\max(c_{BSM}(S, K, T_2), p_{BSM}(S, K, T_2))
 * \\end{align*}
 * }
 * where {@latex.inline $c_{BSM}$} is the general Black-Scholes Merton call
 * price, {@latex.inline $c_{BSM}$} is the general Black-Scholes Merton put
 * price (see {@link BlackScholesMertonModel}), {@latex.inline $K$} is the
 * strike, {@latex.inline $S$} is the spot and {@latex.inline $T_2$}
 * is the time to expiry of the underlying option.
 */

public class SimpleChooserOptionDefinition extends OptionDefinition {}

And here it is after running 'sed', you can see it misses the closing } on the line after \end{align*}:

/**
 * A simple chooser option gives the holder the right to choose whether the
 * option is to be a standard call or put (both with the same expiry) after a
 * certain time. The exercise style of the option, once the choice has been
 * made, is European.
 * <p>
 * The payoff of this option is:
 * $$
 * \begin{align*}
 * \mathrm{payoff} = \max(c_{BSM}(S, K, T_2), p_{BSM}(S, K, T_2))
 * \end{align*}
 * }
 * where $c_{BSM}$ is the general Black-Scholes Merton call
 * price, $c_{BSM}$ is the general Black-Scholes Merton put
 * price (see {@link BlackScholesMertonModel}), $K$ is the
 * strike, $S$ is the spot and $T_2$
 * is the time to expiry of the underlying option.
 */

public class SimpleChooserOptionDefinition extends OptionDefinition {}

However, I knew I wanted to manually check each file before committing the changes anyway, so I would be able to fix these issues and tidy up any other little niggles that I spotted. It wasn't worth spending more time on the .sed script, instead I needed to find out how I was going to identify all of the changed files and make sure I checked each of them.

I use vim as my text editor. I hoped that a solution might be found using some sort of plugin for vim which would let me systematically work through the list of modified files generated by running "git status". Happily, I found exactly such a tool in fugitive.vim. After I installed it using pathogen.vim (by the same author), and watched the first and second vimcasts about fugitive, I quickly had a very efficient workflow going.

I would run :Gstatus to open a new buffer listing all the modified files in the repository - which were the files that had been changed by running the final version of the sed script. I type return while over the first file in the "Changes not staged for commit:" list, which would open that file for editing. Then I would run :Gdiff on that file which showed me the current changes using vimdiff. Then I could tweak the Javadoc in the working copy, fixing anything that the sed script missed and making any other small edits. When I was finished I would save my changes, return to the status window and type a -, which would add the file I had just been editing to the "Changes to be committed:" list. At this point the file would disappear from the "Changes not staged.." list and the cursor would already be positioned over the next file, so I just had to type return to start editing it, and the process begins again. Here's a picture of what it looked like:

With this approach it was very fast and straightforward for me to work my way through the list of modified files, and I could be confident that I had checked every file before adding it to the list for the next commit. I could also exit vim at any time and easily pick up where I left off later by issuing another :Gstatus command. I did this a few times to make sure that I didn't have any files lurking with unsaved changes, and to generate the Javadocs from outside of vim.

The result is a much cleaner javadoc build, several dependencies removed from the Ivy repository and Ant configuration files, happier developers who can much more easily update and input mathematical descriptions, and beautifully rendered LaTeX that can be re-used in multiple contexts, both for print and web.

Mathematical equations show up in the OpenGamma codebase to describe statistical distributions like the Generalized Pareto Distribution, functions such as those used to add, subtract, multiply and divide curves, calculators like the JensenAlphaCalculator and SharpeRatioCalculator, and of course models such as the BlackScholesMertonModel and the BjerksundStenslandModel.

Conventions, Bloody Conventions!

2012-04-05T00:00:00

In our experience, as Quantitative Analysts, Risk Managers, Back-Office Officers or Traders, we've all one day or another tried to look up a small detail about a very familiar instrument without finding it. Does Euribor use the end-of-month rule? What is the standard payment frequency for a three-year AUD swap? What is the last trading date of a mid-curve option on Liffe? Those questions may sound familiar. Often, the only way to find answers is to ask your colleagues, search the internet or call a counterpart. Everybody is supposed to know about them, but, to the best of my knowledge, and to my longstanding despair, they are not available in one unique, easily accessible place.

After ten years of looking for those details, trying to remember them only to forget them the next week and look for them again, I decided not to trust my memory anymore and to write them down. The writing did not happen overnight, not even in a fortnight. The details were collected over almost one year while we were implementing the different instruments in our libraries and creating test portfolios.

Now that I have them in writing, when a colleague or a friend asks me for those details, I just have to find the right section and copy and paste it.

Introducing the Interest Rate Instruments and Market Conventions Guide

At OpenGamma, we have decided to go one step further: shape this information in the form of a booklet and make it available to everyone in the industry. Being an Open Source company, we have the tradition to distribute widely what we do when we believe it could be useful for others.

Nowhere in the document do I describe pricing or valuation mechanisms, even for the simplest instruments. The link to valuation is that any valuation technique for any instrument presented should include all the relevant instrument features. Most of the standard books and articles smooth the roughness of real life. Settlement lags are nowhere to be seen; day counts and business day conventions are supposed to appear magically, when they are mentioned at all. We all know that nothing appears magically and that there is no such thing as a free lunch. I do not offer you any of those free lunches, but hopefully the document can help you find the salt and pepper to season your own lunch.

The goal of the document is to present conventions and market standards for the most common interest rate instruments. I have done my best to collect the information and check it. Obviously the document is not perfect and we plan to add, complement, or correct it when necessary. Do not hesitate to suggest corrections and additions.

The market standards are relative, and they evolve. For the same instrument, two groups of people may use different conventions. This is the case with USD swaps: some use an annual money market basis on the fixed leg, and others semi-annual bond basis. The standards evolve; this is the case with swaptions for which the standard changed from an up-front premium to a forward premium in September 2010.

The document is certainly not intended to be read from start to end like fiction. If quantitative finance is to be compared to a novel, this booklet would be the introduction of the main characters. It is a reference document and I expect users to read at most one chapter at a time, and more often one section or even one single line. This is also the way it was written, adding lines, currencies, and instruments when they were required.

You can download the booklet here. It is published under a Creative Commons license (CC BY 3.0), so you are free to use it in any form and redistribute it. However, we do ask that you indicate that the source is the OpenGamma Interest Rate Instruments and Market Conventions Guide. If you want to pass it to someone, we'd suggest that you point to the original document so one always has the latest version (and the credit is given where due). The current version is named 1.0, just like the version of the OpenGamma Platform just released on 2nd April.

The conventions described in this booklet are naturally already implemented in the OpenGamma Platform and our Analytics Library. If you haven't already adopted it, we highly recommend downloading the fresh new Platform available since Monday!

The devil is in the details.

OpenGamma Platform 1.0 Is Here - Pop the Champagne!

2012-04-02T00:00:00

Today OpenGamma has released version 1.0 of our flagship technology stack, the OpenGamma Platform. While Jim, our Head of Platform Development, has details on the technical side, I wanted to talk about where this takes us as a company and ecosystem.

When we set out on the OpenGamma journey, our mission was simple: to make the same standard of tools available to the most sophisticated market participants available to everyone. In a world filled with black boxes, we wanted to bring radical transparency. In a market where way too many firms keep rebuilding the same systems, we wanted to allow developers to focus on what’s proprietary about their firms.

It's taken us 2.5 years, over 25 developer years, 11,000 test cases and 750,000 lines of code to get here: the world's first production-grade Open Source system for trading and risk analytics and data management. From a modern, modular analytics and pricing library, to a near-real-time streaming calculation engine, to data management, to client tools, we know that the OpenGamma Platform contains everything you need as the basis for custom installations.

What took us so long?

We released 0.9 when we thought that the code was feature-rich enough that it showed what we were capable of, and was an excellent starting point for evaluation efforts. We also said that we needed to wait to officially call something as 1.0 until we were confident that the system would support your use 24/7/365.

Since then we’ve had thousands of downloads by people evaluating and using the system, providing us with extensive feedback. We're also in use in anger every day by some of the largest and most sophisticated hedge funds in the world.

How do we know we’re ready? They told us we’re ready.

1.0 is battle tested and ready for use in production today.

What's next?

Unlike other vendors, we're not shy about telling you what we're working on. We've got our roadmap on our website so that you know where we’re going and roughly when we'll get there.

However, our roadmap can really be summed up in a few key points:

While we already have extensive asset class coverage (PDF summary / full analytics documentation), we're going to improve it. You've told us that while you can already incorporate 3rd party models easily, you don’t want to have to do it (or go to a traditional black-box analytics library vendor).
While many of you are already comfortable and familiar with building your own end-user tools, you want us to do more out of the box. So we'll be enhancing the end-user tools available to both Open Source users and commercial customers and continue to push our already industry-leading integrations with systems like Excel and R.
While we already have a pretty impressive set of data integrations, we want to make sure you can download the system and have live risk on your portfolios using your existing data as fast as possible. Unlike legacy vendors, we don't make money by dragging on installations as long as possible to sell consulting services: we want you up, running, and in production as fast as possible.

But more than just categories of features, our roadmap is simple: Do whatever it takes to make our customers successful.

Whether you're a developer at a hedge fund, an academic quant at a university, or a risk manager at a bank, we want to hear from you. Download the code, read the docs, or contact us for a conversation and demonstration.

If you're tired of building and maintaining way more code than you need to, or tired of the way legacy vendors have been treating you, you’ve found the right people.

OpenGamma Platform 1.0 Released

2012-04-02T00:00:00

The 1.0 version of the OpenGamma Platform is finally out. It’s been in the making for longer than anticipated, but we believe it was essential to wait until we have all the main components ready to ship.

In this blog post, I’ll briefly go over the most significant changes. If you can’t wait to see it for yourself, head over to our developers’ site to download the Platform.

Bloomberg Integration

After Bloomberg’s recent announcement to open source its API, we’ve been able to include the Bloomberg module in our 1.0 release. This will allow you to load automatic reference data for exchange-traded securities, as well as load and update time series. Our integration module also now Open Sources our OpenGamma Live Data Adapter for pulling real-time streaming data into your OpenGamma environment from your Terminal, SAPI, or Managed B-Pipe instance. You can now download the Platform, hook it up to your existing Bloomberg infrastructure, and be doing live trading and risk analytics with real world data in a matter of minutes!

This means that when evaluating, you have two basic options. You can choose the classic “Examples” package, which works with entirely mock data, just as in 0.9. However, if you have access to a Bloomberg Terminal, SAPI, or Managed B-PIPE instance, you can use the “BloombergExamples” package to work with real data and get your own portfolios up and running in the system in a matter of minutes.

New HTML5 Web GUI

Our HTML5 Web GUI continues to improve. While we’re busy rewriting our Web Analytics Viewer from scratch (to make it easier to include in your custom portals and applications), there have been a number of improvements to the main Open Source Platform GUI:

The Web Analytics Viewer now supports Market Data Snapshots as well as different Live Data configurations as data sources and fallback to Historical Time Series for market data;
It also supports dynamic re-aggregation of your portfolio on the fly - without reconfiguring your View Definition;
Many of our Data Master viewers (like Portfolios, Positions, and Securities) are far better integrated, and can pull in key historical time series directly into the viewer for Securities to make it quicker to do common tasks.

Finally, our whole GUI has been retrofitted with our new “Push REST” capabilities to notify all clients of data changes whenever they happen anywhere in your environment: reloading to get changes other people on a desk have made is a thing of the past.

Database Masters

We continue to improve our Database Masters. Aside from numerous performance improvements throughout (particularly our Historical Time Series database schema and Master code), we’ve completely rewritten our Batch Risk database and masters to allow far more types of data to be stored and queried with the most complex of View Definitions.

Our data masters have also been enhanced to allow runtime tagging of Securities, Trades, Positions, and Portfolios. This allows you to put in any custom attributes required on any of these data types, and incorporate that with other new elements like our expression language for dynamic portfolio filtering and aggregation exactly as the “native” data fields.

Asset Classes and Analytics

The OpenGamma Platform continues to extend its support for new asset classes and analytical methods.

Perhaps the most important change since 0.9.0 has been the amount of time that our Quantitative Development team has spent on making sure that we have a single source of all major market conventions for the G8 (and nearly the whole of the G20). Not only have we incorporated this into the Platform so that we have accurate cashflow determinations for assets in these currencies, but we have a booklet (forthcoming) that has all this information in one place as a guide you can use for your own development.

Notable new assets include:

Caps/Floors (including CMS)
Inflation products
Additional types of IR Swaps
OIS
Equity Variance Swaps
FX Futures
Digital FX Options

View the full list of asset classes currently covered (PDF)

We’ve also extended the types of analytical methods available, in particular enhancing the number of different analytical methods available for IR Swaptions and applying our Local Volatility and SVI models to multiple asset classes.

But no matter how good our analytics library or data management may be, there will always be times when you won’t want to use it, but have a system that can generate sensitivities you want to aggregate with the rest of your portfolio (common examples are credit derivatives, ABS, and RMBS). To handle that, we now have full support for External Sensitivities.

You can now create an External Sensitivities Security, assign the appropriate risk factors to it, and use the following features:

Yield curve sensitivity mapping
Separate yield curve/credit/all sensitivities buckets
DV01, CS01, Historical VaR

R Integration Module

We’ve found a large number of quants and risk managers using the R statistical programming environment to drive deep and custom statistical analysis of their portfolios. In keeping with our dedication to supporting the tools that end-users actually want to use, we have taken our industry-leading Excel Integration Module, stripped away the parts that aren’t Excel specific, created the OpenGamma Language Integration package, and used that to provide the same level of extremely deep, useful integration with the R statistical programming environment.

Everything that you would expect by now from a tight OpenGamma integration is there: you can pull in all types of data available in your OpenGamma Platform instance, and they all appear as native R objects; you can drive shocks and stress tests and historical simulations, including perturbing market data at either the individual ticker or tensor level. You can even create custom trades and portfolios from R to do what-if scenarios.

We think this is extremely powerful, and we’re thrilled that this is in the Open Source OpenGamma Platform! (also, we’re a sponsor of the R/Finance Conference in Chicago in May; if you’re in the area come on by and say hi!).

04/04/2012 Update: The R Installer is now available on the downloads page.

Other Updates

Perhaps the most immediate change developers will notice is the improved build system – it’s actually now a single-step process. We’re confident that you’ll find deployment a whole lot easier. We’ve also included better ant target names.

For those of you evaluating the platform independently, we’ve included sample data for new asset classes, and improved data import/export: there is a standard import format for security/portfolio data, as well as a framework for custom importers.

Finally, we’ve completely overhauled the configuration system. You’ll find that deployment and maintenance have been made significantly user-friendlier, using the distributed component management system.

As ever, we welcome your feedback; please add your comment below. For any technical questions, we recommend contacting our technical team through our forums. A special thank you to those of you already evaluating the Platform who have posted questions, comments and suggestions on the forums over the past months.

Download OpenGamma Platform 1.0

Request a demo

Browse documentation

Partial Application in JavaScript Revisited

2012-03-22T00:00:00

A few years ago, John Resig wrote a blog post about partial application in JavaScript and proposed extending the Function prototype with a new method: partial. This technique is quite powerful, as it allows for much more concise and expressive code. We use partial application extensively in our UI codebase, particularly in our wrapper for OpenGamma's REST API, where it has saved us several hundred lines of code. Fewer lines of code often means fewer bugs. And this particular technique cuts down on repeated code and copy/paste errors.

The technique takes advantage of the functional nature of JavaScript (treating functions as first-class objects) and allows information to be encapsulated inside a function. It returns a function with some of its arguments pre-populated. It uses JavaScript's prototypal inheritance to extend all instances of functions:

The problem with this is that the args array gets cached. Here is the example use case from the original blog post:

But suppose that function is used again. The results are unexpected:

What's happened is that the first function has been cached inside the args array and all subsequent calls to delay are being pre-populated with that first function. Additionally, JavaScript Function instances can have arbitrary properties attached to them, but this particular implementation of Function.prototype.partial will lose the references to any additional properties that may have been added to a Function instance. Even though extending Function instances with custom properties is fairly uncommon, it is nonetheless a feature of the language that partial should retain. If a function is being pre-populated, the resultant function should contain references to all the original function's properties. So, with those two concerns in mind, here's our implementation of partial application:

If we run the same test case, we get better results this time:

Extending native objects in JavaScript is a controversial subject. The biggest philosophical difference between libraries like jQuery and Prototype is that jQuery does not extend native objects. We did not choose to extend the Function prototype lightly; there are a very small number of cases where we have elected to extend native objects. But in this particular case, the gains in expressiveness and the ability to write less (and more concise) code are compelling.

Managing a Large Mixed Public/Private Java Project Using Git

2012-03-13T00:00:00

Having a mix of public and private source code repositories poses an interesting challenge to any organisation. How do you achieve the necessary combination of openness (which is assumed by Git at an architectural level), and the necessary level of confidentiality for clients and commercial components?

In this post, I wanted to shed some light on how we’ve managed to address that challenge and come up with something that works pretty well for us at OpenGamma.

Why we chose Git in the first place

When we first started OpenGamma, the version control system that I was most comfortable with was Perforce. While Perforce is a great system, it didn’t really seem appropriate for an Open Source project. Although the Perforce client software is free to download, we felt that the Open Source community in general frowned upon using proprietary software for version control. We did consider Subversion, but given that the world seemed to be moving to distributed version control, we decided to take the plunge with Git. We immediately decided to use GitHub as a host for our repositories as part of our general approach of using hosted services where possible.

While OpenGamma is primarily an Open Source platform, we do have some proprietary components only available to commercial customers. In addition, our support methodology allows us to share code directly with customers to provide hands-on, proactive support. Unfortunately Git is really not designed to support a mixed public/private model. What we really want is to be able to wall off access to certain parts of a repository, but when you think about it more deeply, it’s actually pretty understandable why Git doesn’t let you do this. For example, if you commit across public and private code, is it acceptable to even expose the file names of the private files? Probably not. What about history? What if something is moved from private to public or vice versa? Do you want all these cross-linking chains of commits? Probably not.

Because OpenGamma is a heavily componentized system, we’ve always operated on a quite fine-grained project basis, and have formed a moderately complex tree of dependencies between modules (primarily managed by Eclipse and Ivy). When we first started, the easiest way to manage the public/private split seemed to be to set up a separate git repository for each project. We also had a top level repository, OG-Platform, that contained the common top-level build files and configuration. The idea was that you cloned from OG-Platform and then cloned each sub-project into the projects directory underneath it.

That worked relatively well until we got to about 10 projects, at which point we found that it took so long to commit changes across all projects that someone else had committed in the meantime and you had to merge again. We also found it was very easy to forget to sync up some of the projects and you’d get strange errors when a rarely used project was slightly out of date.

To address these problems we did two things:

We merged all the Open Source projects (excluding Fudge, the self-describing binary message encoding system we are the primary sponsors of) into a single repository, OG-Platform. OG-Platform would now contain all the Open Source projects in the projects directory straight away after a clone. This hugely reduced the number of repositories we needed to manage.
We added some tools to perform operations across repositories. The ant task ‘clone-or-pull’ does either a clone, if the repository is not present, or a pull if it is. We later added ‘ant pull’ to do just the equivalent of ‘git pull origin master’ on every repository, and ‘ant status’ to do cross-project status reportings. Lastly we added a bash script ‘gitstats.sh’, which (parameter is the root directory, e.g. ‘.’) reports the status in a more compact tabular form.

Putting it all together for a public release

We still had a problem though: how to provide different sets of repositories to different users? For example, a user who was paying for our Excel integration module might be able to access the code for that, but it isn’t part of our Open Source release. One approach would have been to just attempt to clone [or pull] each of the git repositories we had, and rely on the permissions to fail if it wasn’t available, but this felt ugly and could potentially lead to information leakage about our private client projects.

To get around this we came up with a system that downloaded a small set of ant build and properties files from a web server, using HTTP authentication provided during the build. Open Source users would use one user name and password (the defaults offered), but commercial customers could use other user name and passwords to get customized build setups. This is where the ‘ant init’ task comes from. It allows the user name and password to be set either by the user as an environment variable, or specified on the command line via an interactive prompt.

What this means is that we can now provide a completely customized set of projects to each customer based purely on a user name and password combination. And as long as we’ve set up their permissions in GitHub, they can clone the code to wherever they wanted.

A slight downside, however, was that it added a step to an already fairly complicated build process. To address this, we’re changing the behaviour for 1.0 to default to the Open Source user name and password unless one is explicitly set up. We also made the clone-or-pull part of a single-step build process, meaning there’s truly one command after the initial clone to build a binary installation. Lastly, we’re changing Fudge, which is now pretty stable, to be an external dependency. This means that out of the box, the Open Source release only requires a single clone rather than three.

What we’ve come up with seems to satisfy all our immediate requirements. It’s not perfect, and we considered many other approaches (e.g. sub-modules) before settling on this. With the right attention to detail in smoothing the path for Open Source users while allowing commercial customers access to the extra code they’re paying for, it seems to be a successful approach for us.

Of course, there’s always more than one way to skin the cat. Have you faced similar issues in your project? How did you solve them? Share your experiences in the comments below.

Open Source - And the Value of Becoming a Client

2012-02-15T00:00:00

As an Open Source business, we often get asked what our business model is (or, more directly: How do we make money?). I like that question because it invites us to talk about the relationship with our clients, which is probably the key element of what our commercial clients get and value.

Open Source is not only a better way of developing and delivering software, it also allows for a much deeper relationship between the vendor and the client. The openness in code and architecture encourages openness in other areas too, and creates a more collaborative relationship.

All the firms we work with - banks, hedge funds, inter-dealer brokers, etc. - are fed up with poorly-supported solutions with output from out-dated software that no longer reflects the needs of the people using it. Building in-house is an option, but it's becoming increasingly difficult with tight budgets and hard-to-hire talent. Most good investors realise that they aren't, and shouldn't be, great software houses. So an Open Source solution is a desirable alternative, providing the transparency and openness you get from an in-house system, with the development skills of a dedicated software firm.

No need to go unsupported

But Open Source doesn't mean you are on your own. With an OpenGamma subscription you get high-quality support and maintenance from relevant specialists who know your installation intimately.

We have dedicated staff with deep technical knowledge supporting our clients. With specific insight into your installation, from initial set-up to on-going maintenance, upgrades and expansion, they are at hand to ensure you have a stable and current solution that evolves with the rest of your infrastructure and business activities.

Access to expertise

And we really don't like just giving you a phone number to call and pray someone is there if things go wrong. As part of our own learning experience, we would like you to call us not just when something fails, but also when you simply wish to exchange ideas, or question the way we've implemented a particular feature.

When you call, you get access to the engineers who actually built the system, which is great for both parties. You can talk to someone who knows what they are doing and can explain why we have done what we've done (which of course is transparent), and we learn what our clients are working on and what matters to them.

We believe this dialogue is much more than effective support. It forms an important part of the community - or ecosystem - of users and developers and benefits us all. We learn what's happening in the market and can share that with the community at large, as well as feeding it into the Platform to create a better solution.

As we focus on the technical issues that are common to many, rather than that which is proprietary, we think this sharing of information helps all our users to understand what works and how, saving money and improving quality for all.

The OpenGamma Platform is also designed for integration, with APIs to your other systems. This means we have a lot of experience working with different solutions and usually have a pretty good understanding of the relevant strengths and weaknesses of those we integrate with. We're not tied to other software or hardware vendors, but we are happy to share our views and recommendations with our clients.

Let's talk about quants

Another aspect of being a client is that you have access to our quants. This is a highly regarded team of quant specialists with diverse, but highly relevant backgrounds. They have all come to OpenGamma to be able to work on something that is state-of-the-art and has industry-wide impact. We love to offer them up to our clients who engage with them directly - sometimes clients come by for a "deep dive" chat, and sometimes to sound us out on how to generate the analytics for a certain trade for example.

As with our developers, our quants are not here to replace you, but rather to provide the tools for you to do more and better.

Commercial transparency

The OpenGamma Platform can be installed in parts or enterprise wide. What is built to support large investment banks or huge hedge funds is also available to smaller firms, on a smaller scale. We typically don't charge large upfront fees, preferring instead to spread the cost out according to a clear subscription agreement that we negotiate upfront depending on the complexity of your installation and the level of involvement from us. You don't pay per-user licences or per-incident support fees, and we aim to make the cost both predictable and transparent. We are convinced the OpenGamma ROI is very compelling and would be happy to discuss it with you in more detail.

Finally, having access to the source code frees you from the tyranny of vendor dependencies. If you don't find being an OpenGamma client of value, there is nothing tying you to us; you are free to keep your installation and continue using the Platform! Kind of keeps us on our toes too.

Bloomberg Opens Up Its Market Data API - OpenGamma 1.0 Release Being Prepared

2012-02-02T00:00:00

What a day for Open Source and the financial services industry at large: Bloomberg announced yesterday it is opening up its market data API under the open source MIT license - in stark contrast to its competitors.

The implications for OpenGamma users are huge, and positive. We know many of you evaluating the software already have Bloomberg terminals available, yet due to Bloomberg's previous policy have been forced to rely on mock data. We've provided some of you with the integration module for evaluation purposes under a proprietary license, but we also understand that many of you prefer to self-evaluate the product without having to contact us. Bloomberg's announcement will allow us to change that.

Where's 1.0?

We know you've all been anxiously waiting for the 1.0 release of our Platform. It's actually been broadly stable and feature complete for a while now; we already run it internally 24/7 and have firms using it in anger. The few remaining pieces are mostly related to documentation and examples.

After yesterday's announcement, our goal is to go back and amend our 1.0 plans to include the Bloomberg Integration Module - we'd like to ensure there are evaluation tools ready with the upcoming release. As soon as the module is ready to be released open source - assuming this will happen before the 1.0 release - we'll aim to publish it on our GitHub account, as well as make an announcement on the forums. The module will then allow anyone with a valid Bloomberg terminal or Server API instance to directly access Bloomberg data from within the OpenGamma Platform.

We've been working on and with the OpenGamma Bloomberg Integration Module for several years now, and the difference in evaluation will be pretty extreme. Our goal is for you to be able to download the Platform, connect it to a Bloomberg Terminal running on your desktop, and have real-world analytics and risk in a matter of minutes. We think that's worth waiting a few more weeks for! (Want to make it happen faster? We are hiring.)

We are now working on the final preparations, and will be updating you shortly - stay tuned.

The Sparse Farce

2011-12-21T00:00:00

What Was Said Last

In my last post I briefly outlined some performance gains we were achieving in our tuned dense BLAS library. Evidently this was received with mixed opinion! Tuning Java does seem like an insane idea, and indeed some people pointed this out, but given that these are our constraints:

Code has to be as fast as possible.
Assume the hardware is fixed and tune to it.
Code has to be in Java: no JNI, no hitting Fortran/BLAS/<insert thing we'd normally use here>.

So we don't really have a lot of choice! Once every piece of algorithmic tuning has been taken into account, the main point of performance death is going to be JVM and the fact that the JVM doesn't provide primitives those of us from the scientific computing world are used to, both of which are immutable as selections. However, trying to force their combination to produce something reasonable performance wise (as exhaustively pointed out by the people at LMAX) requires effort in terms of the acknowledgment of the hardware and a trip back to basic computer science.

What's Next

Having been relatively successful with increasing performance in dense BLAS kernels, we turned our attention to the invariably hard sparse BLAS operations: It is our notes and observations on this topic that form the bulk of this article. For those unfamiliar with the notion of a sparse matrix, a sparse matrix is categorised as a matrix with a only a small number of non-zero elements (yes, defining small is part of the problem). Sparse matrices arise a lot in numerical methods and indeed as a result of certain mathematical operations which tend to create some form of diagonal dominance, but this is not strictly so. Additionally, there is no real condition on the pattern of the non-zero elements or what constitutes a sparse structure.

Why?

The reason that we categorise sparse matrices separately is for gains in both performance (we don't need to do operations involving zero entries) and memory foot print (we don't need to store zeros). For example, multiplying a sparse matrix by a vector should proceed as a linear function of the number of non-zero elements of the matrix as opposed to the dimension of the matrix. These performance gains can be carried over into other linear algebra operations but as we'll see later this is easier said than done, particularly with the utter lack of memory manipulation presented by the Java language.

Storage Formats

First, we shall address the manner in which to store the sparse matrix data. Most Java programmers when asked this question reach for a Map implementation, and rightly so, a Map<Pair<row number, column number>, value> would certainly suffice and be very easy to use. The problem, as always with built-in overkill, is that it is probably slow. Notably, the more we play with getting performance from Java, the more we have come to the conclusion that if the Java code doesn't look like a variant of Fortran, it's going to be slow. So although a Map has some advantages, as we'll see later, the bloat is a speed killer.

From the idea of a map, we deduce that to describe the matrix all that is needed is a tuple of <row number, column number, value>. The Coordinate Format matrix (referred to as COO) is perhaps the most basic sparse matrix storage format and it directly reflects this tuple layout by storing the data as three vectors, two int pointers (one to row indices and another to column indices) and a double/float pointer to the corresponding value. Using the following matrix as an example:

    1    3    0    0   12
    0    4    6    8    0
    2    0    0    9   13
    0    5    7   10   14
    0    0    0   11    0

A COO matrix would store the data as:

int[] columnIndex = {0, 1, 4, 1, 2, 3, 0, 3, 4, 1, 2, 3, 4, 3};
int[]    rowIndex = {0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 4};
double[] data     = {1, 3, 12, 4, 6, 8, 2, 9, 13, 5, 7, 10, 14, 11};

This example highlights two important points. First, the memory footprint of the stored data is larger than that of the original data (3*14 vs 5*5). Second, there is exploitable information redundancy in the rowIndex variable.

Our next matrix format is the Compressed Sparse Row storage format (CSR), which takes advantage of this redundancy by solely storing a row pointer from which it is possible to compute an offset into the data. This row pointer is formed by keeping a cumulative count of the number of non-zero data elements encountered up the start of a given row in the matrix. This is probably best described with an example, so continuing with the above matrix:

int[] columnIndex = {0, 1, 4, 1, 2, 3, 0, 3, 4, 1, 2, 3, 4, 3}; // This is the same as for COO
int[] rowPtr = {0, 3, 6, 9, 13, 14}; // This effectively stores a cumulative data count
double[] data = {1, 3, 12, 4, 6, 8, 2, 9, 13, 5, 7, 10, 14, 11}; // This is the same as for COO

Note that the rowPtr variable contains (number of rows + 1) entries. This is to account for the number of elements in the final row and to allow easy computation of the number of elements in a row for the sake of induction variables that index in the column space. For example, to walk through the data in row 2 (assuming 0 based row indexing):

int index = 2;
for (int i = rowPtr[index]; i < rowPtr[index+1]; i++){
  data[i]; // accessing data belonging to a row
}

From this, is it clear that due to the storage of an additional rowPtr index, the same code can be used for all the rows, which considerably simplifies algorithms.

In addition to the CSR there is an equivalent storage format whereby the rows and columns reverse roles such that offsets are computed by column, and rows are used to give the locations within each offset. This storage format type is known as Compressed Sparse Column (CSC). By inspection, it is obvious that CSR and CSC formats will have different total information content for non-symmetric matrices. However, both will have a lower information content than COO (in most real world cases).

So we've visited the basic matrix storage formats for sparse matrices. Unsurprisingly they are also present in Python, GNU Octave and hardened system maths library collections like suitesparse. Now for the interesting bit...

Why Sparsity Is A Pain

Sparse matrices are an invariable pain to deal with. As demonstrated previously, it's possible to actually increase memory usage if the matrix is insufficiently sparse for the storage format chosen. However this isn't the main gripe with these matrices.

The main problem arises from the fact that the code will execute on hardware and this hardware behaves in a certain way. Without going into the ins and outs of processor behaviour and memory hierarchy performance, the full deviousness of tasks involving sparse matrices is hard to explain. However, for now we need to bear in mind two things:

The processors can quickly access data so long as the next element to be used is stored in memory directly after the element presently being accessed.
That operations are best pipelined by making use of a mix of instructions, and that for maximum throughput on the floating point units the data should be coalesced as SIMD operations (as seen in my previous post, SSE isn't that great in Java).

Now, if we take as an example the CSR format matrix "A", containing the information previously presented, and want to do a sparse matrix vector multiply operation (SpMV, as in, result:=A*x), we can use the following code:

Performing a bit of basic analysis on this algorithm with regards to points 1) and 2) above, we can note that the variable "x" is being accessed by the index dereferenced by the column index. This does not bode well for point 1) above. Although values[], result[], rowPtr[] and colIdx[] are being indexed in a stride 1 fashion, "x" is not, and random data access is rather slow (pipeline stall, lack of hardware prefetching etc.) As to point 2), if the data cannot be fed to the floating point units fast enough, then their SIMD nature cannot be exploited (ignoring the lack of Java's ability to easily do this - it's a language invariant problem).

To attempt to improve the speed of this operation, we threw the standard set of techniques at the code (loop unwinding, strip mining, etc) and even weirder attempts like trying to force prefetching via loop skewing/peeling and reducing indexing to single bytes. However, none of this could overcome the inherent slowness caused by poor locality of reference.

A Quick Look at Basic BLAS Results

For our own interest we compared the OG implementations of COO and CSR SpMV with equivalent operations from:

Colt - has sparse support and uses threaded SpMV().
Commons Math - has sparse support
Universal Java Matrix Package (UJMP) - has sparse support
JScience - has sparse support

The following were not used (reasons given):

Efficient Java Matrix Library (EJML) - no support
Jama - no support
jblas - wrapper to native code, cheating!
Matrix Toolkit Java (MTJ) - no support
OjAlgo - no support
Parallel Colt - wrapper to native code, cheating!

For the tests we used a set of increasingly large matrices with a fixed sparsity and ran essentially the same setup as before (harness code thrashes cache, warms up JVM, does multiple timings. Machine is a reasonable desktop machine). The results we obtained are below, each pair of graphs corresponding to the percentage of non-zero entries as given in their titles. The horizontal axis of each graph indicates the matrix size being tested (matrices were square n x n). The graphs in the left column present the raw timings obtained and are plotted on a log₁₀() scale in the vertical axis. The graphs on the right show the relative speeds of the sparse matrices from the tested packages (as mentioned above) against the two OpenGamma sparse matrix types, the dashed lines are comparing against OpenGamma COO and the solid lines against OpenGamma CSR, again the vertical axis is at log₁₀() scale. The plots referring to 25% and 50% non-zeros are missing results from the jScience implementation as the run times were so long (running into days) that we stopped them!

As a small post analysis, maps have an advantage in that adding elements is relatively easy whereas adding elements to CSR or COO is less so and can involve some quite expensive memcpy()ing, evidently we haven't tested the performance of this yet! However, if the data structure is immutable, as it often is in numerical applications (as a result of some properties of a numerical model, i.e. element density and ordering), then operations involving the fixed sparse matrix are of more importance. Furthermore, a large number of numerical methods rely on the repeated computation of matrix vector products, for example, a huge number of Krylov subspace methods, and so it is of importance that this kernel is fast. Evidently it is this sort of operation that was tested above and the OpenGamma implementations perform relatively well. It is also interesting to note that throwing threads at the problem, as done in the Colt library, does not actually increase the performance of the operation unless sufficient arithmetic density is available within the data set. This result can be seen first in the n=900 and n=1000 matrices with 10% non-zeros, it is at this point that sufficient work is present for the threads to be kept busy (perhaps as a result of the less false-sharing or better out of core cache characteristics). Finally, for reference, an additional screen shot of jConsole (below) shows that the CPU usage is incredibly poor and this is like due to the poor locality of reference which inherently hinders the floating point operations (the sudden jump in the number of threads used is due to the threaded Colt library).

Things That Are Going To Make Life Hard

This leaves us in an interesting place, and it turns out that solving the previously discussed locality of reference problem also assists with other associated problems occurring in functions such as matrix decompositions. Given that most whole matrix operations can be rewritten in terms of a row or column permutation of the matrix (assuming infinite precision arithmetic) we can try to apply techniques to create a more amenable form with respect to a given performance model. For example, in the case of matrix vector multiply, it would be ideal if we could find a permutation that moved the data in such a way that contiguous blocks formed near the diagonal such that at least some stride 1 access in the "x" vector were possible. It turns out that a similar behaviour is also needed for algorithms such as sparse LU decomposition and sparse Cholesky decomposition. This is to reduce the number of fill-ins (writing data to an originally non-zero part of the matrix structure) caused; basically the same permutation idea but this time with a different optimisation criterion. Amusingly it transpires that this problem is NP-Hard. However, in large part thanks to the GPU community considerable research has continued into this area as any poor locality of reference problem on a CPU is considerably worse on a GPU. Furthermore, a number of further highly specialised storage formats making use of combinations of compressed sparse row, blocks and multiply packed indexes have been derived. These formats will also increase performance but generally rely on permutations found by attempting to solve the NP-Hard problem and having a sufficiently high level of arithmetic intensity (due to a lot of data!) such that reordering and branching code costs can be amortised.

To demonstrate a possible bandwidth reduction on the previously used matrix, the permutation A(P,P) as given below tightens the matrix around the diagonal thus increasing the possibility of performance gain.

P = { 5   1   3   4   2 }

A(P,P) = 12    0    9    0    0
         11    1    7    0    0
          0    0    0   10    5
          0    3    8    0    6
          0    2    0    0    4

Conclusion

Essentially we reach an impasse. We are currently working on bandwidth-reduction and fill-in reduction algorithms to try and increase the performance possibilities within the sparse classes. Sparse decompositions are also a rather challenging area programmatically partly due to the algorithmic complexity required (building permutations, computing up-date trees, pruning trees, updating partial matrices, etc.), and partly due to the lack of pointers in Java making life generally hard work. Still, we shall persevere, and next time I write we should hopefully have some working prototypes.

First R Sample - Market Data Plotting

2011-11-03T00:00:00

We've been overwhelmed since we went public with the OpenGamma Platform with the number of developers, analysts, and risk managers who want to combine our data management and calculation capabilities with the statistical power of the R environment.

Much like we decided to actively embrace Excel as a front-end for driving an OpenGamma Platform installation, we thought that if people wanted to use R, it was up to us to make sure that we had the best support for it possible.

As the first public sample of just what we're getting at, consider the ChartSeries3D component. The example they provide is plotting a year's worth of yield curves, a pretty ideal way to showcase that rather than pulling this data from a file, you can pull it directly from the Historical Time Series service in an OpenGamma Platform installation.

I wanted to start with perhaps the simplest thing possible: plot the market data points that form your yield curve definition over the course of a year.

All that from this relatively small sample of R code:

In the next installment, I'll show you how we've modified this example to not only plot the raw market data, but to actually fit the yield curve using the OpenGamma Analytics library from data loaded into the R environment, and then bring back the fully fitted yield curves for plotting.

And that's before we show you the real power of what we're doing: driving full shocks, stresses, and historical regressions of whole portfolios from R and using R to analyze the results. We think R users in finance are going to find this exciting.

OpenGamma Maths Library Development Kicks Off In Earnest

2011-10-14T00:00:00

I'm a new face here at OpenGamma, having joined this summer, and I wanted to introduce myself and what I'm doing here. My job here can be stated simply: make the OpenGamma maths library as fast as possible!

Having come to OpenGamma with a background in applications of high performance and accelerated computing in numerical modelling, I have a fair idea about how to make code performing mathematical operations run fast. However, doing this in Java as opposed to my beloved Fortran or C is a completely different game.

First, some comments about why on earth we are doing maths in Java. Well, "write once, run everywhere", coupled with the rather magic IDE behaviour of Eclipse (although I still prefer vi ;-)) means that the code written can be guaranteed to run without having experts in cross-platform compiling around to deal with any issues. The Eclipse editor also makes unit testing, debugging and general code writing considerably less error prone. Java also has the ability to call out to native code via JNI or JNA, both of which can be used in the future, should the maths library interfaces be written in a sane manner conducive to their integration.

What currently exists for maths in Java?

There are a number of well-established maths libraries for use in Java. Two of the most commonly used are Colt, and Apache Commons Mathematics, and it is these libraries that we currently use in-house. Having briefly looked through the source code, there are some features/methods that are lacking (for example, more advanced gradient-based linear system solvers) and experience suggests that the performance (in terms of speed) of these libraries is not particularly good in comparison to native code (expected) or more heavily tuned Java code (as will be seen later). On the positive side, both libraries are tried and tested and have different, but relatively easy to use, APIs.

Why do we want a new maths library?

At OpenGamma speedy maths is essential. Financial analytics uses a wide range of numerical techniques from very simple models and operations (basic integration and curve fitting) through to the considerably heavier traditional numerical modelling techniques. Clearly these models have to run in near real time or at least fast enough that their execution completes between ticks so that new analytics can be displayed soon after data arrives down the wire. Obviously, to achieve this, speed is key.

How are we writing the new maths library?

A large number of mathematical operations are essentially matrix related and in acknowledgement of this, the industry standard BLAS APIs are quite wonderful as building blocks. Furthermore, evidence of the importance of this API comes from the chip vendor specific implementations of BLAS (along with extensions to sparse matrices and FFTs). Sadly, there is no complete and heavily optimised BLAS library for Java. This is where OpenGamma comes in.

So what exactly are we doing?

In the last few months I have been working on identifying code patterns and techniques that the JVM can spot for optimisation. In general, these techniques are relatively simple and already well known, and are extensively used in the high performance scientific computing world. I've been concentrating on BLAS operations, specifically DGEMV() with alpha = 1 and beta = 0 such that the operation is essentially:

y:=A*x

where y is a [m x 1] vector of doubles, A is a [m x n] matrix of doubles and x is a [n x 1] vector of doubles.

Following this, a naive implementation of the operation can be written, and it is this version that appears in many Java linear algebra libraries:

So, from here we applied a whole load of optimisation magic and we've outlined this in a paper that you can download from here. Have a read; most optimisations aren't too hard to do and we found out some interesting things about the JVM.

Results:

In general, it is nice to know that it is possible to improve the speed of code, and indeed knowing what happens in the metal is worthwhile. All the results relate to the computation Y:=A*x for square matrix A of varying size. Implementation details can be found in the previously mentioned paper.

Comparison to Colt and Apache Commons Mathematics:

The purpose of this investigation was to find out if it was possible to improve on naive implementations of a simple BLAS operation, and to provide comparison to the Colt and Apache Commons Mathematics libraries. Results from racing OpenGamma code operating on fairly typical matrix sizes, against a) a naive double[][] backed matrix implementation b) Colt library c) Apache Commons Mathematics library are displayed below.

From this graph we can see that OpenGamma code runs on average at least 1.6x as fast as Apache Commons Mathematics. Furthermore, the OpenGamma code consistently runs at over 2x the speed of the Colt library code and peaks at around 2.4x. These comparisons demonstrate what is possible if you push Java hard and they give a hint of what is to come... watch this space, maths is about to get fast!

Update on 18 October, 2011

Based on a number of discussions on mailing lists and websites, we wanted to provide some common questions and answers.

Question: Why is OpenGamma not just using XYZ that already exists to provide BLAS?
Answer: In al cases we have found so far either the adaptability, the possibility of interfacing with native libraries, or the performance has been insufficient for what we want/need, hence us writing our own.

Question: Why not just use the netlib Java interface to native BLAS or the netlib f2j translation of BLAS?
Answer: We are fully aware of this project. To use a native BLAS, the calls have to go through the Java Native Interface (JNI). All calls through the JNI incur a penalty from crossing the JVM->native barrier and there is uncertainty over the cost of this jump. Clearly, for some BLAS1 (and perhaps some BLAS2/3) calls on smaller data sets, the cost of accessing native libraries will be prohibitive. Therefore, the interface would need tuning to be performance optimal, on top of resolving and difficulties with running native libraries alongside Java. As to using the f2j translation of BLAS or the higher level netlib-java API (MTJ), we really want to have something that performs access directly on native types and can be performance tuned and don't feel that MTJ offers this opportunity.

Question: Why not just write in native code/use a native BLAS?
Answer: We want to keep everything in Java for now; by doing this users can just download one package to get a system that JustWorks(TM). We are writing our library in such a way that jumping out to a tuned BLAS library when it becomes efficient to do so (in terms of setup time/user experience) will not be a problem.

Question: Why do I get different performance results on my machine?
Answer: Different hardware+JVM combinations are going to produce different performance results. The specification of the hardware+JVM we used is given in the above paper. The hardware we used is a fairly powerful number cruncher and so may well give better results for maths performance. Also, factors like how "warm" the JIT is, how much garbage collection goes on and how much heap is allocated will also affect the performance.

Question: These optimizations are really basic. So what?!
Answer: Correct, they are basic! It doesn't mean that they don't work though. Part of the investigation was to see which optimisations were available by default within the JIT and what could be coaxed out of the JIT with a bit of effort. We are fully aware that tuning to hardware is not a very Java like approach, but given the constraints (performance+staying in Java), it is something we are investigating and believe can be made to work.