Grid Designer's Blog

Data-Aware Routing on a Cloud, featuring Sun Grid Engine, GemFire and EC2

2009-05-18T04:19:00.000-07:00

We are excited to announce that we have taken our Convergence project to the next step in the last few weeks. Last time we demonstrated how data aware routing can speed up the combination of compute grids and data grids. Since then, we have developed new grid adapters for our Convergence project and moved to the cloud.

New adapters: Sun Grid Engine, GemFire

Sun Grid Engine (SGE) is an open source batch-queuing system, developed and supported by Sun Microsystems. SGE is typically used in a server farm or high-performance computing (HPC) cluster and is responsible for accepting, scheduling, dispatching, and managing the remote and distributed execution of large numbers of standalone, parallel or interactive user jobs. It also manages and schedules the allocation of distributed resources such as processors, memory, disk space, and software licenses. We have developed an adapter that wraps SGE's Java API and enables data aware routing of tasks.

GemFire is an Enterprise Data Fabric (EDF) solution from GemStone Systems, Inc. It is a high performance, distributed in-memory-data-grid (IMDG) that offers very low latency, high resiliency, scalability and high throughput data sharing and event distribution features for high performance computing applications that need access to real-time data. We have developed a monitor component, which can query location of data from GemFire regions with partitioned schema.

Running on the cloud

Along with integrating new grids, we changed our demo application to use a cloud infrastructure. It is deployed now on Amazon EC2. Our demo now allocates servers, completes setup of cluster software (SGE and GemFire in this case) and starts the demo application server on the fly. Just a single click and few minutes later you will have a new cluster up and running.

The deployment process is straightforward and requires just a couple of scripts. First, we allocate one EC2 server and start the master node. When the master is started, it starts allocating worker nodes and sets up the grid software. When the cluster setup is complete, we start the demo control UI on the master node, and from that moment our interactive data aware routing demo is available. Each cluster has a time to live, and when its lifespan expires, all servers are returned to EC2. We are using OpenSolaris AMI, provided by Sun Microsystems, for all our servers. Deployment of GemFire is trivial --we just need to copy few jars. Installing SGE has its quirks, but after we have figured out how to do it correctly, it just works.

From a usability point of view, cloud hosting has huge benefits. Each developer can work with his own cluster, or even several clusters (e.g., comparing the effects of data aware routing between clusters of different size).

Data-aware routing demo

We have adapted our existing demo application to work with new grids and be scalable on a cloud. In our demo application we are simulating a financial application. We have a set of trade objects, which are loaded into a partitioned GemFire region. 50k objects are placed on each server (size of each object is about 2Kb). Those trades belong to different equal-sized books (5K trades in each book). The job here is to evaluate all books using Sun Grid Engine to distribute work across cluster nodes. This means that a job consists of 10*(number of servers) tasks and each task should fetch 5K trades (about 10MB of data) from the data grid and perform some calculation over them. For simplicity, we just sum the IDs of the trades and return the result.

Unlike DataSynapse GridServer, SGE is a process-oriented computational grid. Each task in SGE is an operating system process, and execution time includes all JVM startup and class loading overheads. Usually tasks are long enough, and process start up time does not make much difference, but in the case of an interactive demo we should find a balance. We cannot make the user wait half an hour to see the first result, but if we make tasks too short, the effect of data aware routing will not be visible due to JVM startup overheads.

The demo runs a constant flow of jobs and constantly measures job completion latency and task completion latency. To illustrate data aware routing advantages we introduced 3 different scheduling modes. “Data aware” mode is where we perform data aware routing, ensuring local space access for the engine. “Neutral” mode is where we do unguided DS scheduling as SGE sees fit. “Anti data aware” mode is where we deliberately violate data awareness and ensure network space access. The user can change the task scheduling mode on the fly and see the impact of the scheduling mode on performance.

The diagram on the left shows the completion time of the job and small gray bars show cumulative task execution time on each server in cluster. Gray bars do not include JVM start up time, so cumulative execution time on each server is considerably smaller then job execution time.

The diagram on the right shows average task completion latency (without JVM startup time), so you can see effect of data aware routing on task and job level.

Conclusion

We have demonstrated how a generic data-aware routing approach implemented in the Convergence project can be used with different grid products. We had a very good experience migrating our research/demo platform to the cloud. Using a cloud provided us flexibility and comfort of development, which are hard to achieve in traditional resource constraint environment.

Stay tuned for more advances in the Convergence project!

Waters Power 2009 Conference

2009-05-05T09:50:00.000-07:00

Grid Dynamics was one of the sponsors for Incisive Media's Waters Power 2009 event that was held in New York City. Main agenda of the conference was to showcase the most up-to date developments in HPC with in-depth analysis of cloud computing, virtualization and SOA solutions, bringing the latest strategies, techniques and technologies that give optimum performance and maximum efficiency for any data center. Major HPC vendors and many wall street firms were represented at the conference

Key note speech was presented by Jeffrey Birnbaum from Merrill Lynch. It was a well presented speech that highlighted the opportunities, challenges and approaches of cloud computing for HPC. Highlights from his speech are

Main attraction of cloud computing is to drive the cost of computing down. Google and Amazon had done a good job getting the cost pretty low. How can the enterprises do the same, or come close?
Most cloud providers like Amazon, Google etc. are GbE Based. Everyone is moving towards 10GbE and this serves as foundation for viable clouds
Enterprise cloud infrastructure needs a global file system. All software is installed on that file system. This makes it simple for the end users, in order to run any compute environment - just mount a file system
In order to get global scalability - use multiple copies of the files
Replicate the files in real-time on any update. It might be better to wait on update than deal with eventual consistency
For better performance cache the files in regional locations
Do not provide node-level redundancy and increase the cost of hardware, buy commodity hardware and design for failure. Route the workload from a node that failed to some available node (like what Google and Amazon do)
Design Data Centers around PODs connected by layer-2, not layer-3.

Technologies that will change the world,

Multi-core/multi-socket commodity compute nodes (run more VMs, writing more parallel code)
10GbE with iWARP or RDMA (lower latency to storage)
Flash-based storage at 200K IOPS (totally changes how you think about the problems)

Implications of the above changes are dramatic and will affect the way we design the Data Centers and Applications

There was a Panel Discussion about "A silver lining for cloud computing". Panel included Victoria Livschitz, Founder and CEO of Grid Dynamics. Questions asked by the moderator triggered insightful discussions and sometimes contradicting answers,

Define cloud computing? This is probably most asked question in cloud computing forums and glad to hear we are closing in on a definition. This brought up another question about what stage of technology maturity is cloud computing in. Victoria's response to this was that it is in really early stages and it will take about 5 yrs to be mature enough for enterprise adoption
What applications are not suitable for cloud computing? It was acknowledged by few panel members that low latency applications may be bad candidates before cloud infrastructure matures but one speaker thought that this is up to the design of the cloud and it is possible to host low latency applications if architected well
An interesting conversation that came up is when cloud will get more adoption. This is same question that keeps coming up when any new technology comes into light. Old companies with huge investments in the existing infrastructure and approaches will need to spend more time adopting while newer players will benefit from them much sooner
Victoria made an insightful argument about the way applications were designed and implemented, this was about the data and its ownership. Most of the applications now are designed with data ownership as the premise but the promise of consuming and providing data as a service has many opportunities. For e.g., cloud computing is making infrastructure, software and computation as a service and having data also as a consumable entity gives the applications unbound possibilities

The thoughts about the data ownership and promise of being able to consume this as a service were echoed by few other speakers in later discussions

Ken Michellini's (from CitiHub) speech about "Keeping your feet on the ground: Unveiling the truth about cloud computing" was educational. It talked about the ROI calculations in a public cloud, 3rd party hosted and internal cloud scenarios and gave some ideas on how you can make these decisions when building your next big application. He also talked about characteristics of a good cloud application and typical challenges in building cloud applications.

There were few more Panel discussions that we attended including "Building the 21st century data center", "Preparing for Next phase in grid computing: What it takes to build the perfect data grid" and "Virtual reality: Optimizing storage, application and network virtualization" and another presentation about real life lessons learned while administering grids "Nuts and Bolts: Practical issues in grid administration"

I would like to extend congratulations to Incisive Media for a well conducted conference that let many financial industry experts brain storm and discuss the opportunities of Cloud. Great job guys!

Full-text & faceted search over In-Memory Data Grids

2009-02-09T22:39:00.000-08:00

Modern in-memory data grid (IMDG) solutions provide different facilities for execution of queries over whole stored data sets with different levels of sophistication. Oracle Coherence provides Query facilities (one time full scan and continuous querying with Cost-Based-Optimized). GigaSpaces has a JDBC Query interface with the ability to use hash and B-Tree indexes. Those solutions work quite well for many problem areas. However, for heavy loads and complex multi-criteria queries those facilities can quickly become a bottleneck.

There is a class of workloads that produce high loads with complex queries on IMDGs. Retail companies that use IMDGs for their item catalogs are a good example. Those catalogs are hit by diverse stream of multi-criteria queries. A typical query you may see there looks like:

give me cell phones with MP3 support, Java and in red color.

Fortunately, the Compass Framework allows you to process such queries effectively. You can build inverse indexes with Apache Lucene and store them on a grid. This capability is based on the very modular design of the Lucene framework. All index I/O operations are well-hidden by the abstraction of FileDirectory.

For now Compass provides implementations for Coherence, GigaSpaces and Terracotta, introducing an unprecedented ability to build a vertical search solution on top of In-Memory Data Grids.

In addition, Compass has a sophisticated object-to-document mapping system that allows you to make stored objects searchable just by adding Java annotations or XML mapping files. Mappings can also be built in runtime.

However, despite its great codebase, Compass documentation is pretty sparse. It may take significant time to dive into the code and docs to get what you want. But the results will overcome all your expectations. Search engine performance on top of data grids easily overcomes any old-generation search technology.

Enjoy!

Speeding up data-intensive HPC applications with Velocity

2008-12-22T00:57:00.000-08:00

Massively parallel, data intensive applications that run on computer grids require timely access to their data. When the application data is distributed among the grid nodes, data access is not a problem because it scales along with the number of computational task. However, when the application data is stored in a centralized database, access to data can quickly become a serious bottleneck.

We encountered such a problem in one of our recent projects, where a single database was used as a centralized application data repository for the entire compute grid. In this case, access to application data became bogged down enough to cause an overall degradation of the system performance, despite the fact that the database was being hosted on sufficiently powerful server.

In this article we will describe different ways of reducing database load and how they affect the overall system performance. We will also explore the advantages and disadvantages of Velocity as one of the ways to reduce data access latency. More specifically, we will show that:

Database load can be reduced by introducing Velocity distributed cache, or by simplifying SQL queries with moving all calculations from SQL to application logic.
Velocity CTP1 distributed cache improves overall system performance dramatically. In our case, querying data with active Velocity cache was up to 31 times faster than without it!
In CTP1, we experienced some scalability issues in particular configurations, but we are confident these issues will be resolved in future releases.

Compute Environment Overview
HPC++ CompFin Labs is a compute cloud that was created to give university students the ability to run massive analytical financial computations.

When a computation needs to be performed, one has to create a custom computational model that defines how the calculation will be performed and how the resulting data will be handled. The computational model operates in accordance with the MapReduce paradigm and includes an Excel-based UI with a list of input parameters, and the .NET assembly with the logic for splitting the computation into tasks, running a single task (map task) and combining task results into computation results (reduce task).

For example, one may write a model that performs some statistical calculation for a given stock symbol over some period of time. Later, someone else can open the Excel UI, change the stock symbol and the time period, and re-run the calculation. The following diagram illustrates how the example model would work in a CompFin cloud:

Picture 1. Original CompFin

The system consists from the following components:

A Microsoft HPC Server 2008 cluster.
A Sharepoint Server, where the CompFin website and the computational model files are hosted.
A SQL Server, where computation results are stored.
A SQL Server Intermediate Storage Provider (ISP), where task results are stored for subsequent consumption in the “reduce” phase.
A centralized market data database, where all financial data is located.

The computation proceeds as follows:

The user navigates to the CompFin website. On this website, the user chooses the model to run and opens the appropriate Excel file. Next, the user fills in the required computation parameters and, with CompFin Excel plug-in, submits the job.
The job start request is handled by the CompFin web service, which typically runs on the Sharepoint server as well. This web service calls the model logic to split the computation into tasks.
All computational tasks are submitted to the Microsoft HPC Cluster.
The map tasks are started first. Each task retrieves appropriate data from central market database, performs calculations on this data, and submits intermediate results in the SQL Server ISP.
When all map tasks are finished, the reduce task is launched. It retrieves intermediate data, combines it into final results data, and stores final results into the result storage.
After the computation is finished, the user may request job results via the CompFin website.
The computation results are retrieved from the result storage and returned to the user.

The above compute grid uses one SQL Server database for storage of stock prices, and one SQL Server database for storage of intermediate and final results. The combined compute capacity of the HPC compute cluster is 400GB of RAM and 200 (2.00GHz) Xeon cores, in particular, 50 machines each with 4 (2.00GHz) Xeon cores and 8GB of RAM. The capacity of each SQL Server is 32GB of RAM and 4 (2.33GHz) Xeon cores. So, the compute cluster had 12.5 times more RAM and 50 times more CPU. Based on these numbers, we anticipated two potential places of bottlenecks – the central market database, and the ISP.

Finding the bottleneck
The computational model, chosen for our experiments, computed correlation between stock prices over some period of time and contained very data-intensive computations with complex queries to the SQL Server that required significant grouping and sorting. The model used significant amount of data, most of which could be cached. The anticipated maximum cache hit ratio for the model was greater than 90%.

First, we needed to conduct a test of the model. We decided to test all three ways of scalability measurement (one data logical unit is roughly equal to 32 millions of records in database or 512MB of tick data returned from queries):

Leave amount of data to process, increase processing power:

Increase amount of data, leave processing power:

Increase amount of data and processing power synchronously:

We tested the model using this test plan and measured the entire time of map task, time spent in retrieving financial data from central market database (GetTrades method), and time spent in SQL Server ISP. There is also the graphic, which shows how the task time will behave if the system is completely linear:

Picture 2. Map task time distribution of original model in CompFin

You may notice that the bottleneck was in the central market database: time, spent in retrieving financial data is almost equal to the entire task time. You also may notice that currently the system is much worse than linear. So, it was a good chance for Velocity to help to improve the performance and scalability.

Getting rid of the bottleneck – Velocity cache for tick data
At first, we created a Velocity distributed cache for financial data. This was just an intra-job cache for reducing the negative impact of data-reuse inside the computation job. Complete replacement of central market database by the Velocity cache was not considered at this stage. This approach is shown as on the picture 3 (blue cloud):

Picture 3. CompFin + Velocity improvements (distributed and local caches)

Velocity cache for tick data ran on HPC cluster nodes. More precisely, the same machines, where the computation tasks ran, were also used as Velocity hosts.

We compared this approach with the Original CompFin:

Picture 4. Comparison of map task time between original CompFin and CompFin with Velocity distributed cache

In the beginning, everything was working well. Velocity distributed cache reduced the number of requests to the central market database and performance increased dramatically.

As the number of processors used for the computation increased to 128, the performance started to decrease. On 32 machines (128 processors), Velocity still dramatically outperformed the original CompFin results, but some performance degradation was experienced relative to the 16 machine (64 processors) scenario. On 50 machines (200 processors), in CTP1, we began to experience some failures and timeouts.

Nevertheless, keeping in mind that Velocity was in first CTP, the results were still quite amazing.

Velocity scalability – local cache & data-aware routing
Fortunately, we had a plan of how to further improve the performance and reduce the load on Velocity distributed cache. We enabled Velocity “local cache” and added data-aware routing in the CompFin (see picture 3). This was possible, because all tasks, which execute on the same HPC node, execute in the same Windows process. The local in-process cache can improve performance if the data-reuse factor for tasks that run in the same process is high. Hence the data-aware routing was required.

Below is the comparison between this approach and the approach with Velocity distributed cache alone:

Picture 5. Comparison of map task time between distributed cache and local cache

The problems with Velocity distributed cache scalability were solved, and we can conclude that Velocity can work well in CTP1 even on large clusters.

This approach performed from 5.6 to 31.9 times better than original CompFin. It also scaled better than any other approach, although not linearly. We believe that this is because the interaction with the SQL Server was still required before data was retrievied from the cache.
However, in the “Finding the bottleneck” chapter we mentioned that the model used complex queries with grouping to SQL Server. So, maybe the performance can be improved by just simplifying these queries and moving all logic from database to compute nodes.

Getting rid of the bottleneck via simplified queries
Following the previous assumption, we tried to reduce the load on the central market database by simplifying the queries used in the model through removal of grouping operations. The results were largely positive:

Picture 6. Comparison of map task time between original CompFin with complex and simple queries

The time, spent on retrieval of tick data, was reduced from 8.5 to 52 times and the task time became almost linear. However, the scalability problem with the SQL Server remained the same, so we tried to further improve the time by using our previous best approach – Velocity distributed cache with enabled local cache and data-aware routing:

Picture 7. Comparison of map task time between original CompFin and CompFin with Velocity local cache. Simplified queries are used in both cases.

Surprisingly, doing so did not improve the task performance significantly. Perhaps, the overhead of putting items in Velocity distributed cache was too big, or the cache services just stole CPU cycles from computations. Nevertheless, we believed that these factors had only a minor impact on performance so we decided to compare only the time spent in GetTrades method, where the financial data was retrieved:

Picture 8. Comparison of time, spent on retrieving tick data, between original CompFin and CompFin with Velocity local cache. Simplified queries are used in both cases.

Our assumptions were right and local cache approach greatly reduced time spent for retrieval of financial data. However, one question remained – why the overall task time was not reduced? To answer this question, we performed a number of tests to investigate the time distribution of task time, when the local cache was used:

Picture 9. Time distribution in CompFin with simplified queries and Velocity local cache.

These tests revealed that the SQL Server ISP, that was typically the least time consuming element of the system, now moved to the foreground. Since retrieval of financial data was now very fast, the frequency of write operations to the SQL Server has increased, and the SQL Server ISP became a bottleneck.

Our conclusion is that Velocity can improve even a very fast model, where only simple queries are used. In this environment, however, the SQL Server ISP remained as a bottleneck. We have yet to experiment using Velocity as the ISP with simple queries.

Conclusion
In this article we have shown that the database is the bottleneck for data-intensive applications, running on compute grids. We tried to reduce database load using two approaches: introducing Velocity distributed cache between the database and compute grid, and simplifying SQL queries to the database by moving all calculations and aggregations to the application logic.

Both these approaches improved the database performance dramatically. In particular, querying data with active Velocity distributed cache was up to 31 times faster than without it. In certain configurations, when experimenting with CTP1, we also observed some Velocity scalability issues; however, for a first CTP, Velocity performed very well. At the moment of publishing, Velocity is already in CTP2 and its development continues to be in very active stages. In the near future we expect the Velocity team to spend a lot of time focused on improving scalability and performance prior to release. Velocity team has big plans and not only will some issues soon be resolved, but lots of new functionality will be added as well.

So, if you have problems with database scalability and you use Microsoft technology stack, you may consider introducing Velocity to your system.

There are still many interesting topics to investigate regarding the Velocity and CompFin. Also, there can be some other improvements can be made in CompFin, so we hope that this project was not the last one, where we were working with Microsoft technologies.

Cloud Performance Reports

2008-09-24T14:45:00.000-07:00

Cloud computing is getting a great deal of attention these days. Unfortunately, there is very little data available about performance, scalability and usability of the cloud deployment platforms.

I was recently invited to speak at Silicon Valley Cloud User Group where I tried to bring a "practitioner's prospective" and present the results of three different recent performance and scalability benchmarks related to the cloud computing. The first benchmark aims to establish the scalability of EC2 on a perfectly parallel mathematical problem, a Monte Carlo simulation, executed by Grid Gain's popular open source map/reduce platform - and to document lessons learned in making the application scale to 512 nodes.

The second benchmark looks at a scalability of a more complex stateful application, typical to Risk Management, that required both in-memory data grid and compute grid. Both grids were running on EC2 and executed by GigaSpaces' data & compute grid platform.

The third benchmark looks at a prototypical data-intensive Portfolio Analysis application used heavily in the financial services industry, and studies the performance impact of data being located close to computing, or "on the cloud" vs. "off the cloud". This work was done in collaboration with Microsoft on their HPC++ CompFin Lab that integrates Microsoft Windows HPC Server, a central market data database and Microsoft productivity products to provide academic community with an online service to publish, execute and manage computational finance models.

You can find the presentation, with summary of results here. Please, note that these results are very fresh and the benchmarks in two cases are still going on. You can find far more details on the first benchmark in our previous blog post. We will be coming with more detailed blog reports for the second and third benchmarks soon.

Scalability Benchmark of Monte Carlo Simulation on Amazon EC2 with GridGain Software

2008-07-31T06:08:00.000-07:00

This blogpost presents the report on recently concluded scalability benchmark of Monte Carlo simulations running on Amazon EC2 using the GridGain framework. It consists of two parts: Part I is a technical report on the benchmark goals, method and results and Part II is an account of the development process and lessons learned.

Part I: Benchmark description & results

The goal of the benchmark was to study the scalability characteristics of massively parallel algorithms executed on Amazon ’s Elastic Computing Cloud (EC2) and managed by GridGain

A Monte Carlo simulation was chosen as an algorithm to represent its widespread use in financial applications. The same algorithm with different parameters was used on a wide range of grid sizes: 2, 4, 8, 16, ..., 256, 512. The parameters guaranteed that the amount of work performed by the whole grid was always linear with respect to the number of nodes. In other words, twice as many nodes always performed twice as much work. Perfect linear scalability would demonstrate the identical *completion time* of Job-1 running on 2 node and Job-2 running on 512 nodes, given that Job-2 had 256-times more work to do.

As Amazon permitted us to use the maximum of 550 nodes in a single run, the upper limit o fthe benchmark was chosen at 512.

The test utilized a full open source software stack, including GridGain, the Linux operating system, Sun Microsystem’s Open MQ JMS messaging and Sun’s Java 5 VM. Amazon’s preinstalled Fedora Core 8 with custom testing framework was used to conduct the benchmark.

The results justified our hopes: we could successfully run up to 512 nodes without significant performance degradation. The results graph is shown on figure 1.

Figure 1. Average task execution times on 2-512 nodes grids.

The performance degradation of 3 seconds (about 20%) should be considered minor given roughly 250-fold increase in scale. The curve rises two times: in the ranges 2-8 and 256-512, while 8-256 remains almost flat.

The range 2-8 growth can be explained by initial rapid growth of grid size, which causes the rise of maximum task execution time among nodes (the overall time is determined by the “weak link”). The range 256-512 growth could be explained by OpenMQ limitations as applied to our use case (the JMS load is quadratic against the grid size). In order to continue scaling the grid beyond 512 nodes, a clusterization of OpenMQ is likely to be required.

We can conclude that the obtained result showed near linear scalability and performance improvements from 2 to 512 nodes in all test runs.

Part II: How did we do it?

To start using EC2, just open Amazon EC2 Getting Started Guide, download EC2 API tools and follow the instructions. You can use one of the predefined public machine images or create a bundle with your own image.

The relatively big problem is the lack of persistence options in EC2. When an instance goes down, all the data is lost. However, there are third party solutions capable of mounting Amazon's S3 storage interface as a Linux filesystem.

Now, we needed to create a simple framework allowing us to manage the grid on Amazon’s EC2. The basic functionality would look like:

start the grid;
add/remove nodes to/from the grid;
display the grid health;
run single calculation task interactively;
run a benchmark (batch task execution on 1, 2, 4, …, 512,... nodes).

The first problem we encountered was that multicast is not supported in the EC2 infrastructure. By default GridGain is configured to make initial discovery by IP multicast. Unfortunately, as far as we know, the EC2 guys don’t plan to create fully functional multicast support anytime soon.

Luckily, GridGain is shipped with a set of various Service Provider Interfaces (SPIs). So, we could choose another DiscoverySPI that does not use IP multicast for discovery purposes, and we chose JMSDiscoverySPI. At first, we picked Apache ActiveMQ as a JMS implementation and soon ran into stability issues. Then we switched to OpenMQ and it proved to be sufficiently robust.

Our second problem turned to be the default maximum number of running instances per user – 20. Since we were going to run grids much larger than 20 nodes, we needed to override the default limit. It took several steps and a few more days to negociate with Amazon EC2, but eventually, we were granted the right to run up to 550 nodes. It seems that the business process of requesting a large amount of nodes is still not very well-defined by Amazon.

The image creation was not a very hard task: we got the public Fedora 8 i386 image, ran it, installed Java Runtime Environment 6, GridGain and ActiveMQ, then bundled this new image.

We still required some extra configuration since the grid node must start on system startup. The hardest thing was to write shell scripts for starting the grid. These scripts had to parse user data sent to the instance (different for the Head Node and Worker Nodes), determine the node type and start the necessary software in each given configuration. Later these scripts were replaced by more convenient Java-based tools and Head Node’s ActiveMQ was replaced by a standalone OpenMQ.

Figure 2. Initial GridGain on Amazon EC2 architecture.

So, how does the grid work? First, we start the Head Node (see figure 2). Head Node automatically starts ActiveMQ (only in early versions), GridGain in master mode (will be described later) and runs the requested number of Worker Nodes. Since Workers should connect to the JMS server, Head Node passes its IP address as user data for Worker Nodes. Worker Nodes just start GridGain.

We realized quickly that we needed some user-friendly interface to manage the grid. So, we rewrote the scripts in Java using the typica library. Thus we integrated Amazon and GridGain management. Now we can automatically start EC2 instances, wait for them to start, get their IP’s, track status, etc. Together with GridGain management it becomes a very powerful tool. Let’s imagine the grid running completely autonomously. The management module can automatically bring up more EC2 instances or shut them down depending on the current grid load.

When we had the simple web UI with capabilities of seeing the table of grid nodes and running our benchmark, we felt ready to run grids larger than 20 nodes. We asked Amazon to increase our running instances limit to 1050 nodes. Amazon agreed to let us run 550 instances. The know-how of our UI was the embedded scripting engine, allowing us to change a benchmarking schema without restarting the grid. To understand how simple it is to run a benchmark consisting of several task runs on different number of nodes, just look at this code:

var itersPerNode = 5000;
var cnode = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512];
for (var i in cnode) {
var n = cnode[i];
grid.growEC2Grid(n, true);
grid.waitForGridInstances(n);
runTask(itersPerNode * n, n, 3);
}

Listing 1. Sample benchmarking script.

The closing remarks:

The framework we built is a good place to start developing a GridGain appliance. But what if we want to create, for instance, an In-Memory Data Grid on this basis? Our next step is making the framework more generic to allow easy integration with virtually any grid framework (either computing or IMDG). There are also other ideas such as creating a generic image and storing specific grid software/configurations in S3, which should ease debugging and small alterations to the appliance.

We are currently evaluating if our framework will be something more than just a GridGain testing framework. We'd love to hear from the community if this line of work is interesting to others beside ourselves. If you'd like to know more about this benchmark, or get access to the full source code, please contact me at mgorbunov@griddynamics.com

Grid technologies in middle-size applications

2008-06-19T02:03:00.000-07:00

Grid technologies were born to solve extreme problems and currently they are used primarily by large-scale applications. However, like computers, which were initially used only for solving complex scientific tasks and later came to almost every house, grid technologies are coming into middle-size enterprise market. In this article I will try to answer a question about how medium application can take advantage of grid technologies.

When writing large application that solve extreme problems, scalability is always an issue and you do not have a choice: you must invest resources in scalability and developers often have enough time and knowledge to solve this problem. When writing small applications, you usually do not care about this kind of problems, because all will work fine on a single server. But when writing middle-size application, you are in trouble, because you are already big enough to start thinking about scalability, but you are not big enough to invest a lot of resources in solving this problem. When middle-size applications grow from little ones the trouble becomes really serious. However, scalability problems in middle-size applications are usually caused by a very limited set of architecture decisions. Knowing these causes and their resolutions will help to build a more scalable application.

Whatever architecture you choose for a middle-size application, it will usually have a web server and a database. In the best case, it will have a single physical machine with both. In the worst case, it will have web servers, application servers and database server on separate machines. In all cases, the request processing chain will include several processes and will take a lot of time. Usually, the database server is also a bottleneck for the entire system, because it doesn’t scale well. While resolving these problems, mankind invented caches. Caches are useful to store some data that is costly to compute or retrieve. They can reduce both database load and request processing time a lot. However, caches can also be scaling killers.

Assume you have an architecture with a dedicated web server that hosts your application process. It is relatively easy to maintain a cache on a single server. But suppose you want to scale and the single server becomes a load balancing cluster, where each machine should maintain its own cache, and this cache should be synchronized with caches on other servers. For example, if some item is removed from the cache in one server, it should be immediately removed in every cache on every other server in the cluster. This is extremely hard to accomplish. But since developers often implement simple local caches by themselves, they try to enhance their caches to support distributed behavior. The problem is that developers often do not have enough knowledge and experience to do this. Fortunately, the problem of distributed caches is well known and the solution already exists.

The solution is to use third-party distributed caches or In Memory Data Grids (IMDG). In Memory Data Grids were created to solve problems of scaling data in extreme applications, where the cluster contains a hundreds of servers. Data stored on these servers can be partitioned between them or replicated. If data is partitioned, each server contains only one chunk of data and each chunk is stored on multiple servers to provide failover. This allows huge amounts of data to be stored in memory. If data is replicated, it is stored in full on each server. Of course, the data cached on each server is synchronized with data on other servers. This allows very fast access to data, because it is always available locally. In Memory Data Grids provide lots of other interesting features, which deserve an entire book to describe. Distributed caches are essentially a simplified form of an In Memory Data Grid. Currently there are many implementations of both IMDGs and distributed caches and if you choose to use these technologies you have a number of options.

The concrete choice will depend on what technology or framework you use in your application:

If you are using .NET, you may use ScaleOut, NCache or Microsoft’s new distributed cache Velocity, which is currently available as a Community Technology Preview. They all provide an ASP.NET session state provider – the easiest way to gain a benefit from grid technologies. With this session state provider you will not need to maintain a special SQL Server for storing ASP.NET session data, because all data will be distributed between web servers in the cluster in a reliable and robust way.
If you are using Java, Oracle Coherence and GigaSpaces are the most famous In Memory Data Grids and, hence, can be used as distributed caches. They both provide a second level cache for Hibernate, so, if you use it, you can scale easily with no additional development efforts.
If you are using C++, PHP, Ruby or Python, you should consider memcached. This is a very famous distributed cache initially developed for LiveJournal, which has already helped to scale many extreme applications, like Wikipedia, YouTube, Facebook and others.

All these implementations will help you to solve distributed cache problems and scale well. In simple cases, to start using them, you will need to replace your old local caches with new distributed ones. If you need to scale a Hibernate second-level cache, or an ASP.NET session state provider, you will not be required to write any code. However, in the case of complex and serious scalability problems you can consult with us at Grid Dynamics any time and we will help you to solve them in a most effective way.

Open Source Grid Expertise and Thought Leadership

2008-05-31T15:22:00.000-07:00

Last week, an exciting event took place here in the Bay Area: The Open Source Grid and Cluster Conference was held in Oakland. Engineers and scientists who care about the complex interactions between open source and grid computing from around the world, discussed their projects and research insights. Grid Dynamics was represented in two ways:
Eugene, our CTO, gave a presentation about Convergence, one of our key project about integrating Data grids with Compute grids. Check out his presentation.

Eugene Steinberg, CTO, Grid Dynamics

Eugene presented the latest version of Convergence which integrates DataSynapse GridServer5.0 and GigaSpaces XAP6.0. Future versions of Convergence will provide adaptors for Sun Grid Engine, Oracle's Coherence and GridGain.

The second event was moderated by Victoria Livschitz, our CEO: A panel about Trends in Open Source Data Grids with experts from the open source grid computing industry: Doug Cutting, creator of Hadoop, Nati Shalom, CTO and co-founder of GigaSpaces, Nikita Ivanov, President and Founder of GridGain and Daniel Templeton, Manager at Sun Grid Engine.

The discussion started with each of the panelist introducing their products, current initiatives and their commitment to the Open Source community. Each participant had an interesting perspective particularly to the question on why Data Grids technology is deeply anchored in the Open Source community, but many Compute Grids are not.

Attendees directly participated by asking the panel specific questions around their technologies and development efforts.

Left to right: Nati Shalom, Nikita Ivanov,
Doug Cutting, Daniel Templeton

The discussion continued informally after the panel with conference attendees and speakers discussing product specific questions and key future technology.

Doug Cutting in discussion with conference attendees

Victoria Livschitz and Nikita Ivanov

Nati Shalom with conference attendee
The conference setting allowed attendees to actively share their experience first hand, ask questions in an informal setting, hear the experts talk about technical details and strategic directions about the industry.

If you want to know more about the Open Source Grid and Cluster Conference, here is link to this year's conference. I also found a few bloggers taping the key note speech by Fritz Ferstl and some pictures of the event. Enjoy!

Match Making in Grid Computing

2008-05-29T04:15:00.000-07:00

Data partitioning is widely used to store and manage huge amounts of data across several servers. Often there is a need to introduce global distributed transactions to keep the data consistent. Those global transactions represent a natural limit to the scalability for the entire system.

Project Convergence was born as an example of managing interaction between in-memory data grid and computational grid to address specifically problem of data consistency in systems with global distributed transactions. Based on the idea of data aware routing, network utilization and overall performance will be increased. In addition, data aware routing can also improve scalability.

How does it work? Let me illustrate the solution by using a simple example: You are the owner of a dating agency. A candidate comes to your agency and looks for a person with specific parameters such as the love for operas, an interest in cooking and a liking for dogs . Let's assume that the set of these parameters is unique for each person and all parameters need to be matching. So, your agency will find him a perfect match or he will be put into your database as a single new customer waiting for a match. Considering your candidate’s specific parameters, you expect that it will take some time to find him an exact match.

Your company has an outstanding reputation and you are flooded with candidates and your company cannot handle all the new candidates. You realize that you need to get a business partner and you partner with another dating agency in town. You also hire as secretary who greets your candidates and sends them randomly to one of the offices.

One day, two candidates with the exact same parameters (a perfect match) enter your office and your secretary sends one of them to your office and the other to your business partner’s office. Both of you start looking independently for an exact match simultaneously, but both of you cannot find one in your databases. As a result, you and your companion mark both candidates as single, waiting for a match while you miss the opportunity to generate a fee from the match. How can you solve this unfortunate incident?

First, you can have your secretary manage the assignment better by having her search your and your business partner's combined database: But while she is using the database, nobody else can access the database until she is done with her search. This may produce a match of the candidates, but create a bottleneck and hence impact the performance of your match making capabilities.

Secondly, you divide all candidates in two groups (for example, those that like dogs and those that do not, assuming there are about the same number of candidates that like or dislike dogs). This way, you and your partner are responsible for an equal amount of candidates. You will instruct your secretary to send incoming candidates to the offices depending on the likes/dislikes of dogs. Because you are processing only half of the candidates with one parameter already matched in your database, you will achieve a match at a faster rate than before.

This is exactly how in-memory data grids and computational grids interact and how data aware job scheduling can address the ‘matching of data grid and compute grid requests.’ As in our example, a database or an in-memory data grid (Oracle Coherence or GigaSpaces XAP), represents the databases with all the parameters of the candidates. You and your business partner’s represent the computational grid (GridGain, DataSynapse GridServer or Sun Grid Engine), and your secretary fulfills the function of a data aware job scheduler.

Choosing graph manipulation library for your Flex app

2008-05-27T02:22:00.000-07:00

The open-source movement plays a pretty big role in everyday life here at Grid Dynamics. We always try to do our best to improve the software we're working with and contribute back into the community.

While developing an RIA project we faced a need to display complex graphs within an Adobe Flex application. It turns out that there are not many solutions to this problem existing at the moment. Actually AFAIK there are only Flare toolkit, yFiles Flex, SpringGraph and Flex Visual Graph Library (further on FVG). Having done the comparison between the libraries we have chosen Flex Visual Graph Library as our current graphing solution.

Good Things and Bad Things

Here is the list of the most distinctive features of the FVG:

It is an open-source project so everyone can use it for free and modify it in the way he likes;
It can render not only trees but also (relatively) complex graphs;
It contains a number of algorithms for performing layout;
Laying out the graph is performed on the client-side and no server resources are involved in the process.

The library has some documentation and samples, although sometimes (and maybe too often) one has to dig into its source code to figure out the way it works.

However, it is a young project so it is expected to be imperfect. Here are the main drawbacks that sometimes annoy me:

A somewhat monolithic library design, god-classes anti-pattern here and there; favor is given to state changes instead of the parameter passing. I would also mention here coding conventions, although it is surely my personal impression;
Lack of documentation on layout algorithms; incomplete design diagrams, tutorials and examples.

When dealing with someone's else products instead of the homegrown ones you will almost always find its feature set is incomplete. This is the case not only for open-source projects but also for closed-source ones (including commercial) and the greatest advantage of open source is that it allows you to implement the desired features and not wait until someone else does it. And surely there were missing features in FVG that our customers and we ourselves wanted to have.

Our humble help

So here is a list of the things we improved:

Fixed and successfully harnessed functionality for adding and removing of nodes. Before the fix it was impossible to do that after the graph was initialized. It seems that the developers of the library just did not need it so they haven't got it into shape.
Fixed support for directed graphs. It was one of the most controversial changes to the library. Before, the layouting algorithms were tested only on nondirected graphs so the spanning tree in the case of a directed graph was computed in a wrong way. This led to the graph being laid out inappropriately. Now the user can choose if he wants to get the new behavior or emulate the old one.
Enhanced hierarchical layouter. It now supports interleaving of nodes. Sibling nodes (the nodes on the same level) are placed now in a checkerboard formation so every even node is put a bit higher than an odd one. This enhances the readability of the graph and the overall appearance if nodes have text labels under them.
Added possibility to sort sibling nodes in a specified order. This again enhances readability. Suppose that you have some text on your nodes. You can sort the siblings in the lexicographical order so you can more easily find the node you want with your eyes. Or you may want to display a certain type of nodes first. Plugging in your own sorting logic is tremendously easy.
Added a very cool feature: graph zooming. There was nothing like that in the library before. One could only try to tune some parameters of layouters in order to achieve zooming effect but the real zooming for every layouter was impossible.

In addition, we've been implementing numerous small fixes and improvements — and we are definitely going to continue this effort in the future.

Conclusion

In general, supporting the open-source library has the following noticeable peculiarity: as with other source management models, it is always needed to keep compatibility with those who already use your library in their applications. But when a need to make significant and incompatible changes to the library appears, one can always commit the changes into a separate branch. It turns out to be a rather common practice in the open-sourced world. Although this is not as convenient as just committing the changes directly to the trunk, the approach has advantages. It allows you to not break the compatibility and let other developers test and review the new functionality. We've been maintaining a separate branch for quite a long time, partly because of tensions with the maintainers, partly because of a lack of time to perform all the necessary merges and testing. However, not long ago we decided that this will likely cause the branches to diverge in some point of time — definitely that will not add value to the library, but rather subtract from it. So we've consolidated our efforts with the maintainer of the library and presented a demo of the new features. The guys were very collaborative and accepted our changes into the trunk with just a couple of minor changes.

As we going to use the library further we will definitely continue to put effort in. So our next step is going be reviewing the FVG's architecture and proposing plans for refactoring the code. On the features side, we're going to have a hierarchical layouter that will be able to support multiple root nodes, a zooming engine that will preserve label and node size, node renderer that supports overlay icons, improved edge-node connections... lots of fun things to come.

Stay tuned!

Partnership with GridGain

2008-05-05T08:52:00.000-07:00

Today we announced a new partnership with GridGain (see the press release here). I am extremely pleased to be launching this alliance and hope it will accelerate the already impressive adoption of GridGain's new and exciting open source Java platform for high-performance grid applications.

At Grid Dynamics we see GridGain as a simple, elegant, extensible, open-source, pure-Java platform that makes traditional high-performance computing and all modern map/reduce applications significantly simpler and cheaper to develop and deploy than it has ever been possible before. The platform's extensibility also allows GridGain to be used for a wider set of applications and enables the integration with other tools, systems and frameworks.

The partnership gives GridGain's customers an integrated Software + Service offering, enhancing GridGain's innovative technology with Grid Dynamics' expertise and track record in designing complex, scale-out applications. Customers are getting best-of-breed solutions from the industry leaders in their respective areas and should expect a seamless experience working across both organizations. Grid Dynamics and GridGain share the same core values, highly regarded by their mutual customers:

Both are fundamentally engineering companies, nearly obsessed with the performance, scalability and quality of applications we write.
Both recognize the importance of the Open Source as a medium for cost-effective and open software products with almost "organic" properties, which thrive and evolve in the marketplace under a watchful eye and the collective efforts of their designers and users.
Both are cost-efficient, global-from-the-cradle organizations with significant R&D operations in Eastern Europe and headquarters in the Silicon Valley.

Under the terms of the relationship, Grid Dynamics is creating a "Center of Excellence" around the GridGain product and extending its design patterns for scalable high-performance applications to include mappings to GridGain technology. Both companies are collaborating on the joint R&D projects, including the planned release of a GridGain plug-in for Grid Dynamics' open source project Convergence, designed to enable interoperability between the leading compute and data grid platforms. Together, Grid Dynamics and GridGain simplify high-performance computing, lower the barrier of adoption of map/reduce paradigm for thousands of customers and ultimately enable a better scalability of a wide range of applications, networks and web sites.

I am personally looking forward to a lasting relationship between our companies that benefits GridGain users and the entire open source high-performance computing community.

Binary Calculator Project - What is the footprint of my GigaSpaces Entries?

2008-04-03T06:57:00.000-07:00

In the initial stages of every Data Grid project it is always essential to get good estimates of memory requirements. How much memory will my domain objects converted to Entries consume and what will be the indexing overhead? The answer to this question defines what JVM heap size to choose and how many of JVMs will be needed to store the intended dataset - or determine how much data can be stored within a given hardware footprint.

For GigaSpaces, it is hard to provide accurate theoretical estimates of your Entry size. Here are the reasons:

Space doesn't store entries as heap objects - they are stored decomposed to fields
String uid is generated and stored along with each entry
Index overhead is dependent on type of index (ordered or unordered ) and on a field dataset cardinality.

For these reasons it is far more practical to take an experimental approach in measuring entry footprint than trying to apply a theoretical formula. The goal of the Binary Calculator project is to build a convenient toolkit for this problem. The main idea is quite simple:

First, collect the basic statistics on the efficiency of storage of the individual entities:

Connect to remote space
Get the batch of tested entries from some entry source
Write a batch to remote space
Perform remote garbage collecting
Measure memory usage
Repeat step 2

After a sufficient number of iterations is done, we will get a number of data points in the format (entriesWritten, memoryUsage). Performing a linear approximation (e.g. min square linear fit) we receive the approximation of single entry footprint (including index overhead).

Implementing this idea, we have built an initial version of the Binary Calculator, which can be used as a toolkit for measuring arbitrary entry footprint. It has a very simple GUI that shows progress of the memory experiment.

We are planning to turn this simple toolkit into a much more powerful tool, which will generate entries on the fly, based on user-supplied meta data. This way, the user can specify an Entry Description as a simple table in a GUI:

Type	Indexed	Number of fields	Avg Length
Long	Yes	1	N/A
String	No	3	1000
String	Yes	2	5000
Integer	No	3	N/A

BinaryCalculator will generate Entries at runtime based on this description, populate it with random data, perform memory experiments and show estimated entry size.

Also, we are planning to build a lightweight plugin system to supply a custom EntrySource,
for example, your own random entry generator or JDBC or Hibernate data source. Consequently, performing full fledged capacity experiments loading real data from the database will be much easier.

We hope that this tool will be quite useful for GigaSpaces implementors in the field.

Introducing the PackRat Project

2008-04-01T07:12:00.000-07:00

Remember the legendary Fallout 2 game? One of its cool features was perks. You could increase the number of action points, strength, critical strike percentage, etc. One of the most useful perks for those who wanted to carry as much as possible was the perk called Pack Rat.

But how does it relate to Data Grids? The answer is very simple: it is possible to be more efficient at storing the data in the data grid by "packing" the objects. That's the goal of the PackRat project, released yesterday on OpenSpaces.org. The initial idea came out of a customer project with TuneWiki, a lyrics distribution site developed with GigaSpaces XAP 6.0 where it was important to pack as many lyrics as possible in a space without noticeable loss of search and retrieval performance.

Let's say we want to develop a hypothetical lyrics distribution service. First, we need a simple mechanism that packs together all entities related to a single lyric:

import org.openspaces.packrat.annotations.*

@CompressedClass(type = "entry")
public class Lyrics {
  @IndexField
  private Integer id;
  @IndexField
  private String title;
  @IndexField
  private String artist;
  @CompressedField
  private Integer version;
  @CompressedField
  private String language;
  @CompressedField
  private String comment;
  @CompressedField
  private String album;
  @CompressedField
  private String year;
  @CompressedField
  private String genre;
  @CompressedField
  private String lyrics;

  // other stuff
}

Let's say a song can be searched by artist and title (kind of a composite key). This means that all other non-searchable fields (like our lyrics field) can be stored in compressed form in a binary field and unpacked on demand. The definition for such entry looks like this:

public class LyricsPacked implement Entry{
  private Integer id;
  private String title;
  private String artist;
  private byte[] binary;

  // other stuff
}

To pack and unpack entries we need to create an instance of Packer. This class implements Entry interface and has a method pack, which packs a given object. This is how you may write a packed entry to a space:

Packer packer = new Packer();
IJSpace space = (IJSpace) SpaceFinder.find("jini://*/*/mySpace");

Lyrics lyric = new Lyrics();
// assign necessary fields

space.write(packer.pack(lyric), null, Lease.FOREVER);

At some point in time, you'll need to search the lyrics that are stored packed in a space. This code snippet shows how you'd do it using
Packer.packForTemplate(Object template), which packs the template before searching for it:

Lyrics templateLyrics = new Lyrics();
Entry[] entries = space.readMultiple(packer.packForTemplate(templateLyrics), null, 1000);

for (Entry entry:entries) {
  Lyrics lyrics = packer.unpack(entry);
  // process unpacked lyrics
}

For more examples, check the project documentation page.

Update: PackRat was submitted to the OpenSpaces Developer Challenge. It has won the Early Bird Draw.

April Fool's Day - Big OpenSpaces.org Release Date

2008-03-31T20:43:00.000-07:00

Late last year, our friends at GigaSpaces launched an open source community OpenSpaces.org with the following mission (quote from the community web site):

"OpenSpaces.org is a website sponsored by GigaSpaces Technologies and dedicated to creating and serving a community that develops and shares open source software for the OpenSpaces development framework."

For a company like ours this was a very good news. As an independent consulting firm with deep expertize in grid technologies and a whole practice dedicated to the popular GigaSpaces XAP technology, we constantly create tools, utilities and frameworks that make GigaSpaces product better and easier to use for our customers. Once the tool is created, we want to give it to the hands of all GigaSpaces users, and ideally let the users help us - and everyone else - to refine and improve the tool. OpenSpaces.org is a perfect venue for such projects.

Let's take an example - project Gigapult. If you ever had to write a bunch of scripts that start up a cluster on your Windows-based development environment, then re-write them to work on a corporate Linux-based testing lab and then do this again for a differently configured production environment, you know how frustrating and counter-productive this is. We had to do something like this last summer for a customer where the production environment wasn't even known to the developers who were creating the deployment scripts. This script-porting business led to hard-to-find bugs and delays in the production roll out.

Kirill Ishanov, the project lead, finally had enough and suggested an idea: let's make Gigaspaces configuration and bootstrapping work cross-platform, and in the process simplify the API and make the configuration files more maintainable and readable. After all, if everything else executes in the JVM, why should deployment logic be any different? Kirill explains his motivation in more details in his two blog postings, Project Gigapult and GigaSpaces Cluster Bootstrapping and Automated Testing, but this is basically how project Gigapult was born.

Today Gigapult is a JRuby-based domain-specific language designed to specify all aspects of grid configurations in a simple and intuitive way. For more information and to download the code, please, visit the project's homepage. We've used Gigapult internally on several projects now, and the technology does make life easier.

Now, along comes OpenSpaces.org and offers Gigapult the vehicle to release the technology into the hands of developers who need it most and can contribute to its enhancements going forward. Since the launch of OpenSpaces.org, Grid Dynamics opened three more active projects:

Convergence: a pluggable architecture for interoperability between computational grids and in-memory data grids capable of data-aware scheduling, with initial bindings for GigaSpaces XAP and Data Synapse GridServer

PackRat: a library that helps increase the space capacity by efficiently packing a non-indexed part of entities in binary format

OpenSpaces Binary Calculator: a tool to accurately estimate memory size required to store objects in cache

At the launch of OpenSpaces.org, GigaSpaces announced the OpenSpaces Developer Challenge. Naturally, our four projects entered the competition. And since the challenge ends today, tonight was the first beta release for all four! Nothing like a little competition to encourage the respect for deadlines... Despite today being April 1st, this is no joking matter. We want to win!

On more serious note, over the next few days I expect project leads to blog in more details about their respective projects. Meanwhile, please visit the projects, download the technology and give us the feedback - the good, the bad and the ugly so that we can make these tools better, for you and everyone else.

And please cheer for us as we await the results of the competition!

Notes from MS Developer Conference for Financial Services

2008-03-16T23:43:00.000-07:00

Last week a few folks from our company attended MS Developer Conference for Financial Services in Manhattan. The event was targeting Wall Street developers that came to learn what Microsoft is doing in the area of HPC. It is exciting to see Microsoft get serious about performance and scale-out computing and have a broad plan to address the needs of this already vast and rapidly expanding space.

I will not get into the details of technologies being showcased in this blog. Rather, I'd like to talk a little bit - selfishly - about what we were doing there, and then share one discovery - a grid company we didn't know about that has an interesting data caching solution for .NET community.

1. Excel-on-the-Grid

We were showing a demo of Excel application acting as a client for a grid-based processing. This is a powerful concept that is gaining wide spread on Wall Street. Excel is traditionally used in financial applications for heavy-duty data aggregation, analysis and visualization, all of which requires local data and processing on the desktop. This approach has serious limitations:

- If you wanted realtime data embedded in Excel app, how'd you deliver that data to the spreadsheet?
- If you wanted really computationally intensive spreadsheets, where'd you take the resources?
- If you wanted to share your application with others, how'd you do that?

Turns out, grid computing offers an elegant answer:

Step 1: Move the data from spreadsheet to Data Grid
Step 2: Move the computations from spreadsheet to Compute Grid
Step 3: Instrument Excel to both push the computations to the Grid and pull the results off the Grid as appropriate.
Step 4: Share the "dumb client" easily, it's now light-weight

Many grid vendors - and their Excel customers - are recognizing the business benefits and building solutions that makes Excel-on-the-Grid work better, ideally someday out-of-the-box. Amongst such vendors are Microsoft, GigaSpaces, Data Synapse, and others. We at Grid Dynamics were asked to develop a demo that showed Microsoft Windows Compute Cluster Server 2008 as a computational engine behind Excel-on-the-Grid, backed by GigaSpaces XAP 6.0 as in-memory data grid.

The demo works as follows:

1. Excel sends the job to MS WCCS
2. The job is directed to & executed on the server that runs the right local data cache by XAP
3. Excel gets a notification when the job is finished and displays the results

The main points of the demo are three-fold:

a. Excel-on-the-Grid is COOL
b. One can mix-and-match various compute and data grid solutions to achieve business objectives
c. Data-aware job scheduling (which is the ability of job scheduler to co-locate job execution local to the cached data) gives 2x - 3x performance boost to data-hungry jobs. What's especially important to point out is that the job itself is a "black box" to the grid - no changes are done to the algorithm itself, it may be considered a legacy code for all intents and purposes.

2. ScaleOut Software

An interesting discovery of the conference for our team had been a vendor, ScaleOut Software, who has a pure .NET implementation of data grid with seemingly good architecture and nice features.

Traditionally, Java community had been way ahead of the game on Data Grids. GigaSpaces, Tangosol and GemStone - the three big mature commercial players in data grid space - are all Java-based technologies. Yet the applications that require data grid access are a good mix between Java and .NET these days (there is always some C/C++ and other varieties, of course). While these Java technologies are supporting .NET clients in various ways, it makes sense for pureplay .NET caching and data grid solutions to be available on the market as well.

We weren't aware of such solutions when we ran into ScaleOut Software last week. They offer replicated data grid with a lot of nice features. They don't have two big ticket items that leading Java caching vendors do: data partitioning and strong out-of-the-box DB persistence. Still, for application session data, and many other stateful application needs in Windows environments, I can see ScaleOut providing a good solution for developers. We haven't really had a chance to do anything with ScaleOut software yet, but I hope to try it out soon.

DataGridFS - let your legacy code store data into IMDG

2008-03-10T04:12:00.000-07:00

The one of the most discussed topics on grid computing here in Grid Dynamics is bringing Computational Grids and In Memory Data Grids together. And DataGridFS concept was born in such discussions.

First of all, let me describe where it applies most efficiently. There are a lot of systems, which are built around Computational Grid where jobs produce files as result of computations. These files stored to NFS, so they are accessible locally after mounting on every node and client. Sun Grid Engine and Platform LSF are examples of such Computational Grids. Imagine that such system needs enhancement and such enhancement is based on bringing IMDG in this system. So, the codebase will consist of legacy code which saves results to files and modern code which uses IMDG API to get initial values and store results too. More over, these two parts should communicate with each other somehow. For example, results calculated by legacy code are initial values for new codebase jobs. It is clear, that we need the way to make new jobs able to process results stored in files on file system. There are several ways to achieve that. One is to build daemon process which scans directory on filesystem with results, parse them and stores data into IMDG using it's provided API. This approach has few cons. At least you should be able that this process is always alive and so on. The second way which is to use DataGridFS.

DataGridFS is filesystem which stores files transparently in IMDG itself. So, this filesystem inherits most features of used IMDG. For example, if IMDG allows partitioning then filesystem automatically becomes distributed and so on. So, when job process writes file to filesystem object (or set of objects) appears in IMDG. This object can be accessed using regular IMDG API or DataGridFS API which just "wrappers" for IMDG API calls. "What is about file content?", you may ask, "Is it still to be parsed in object model?" Yes, but this can be done by workers inside IMDG, which is more convenient and flexible way than separate process.

I've managed to code very simple prototype to illustrate idea. It is just read-only filesystem built using FUSE and FUSE-J. File data stored with GigaSpaces XAP. On the screenshot you can see space content via Space Browser, and the same content via UNIX command line utils.

(Click to enlarge)

GigaSpaces Cluster Bootstrapping and Automated Testing

2008-03-03T01:15:00.000-08:00

In projects which use GigaSpaces XAP it's often needed to have a set of functional tests which verify the interaction of your code and GigaSpaces cluster. For example, if an application makes initial data loading it's good to have a test that verifies that the client's code really found a space and loaded all necessary data.

To implement such tests we need to bootstrap a cluster with an appropriate configuration and to run a set of tests for this cluster. It can be achieved by following the next steps:

Use a testing framework which allows to run some set up code before running group of tests (looks like TestNG is the most appropriate choice for it cause it allows to group tests and has a sexy @BeforeGroups annotation)

Put all the code which starts a cluster to the set up method (kinda of prerequisites for tests)

Implement a functional test which uses the standard assertion mechanism to verify the cluster's state

This approach looks very useful, but the 2nd point is very tricky cause GigaSpaces (5.* and 6.0*) doesn't provide a mechanism, which will allow to start a cluster from code without black magic and dances with tambourine.

Here is a tricky workaround (which we've used on a Convergence project for testing purposes). It assumes the usage of Maven as a build tool (sorry Maven folks, I know that it's project management framework, but this definition sounds too mysterious :) ) and the project is a multi-module.

The idea behind it is very simple: to perform a cluster bootstrapping via Maven plugin before testing phase. I haven't found the appropriate plugin in Maven's repository so I wrote my own using Gigapult (check the usage and installation instructions on OpenSpaces.org).

Such approach allows to write the functional tests using a testing framework of your choice without putting cluster configuration and bootstrapping code into the setup methods. Moreover, it simplifies the creation of different build configurations on Continuous Integration server, cause it turns the configuration to the simple enumeration of Maven goals.

But putting the cluster bootstrapping to the build lifecycle and a functional tests, which can take much more time, then an impatient developer can wait, leads to the skipping of these phases. To avoid it, all time-consuming tests can be put to the separate module and run on demand using Maven's profiles. To create such configuration add a submodule to the Maven project (it can be done using the Maven Archetype Plugin) and adding the following section to the parent pom.xml:

<profiles>
  <profile>
    <id>functional-tests</id>
    <modules>
      <module>functional-tests</module>
    </modules>
  </profile>
</profiles>

After that, Maven will add the sub-module called "functional-tests" only if -Pfunctinal-tests was specified. Assuming that CI server supports Maven now we can specify goals for two build configurations:

clean install - for running all unit test and packaging

gigapult:gsm gigapult:gsc gigapult:pudeploy install -Pfunctional-tests - for running all unit tests, packaging, bootstrapping cluster and running functional tests (gigapult goals depend on cluster configuration)

Mathematics of Grid Computing - 2nd Installment

2008-02-27T02:39:00.000-08:00

I've been thinking some more about Nikita's equations, and it occurred to me to expand the system in the following way:

EnterpriseGrid.Next = HPC.Next + EnterpriseApp.Next + WebApp.Next
HPC.Next = ComputeGrid + DataGrid + GlobalCheckpointing + Data-Aware Scheduling
EnterpriseApp.Next = Scalable Stateful Services + Virtualized Resources
WebApp.Next = Scalable Stateful & Stateless Services + Virtualized Resources
Scalable Stateful Services = SOA + Data-Aware Routing + Right-Sizable DataGrid

Already exist on the market: concepts-in-black
ComputeGrid
DataGrid
Virtualization
SOA
Ability to scale-out stateless services

The gap:concepts-in-red
GlobalCheckpointing: theoretically well understood, commercial implementations are too few
Data-Aware Routing: supported natively by DataGrid vendors, but not traditional routing infrastructure
Data-Aware Scheduling: theoretically well-understood, commercial implementation are too few
Right-Sizable DataGrids: active area of R&D for many vendors

These "concepts-in-red" should be of prime importance to the community and the focus of active research, development and commercial implementations.

Mathematics of Grid Computing

2008-02-19T02:46:00.000-08:00

Today, I came across the recent blog entry by Nikita Ivanov of GridGain: Scale Out on Grid = Data Partitioning + Affinity Map/Reduce. Interesting read that inspired comments I've posted under that blog, copied here:

So, Nikita, you propose two equations:

(1) Grid Computing = Compute Grid + Data Grid
(2) Scale out on grid = Data Partition + Affinity MapReduce

Both are interesting and important statements that should be carefully analyzed. The first equation basically comes down to data-aware scheduling which (as you eloquently explain in your last week’s blog) means that the algorithm that assigns jobs to specific compute resources must take into account the distribution of data over the grid and affinity property between the job and the data partitions.

Reasonable data-aware schedulers are still rare and I am very glad to see GridGains coming up with a commercial implementation. We recently build a demo that measures the performance of a “typical” job with and without data affinity. The performance is affected by the factor of 2x to 3x, simply based on data-aware routing being switched on and off. Clearly, this is very important direction for grid middleware; I am convinced that data-aware, affinity-capable grid middleware will someday become mainstream.
Let us not forget that all this concerns job-centric processing. For throughput computing that operates under a shower of real-time transactions, the equivalent concept to “data-aware scheduling” of jobs would be “data-aware routing” of these transactions. Mainstream Data Grid middleware, like GigaSpaces and Oracle Coherence, have long been able to handle this scenario. Nowadays, GigaSpaces is moving towards support of “data-aware scheduling” through the concept of Processing Unit on the Service Grid.

Now, your second equation raises a question. Are you talking about scaling out the data grid in a static or dynamics sort of way? In other words, is the objective to allow for “a-priori” arbitrary large number of partitions with scalable MapReduce or to be able to adjust the number of partitions dynamically in response to the sporadic jumps in the payloads across the entire grid fabrics? If it’s the former, then data partitioning with affinity is the traditional answer. If it’s the latter, well, then we need to solve a lot of hard problems for statefull, data-aware services that are outside of your equation. I see dynamic scaling of statefull services as being increasingly important area of research and commercial implementations. Sun’s project Hedeby is an interesting step in this direction.

Data aware routing DataSynapse GridServer and GigaSpaces XAP

2008-01-25T01:06:00.000-08:00

Let's consider a very simple data-intensive trading application that will illustrate the following discussion. Our trade application works with a large dataset of Trade objects {id, bookId, xmlData}. A unit of work in this application is to evaluate all trades for a given bookId. For this evaluation, the application code should fetch all the trades by its bookId from the central data source and then perform some calculations with them. Sounds very simple, right?

But what if we want to scale this application to huge numbers of trades in the dataset and ensure high throughput of evaluation jobs? Well, we can put our application on the computational grid, spreading our units of work among a large number of computational engines. Systems like DataSynapse GridServer allow us to easily scale computation-intensive jobs on the grid, effectively putting the power of hundreds of its engines at our disposal. This solves our problem with raw CPU power to perform calculation over the fetched dataset, but doesn't solve the problem of remote data access, which will finally limit the throughput of our solution.

No matter how many engines we add to the processing, they all will just wait for data to arrive from the database layer. A mainstream database is bound by the need to search its disk to find the relevant data and time to move a large result set through the pipes. To exit from this dead end we need to distribute the duties of the central database on the grid and minimize expensive disk I/O. So, In-Memory Data Grids (IMDG), such as GigaSpaces XAP or Oracle Coherence, come to help. They both provide partitioned, replicated, transactional, persistent application memory. We can manage really big datasets in a really timely fashion with those products. Comparing those excellent products is out the scope of this posting, so let's pick GigaSpaces for the sake of example.

We can load all our trade objects into the partitioned clustered space of GigaSpaces XAP and thus distribute data layer workload between a number of hosts. Our book evaluation task on the DS Engine node will use the GigaSpaces API to fetch all the trades by bookId from the clustered space. Thus we will get a performance gain by removing the data access bottleneck caused by a single-host disk I/O-based database. It's good, but can we do better? Read further ...

If trades are randomly partitioned on the cluster, getting all trades by bookId will require the GigaSpaces API to contact EVERY cluster partition in parallel to fetch all relevant trades. This adds synchronization overhead and eats network bandwidth. What we can do remove this overhead? We can introduce data affinity by ensuring that GigaSpaces routes trades to partitions according to their bookId. This way, all trades with same bookId will be collocated within one partition. The GigaSpaces API will explore data affinity and will contact only ONE node within the cluster to fetch all the relevant trades at once. This significantly improves the scalability of our solution and saves cluster bandwidth. That's cool, but can we do even better? Yes, we can.

With data affinity, our book evaluation task runs on DS engines and, using the GigaSpaces API, contacts a single node on the network to fetch many trades. This bulk network fetching consumes network bandwidth and may limit our throughput if we do many such fetches at once. However, we can place the IMDG cluster on the same nodes as the computational grid and employ data aware routing, e.g., ensure that tasks requiring access to particular data will be scheduled to the node that runs the IMDG partition with /relevant/ data. In our example we need to make sure that the task to evaluate a particular bookId will be scheduled on the node running the GigaSpaces partition that contains all the trades with that bookId. Data aware scheduling makes data fetching operations local in the sense that data is fetched via loopback, not going through the network switches. This is not only faster, but also saves cluster network bandwidth.

Data aware routing demo

We have put together a simple demo that illustrates the value of data aware routing and the performance gains it offers. The demo is based on the trading application example we just considered and runs a constant flow of book evaluation jobs.

100K trade objects are loaded into a partitioned data grid. Those trades belong to 10 different equal-size books, e.g., each book has 10K trades. The job here is to evaluate all 10 books on the DataSynapse GridServer cluster in parallel. This means that a job consists of 10 tasks and each task should fetch 10K trades from the clustered space and perform some calculation over them. For simplicity, we just sum the ids of the trades and return the result.

The demo runs a constant flow of such jobs and constantly measures job completion latency. To illustrate data aware routing advantages we introduced 3 different task scheduling modes. “Data aware” mode is a mode where we perform data aware routing, ensuring local space access for the engine. “Neutral” mode is a mode where we do unguided DS scheduling as GridServer sees fit. “Anti data aware” mode is a mode where we deliberately violate data awareness and ensure network space access. The user can change the task scheduling mode on the fly and see the impact of the scheduling mode on performance.

The chart on the right side of the demo screen shows latencies of recently submitted jobs. Bars marked with "+" show the latency of jobs submitted in data aware mode, bars marked with "*" show neutral mode, and "-" show anti data aware mode. So, for these kind of jobs we can see 3x performance gain when using data aware routing over anti data aware routing. Neutral routing performance is unstable due to the fact that some tasks can go to the right node due to random scheduling. The probability of that, however, gets negligible when the number of nodes grows.

This 3x performance gain is in good correspondence with standard remote vs local space performance benchmark from GigaSpaces site.

What's under the hood?

At the heart of the data aware routing implementation there are 2 components: Monitor and DataAwareService.

Monitor server

Monitor performs continuous monitoring of the GigaSpaces XAP cluster and gathers information about clustered spaces that are deployed on the grid. Monitor “knows” on which host each GS Partition is located, e.g., knows the cluster topology.

DataAwareService

DataAwareService is a thin client-side wrapper over the DataSynapse Service with a similar interface. DataAwareService is responsible for providing the correct GridServer Condition, which helps to route tasks to the proper Engines. It uses Monitor to get the host IP address running the relevant space partition. Client code just supplies space name and routing key on service invocation and the task gets routed to the proper Engine.

Below is a diagram of the individual task routing workflow:

Conclusion
Data aware routing offers a good way to increase performance in a computational grid/IMDG combined environment. The routing scenario described here can be applied not only to a GigaSpaces + DataSynapse combination, but basically to any IMDG that supports data affinity (like Coherence) and to any grid engine that supports conditional routing (like SGE, LSF, etc.)

We are going to put our demo on the web, so it will be available online soon.
Stay tuned!

References:

Data awareness and Low Latency on The EG by Nati Shalom, CTO at GigaSpaces

Bridging The Paradigms: Convergence of Compute Grids with In-Memory Data Grids by Victoria Livschitz, CEO at GridDynamics

Bridging The Paradigms: Convergence of Compute Grids with In-Memory Data Grids

2007-12-05T03:31:00.000-08:00

Victoria Livschitz, Founder and CEO of GridDynamics Consulting,
gave a presentation on bridging together two most successful cluster paradigms: ComputeGrids and In-Memory Data Grids on IGT 2007 Conference in Israel.

As compute grids are becoming more wide spread in commercial data centers, the bottlenecks in application performance move from raw processing to searching, storing and retrieving the data. In-Memory Data Grid (IMDG) technology solve this fundamental problem by acting as super-efficient application accelerator, taking advantage of unused resources readily available on the grid – disk, memory, IO – to put the data in memory of the same computer that performs the calculations. The talk will explore how IMDG can be easily integrated with existing enterprise grids to create data-aware grid applications and provide application performance acceleration while improving application scalability and reliability.

You can download presentation here:

Bridging The Paradigms: Convergence of Compute Grids with In-Memory Data Grids

Enjoy!

Gigapult project

2007-11-30T03:22:00.000-08:00

Update
I've finally moved Gigapult project to the OpenSpaces, so now it can be found here

Gigaspaces products (such as GigaSpaces eXtreme Application Platform) are shipped with a set of scripts for running them under different platforms. These scripts are good enough for running containers and services with the default configuration, but if you need more advanced control over bootstrapping process, you'll have to spend some time setting up different configuration parameters. These parameters can be a part of space url, special environment variables, Java VM options, etc.

The standard way to override these parameters or provide some additional ones is to create some additional scripts with all needed stuff. For example, this chunk of code was taken from examples of replicated data grid.

Let's see the disadvantages of this approach.

The bin directory in GigaSpaces distribution contains a set of system-specific scripts (shell scripts for Unix and batch scripts for Windows). These scripts provide the same functionality, so if you need to create an application and then support both operating systems, you'll have to write two different versions of scripts: one for the Unix system - the other for Windows. As a result, when configuration logic becomes more complex, the support and maintenance of both scripts' versions becomes a hell, and the administrator of the end system feels like this guy from Eiffel Tower maintenance squad.

Another problem touches shell scripts only. Shell scripting language is pretty old and on different clons of Unix operating systems there are different dialects of it. On most of the linux distributives there is a bash pre-installed, on Sun Solaris there is a ksh, etc. To work properly with all these dialects shell scripts should be written in pure old shell, and it's a pain, cause it has lack of language tools to manage some advanced tasks. The fantom menace here is that on different platforms some tricky shell functions work differently. For example, toNative function from gs.sh (which is written with bash) produces different results on bash under the cygwin and linux and the ksh under the solaris. So, to test compatibility of these scripts on different platforms you'll need to test them manually on every platform. Of course, there are special parameters for each shell to check syntax of the script, but it doesn't validate the semantics. It leads to incompatibility of scripts. And because of incompatibility it is hard to migrate with the same configuration to different platforms, cause it can lead to some surprising bugs.

The third major problem with GigaSpaces configuration process is that there are different ways to configure things. «Where should I place this value?», «In what xml file should I modify the value of this attribute?», etc. There is no unified way to configure all these. By the way, all of the configuration information can be represented as a set of key-value pairs, why do I need to modify them in different places? It looks like the configuration zoo, where different animals from different ecosystems live together on a small area.

Yet another problem is that there are no any configuration validation tools, which will say that all properties were set up correctly, there is no typos, incompatible types of values (for example, when the value of the property should be a decimal number but the string value was provided). Sure, there are schema validation tools for XML files, but what about space url? The only way to get the error message is to run the configuration and look throw the log files with java stack trace messages to find out, where the error appeared. The scripts are silent and cannot notify about a mistake before this mistake causes problems.

The last problem may sound like a caprice, but shell and batch are almost unreadable. As a developer, I just want to configure my cluster, but not to deal with the reverse engineering instead of concentrating on the domain.I had the feeling that I'm reading hieroglyphs when I saw shell scripts for the first time.

That's why the Gigapult project appeared.

The main goal of it is to simplify the GigaSpaces' configuration and bootstrapping process and make configuration files more maintainable and readable.

To do it, we should use the Force - the JDK force :) GigaSpaces is written in Java, so the first requirement for running it is the JDK installed. The JDK provides a great cross-platform API for building the applications so we can delegate system-specific tasks like choosing the correct classpath separator char. So, JDK can become such Force. As a result, we don't need to implement 2 versions of scripts. But Java itself is the compiling language, so it is not very handy for scripting purposes. As the real Jedys we need something interpreted. Fortunately, there is a nice solution: there are plenty of dynamic interpreted languages written in 100% pure Java. For example, Jython - an implementation of Python programming language, JRuby - an implementation of Ruby programming language and Groovy - very young but ambitious programming language with some interesting features.

OK, we can simply rewrite existing scripts, but what about simplicity, readability and intelligibility? Of course, the python or ruby code is more readable and clear, but it is still the code written on general-purpose programming language with it's own syntax and semantics. When we're exporting a LOOKUPGROUPS variable we think about assigning the value to the VM option, not about the configuring cluster. But we don't want to override the value of the variable, we want to configure the container and the space. The difference is almost imperceptible for the programmers, but for the fellow from support it makes sense.

There is a Sapir-Whorf hypothesis which describes this idea. Following this hypothesis we need to create a language to describe the configuration. Such language is called Domain Specific Language (or DSL) and it has been a hot topic in a software development for the recent years. There is a great presentation on this topic from ThoughtWorks and their chief scientist Martin Fowler.

So the sample configuration file can be found here.

The project now in early beta phase and will be available as open-source project (will be distributed under Apache 2.0 License).

GigaSpaces Maven Archetype

2007-11-21T04:04:00.000-08:00

As a consulting company focused on grid solutions we often create, modify and build small and big projects based on different grid middleware technologies.
One of most popular technologies we use is GigaSpaces.
In day-to-day work it is often needed to quickly create IDE project with well-defined structure and all required GigaSpaces libs configured. Also it's often a case that different versions of GigaSpaces platform should be used for testing purposes. This can quickly become boring and annoying process. Here I will describe how to automate it using Maven 2 build tool.

There is a good Maven 2 feature which allow to create your own structure of project and use it in the beginning of project. This is a Maven archetype.

The simplest project that works with GigaSpaces require only 3 jars from GigaSpaces XAP product: JSpaces.jar, jsk-lib.jar and jsk-platform.jar. What we usually do to start project? Run the IDE, create new project, attach 3 libraries to project, write the first class with main method. Boring!

To ease our life, we developed a GigaSpaces Maven archetype which you can download here and use right away.

Steps to install gigaspaces-simple-archetype:

Download gigaspaces-simple-archetype-6.0.2.zip.
Unpack archive.
Run "mvn install" command from the folder with pom.xml file.
Check that installation succeeded (you will have "BUILD SUCCESSFUL" message on console). Now simple GigaSpaces archetype is in your local Maven repository.

Archetype which you just installed allows to create simple project structure for GigaSpaces XAP 6.0.2 (build 2002).

The only thing you need to create GigaSpaces project is to run the following command (you can also find create-project script inside bin directory in downloaded archive):

mvn archetype:create -DarchetypeGroupId=com.gigaspaces
-DarchetypeArtifactId=gigaspaces-simple-archetype -DarchetypeVersion=6.0.2
-DgroupId=com.yourcompanyhere -DartifactId=your-project-name-here

Bingo! You have the GigaSpaces project. You could modify its properties in pom.xml file.

To compile or package the project use "mvn compile" and "mvn package"
commands correspondingly.

I'm sure you will see messages about 3 missing artifacts, it's OK. It's about our required jars from GigaSpaces, so we need to install them in local maven repository or your company dedicated maven repository.

You should download GigaSpaces XAP 6.0.2 (build 2002) distributive and install its artifacts into local maven repository. Following commands will help you:

mvn install:install-file -DgroupId=com.GigaSpaces -DartifactId=jspaces -Dversion=6.0.2-2002 -Dpackaging=jar -Dfile=JSpaces.jar

mvn install:install-file -DgroupId=com.GigaSpaces.jini -DartifactId=jsk-lib -Dversion=6.0.2-2002 -Dpackaging=jar -Dfile=jsk-lib.jar

mvn install:install-file -DgroupId=com.GigaSpaces.jini -DartifactId=jsk-platform -Dversion=6.0.2-2002 -Dpackaging=jar -Dfile=jsk-platform.jar

After that project is ready-to-use. Popular IDEs (like Eclipse, NetBeans, IDEA) natively support Maven 2 today. You could also automatically generate project for your IDE:

Eclipse: mvn eclipse:clean eclipse:eclipse
IDEA: mvn idea:clean idea:idea
or even create Ant build script: mvn ant:clean ant:ant

Enjoy the power of Maven archetypes!

References and used products:

Maven 2 (version 2.0.7)
http://maven.apache.org/

Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project's build, reporting and documentation from a central piece of information.

GigaSpaces XAP 6.0.2 (build 2002)
http://www.gigaspaces.com/

GigaSpaces eXtreme Application Platform (XAP) redefines scalability with a holistic Space-based Architecture (SBA) approach that provides a single platform for managing the data, messaging, and business logic associated with a business transaction.

Semantic grids

2007-10-31T17:08:00.000-07:00

Howdy, I'm Stan, with few quick words about the present and future of grid management.

First, why do we need grid management? Well, the answer is simple: we need it essentially due to the same case we need any kind of management. When the grid is big, you can manage it as a whole. When the grid is too big, you can delegate the management to a small set of nodes. When you have several different grids, each of those too big, you're in trouble.

Well, you're not in big trouble if all of your grids are homogeneous and are not partitioned vertically, so the management issues involve mostly scaling. But what if each of your grids serves a different purpose, managed by own subsystem, and, what's most painful, provided by different vendors with little or no integration points at all?

One of the possible solutions is to build a layer on top of these service subsystems of each grid (Yes, every problem in software engineering can be solved by introducing another level of indirection :) ). But what's important – this level can be totally different from everything that lies beneath it. This level can be built to manipulate terms like “business processes”, “patterns”, “policies”, “configuration” and that sort of high-level stuff.

But that's not new. It's natural for the level sitting on top to operate by metaphors of higher levels than the lower ones. How one could integrate this level with what's underneath?

The idea we're dealing with in present time is to augment the service layers of the different grids with a common interface. It's not clear, however, how common the interface should be? Isn't it another “perfect protocol” no one bothers to support?

Well, yes and no. It's a “perfect meta-protocol”, actually. It's came from W3C and is called RDF. I'm not going to dive into details here, there's enough information about it on the net. Those who's not familiar with it concepts can think about it as a super-XML that allows us to embed semantics in the document, not only the markup. (Yeah, I know that's not correct… technically.)

So what do we need from a service layer of a grid to include it in our “semantic layer”? Hardly anything – to format the data it can provide in the RDF way. Relational databases, REST and SOAP services, configuration files – all of that can be augmented to present RDF in mostly no time.

That semantic meta-layer becomes a grid itself. It has access to all of the information required to operate a grid, but has no idea what to do with it.

Here's when more high-level protocols come into play. The semantic cloud is about to learn what “restart” is, how a given service is “started”, and what it means for the process to be “frozen”. Level of detail, you chose it. The more detailed it is, the more powerful your semantic cloud is. Just don't allow it to attempt world domination :).

Well, that's a large set of data to input. We want our semantic cloud to become our knowledge repository, to keep formalized knowledge of everything implicit in the heads of operators. Fortunately, we don't need to put every bit of that knowledge into the cloud. We can provide it with explicit knowledge, letting it to infer implicit knowledge itself – and correct it if it's wrong.

(It may sound like the neural networks or genetic approach, but it's not. It's about description logic, actually.)

Now your semantic cloud knows about every bit of your system. It can trace every job coming in and out your cluster, has the possibility to optimize the load balancing based on the data it's receiving in realtime, can bring in its perfect copy in no time in case a failure a detected, answers your questions and poses its own :). What's next?

Well… two ways here. One way is down the stack, replacing some of the high-level service layers of the grid with its own parts. The second way is upwards, sharing some of the data with other peer clouds, some day forming The Grid, which can redefine the grid computing the same way The Internet redefined the bulletin boards.

Hope you've enjoyed this essay. I'll continue later, focusing on approaches and technologies we've selected for our semantic affairs.

Fura, open source grid computing middleware – One-day impression

2007-10-23T05:22:00.000-07:00

Recently I came across one particularly interesting grid technology:
Fura. This is an open source product of GridSystems, a company based in Spain…

I have had a chance to play with it for a day.

Conceptually, Fura is a collection of collaborating web services based on top of a WSDL/SOAP technology stack, and all its architecture follows the Web Services/SOA paradigm. This approach offers apparent benefits, such as easy standards-based integration—SOAP is well-known, and in addition, a C++/Java SDK is shipped with the product. The only obvious disadvantage of using SOAP for all communications is that a TCP+HTTP+XML+SOAP communication stack introduces unavoidable ~40 ms communication overhead, which makes the system inappropriate for extreme transaction processing scenarios. However, if the system is used for processing long jobs (from seconds to hours), SOAP communication costs may become negligible.

Installation of Fura went smoothly using its friendly text-mode installer. A GUI installer is available on platforms that support it. The only disadvantage in Fura deployment I noticed is that all Fura services currently use static IP to communicate to each other, which is not very convenient in many environments. Anyway, Fura developers are going to support DNS names in the next minor release.

Fura features a very nice and modern-looking web-based GUI, which allows you to inspect and change all needed system parameters, submit jobs and access a Virtual File System—another useful Fura feature. The Virtual File System is basically a folder hierarchy on the master machine in the Fura cluster that is exposed via webservices to other cluster machines. VFS has full ACL support and can be used as a file sharing service by job processing agents with performance similar to FTP.

Each slave host in the Fura grid runs one or several lightweight agents; each agent can do one job at a time. The number of agents defaults to the number of CPUs. Agents report predefined set of attributes, such as CPU utilization and memory consumption, to the master scheduler service, which uses this information to schedule tasks fairly. Custom attributes are supported via assignment of keywords to hosts; e.g., some of the hosts can be tagged as “fast” or “excel” to indicate an Excel installation. Then, resource requirements can be added to the job description and the scheduler will use substring matching to find an appropriate host. Unfortunately, this model doesn't support custom attributes that require numeric matching; for example, size of the temporary space or number of logged in users.

Fura has a well defined model for creating batch jobs, which offers several kinds of iterators that can iterate over indexes and filesets, thus providing an excellent integration framework for grid-enabling legacy applications. Also, with application packages stored on VFS, Fura is able to provision required software to slave hosts before running the job. The execution subsystem offers convenient support for grabbing task output and error streams, as well as other result files, and moving it to VFS to be accessible for the job submitter.

Overall, Fura makes a good first impression. It has a well defined architecture, API and a nice web GUI. The main advantage of this system seems to be its integration capabilities, due to SOAP based architecture. Fura makes it very easy to grid-enable legacy application and looks like it is capable of supporting the computational grid needs of small-to-medium enterprises. It seems that for high-end job and transaction processing, however, the benefits of its webservices architecture can quickly turn into problems with performance and scalability.

Anyway, good job, GridSystems!