In the proposal stage, we constructed a simple Queuing model (3 nodes: CPU, Disks, Users). For the modeling cognoscenti, it was a closed multi-class model implemented in our modeling spreadsheet (not available on-line). The model suggested that at 600 users, the system had already reached its peak throughput and that it was bottle-necked on the disks. Projection showed that response times would become unacceptable at 800 users.

Our proposal was to reconfigure the software to use Amazon’s Simple Storage System instead of local disks. This would have benefits to their business case aside from performance (this is not the place for Amazon evangelism). Indeed, the revised model showed acceptable performance until about 2,400 users. The before-and-after curves are shown in the adjoining graph.

This would still be far from their goal of sizing to 60,000 users. To achieve that objective, they would need to use multiple load-balanced web servers (or another higher-powered web server).

Epilog: the last time I spoke with the founder, the growth projections had been revised due to the economy and he was reassessing the business case for his web site. Alas.

M/M/m queues

Short for Markovian arrivals, Markovian (gaussian) service times, and m resources. For each resource, we need to specify its service time and visit ratio:

• Service time: the time a particular job spends at that resource.
• Visit ratio: the number of times a resource visits that resource. For a 1-cpu-2-disks system, a database query will have a visit ratio of 1.0 for the CPU. For each of the disks, it depends. If the data is distributed equally among the disks, the visit ratios will be 0.5 each (or less, if the data is sometimes found in a cache).

Multi-class queues

An extension of the M/M/m model where all jobs (classes) are not the same. The service times and visit ratios must be specified for each class. There are a few different cases:

• Different priority disciplines to resolve which class gets a resource (first-come-first-served, processor-sharing, constant-delay <aka infinite-server>, priority).
• Open vs. closed classes, i.e., whether the number of customers of each class is fixed or a variable.

Finally, the algorithms to take load-dependent servers into account — situations where the amount of load on a server changes the service time — are also available in spreadsheet form.

1. We want the system under test to be as close to the production setup as possible.  The more different it is, the more factors you have to correct for, and the less confidence you will have in the results.  A system sized the same as production (and populated with a similar amount of data) is often a budget-buster, especially because it is used for a small part of the system life cycle and is idle for the remainder of the time.
2. If a system sized similarly to production is not used, or not populated similar to production, interpreting the results by extrapolating them from measurements is itself subject to a degree of guesswork. If you are using the results to convey bad news, you have some convincing to do. If you are using the results to convey good news, you still have some convincing to do.
3. The stress testing tools are expensive because they are designed to cover a range of technologies, from windows applications to batch scripts to web applications.
4. Since  scripts need to be written in a specialized language, writing scripts for stress testing is a specialized skill.  Writing such scripts before the application is available is even more of a challenge.
5. It is common practice to use scripts running within the company’s infrastructure to test web applications.  This makes testing more repeatable but it ignores the vissisitudes of connecting through the internet to get to the company’s servers.

There are a few encouraging trends on the horizon.  The new trends have their own limitations but at least there is hope.

• Amazon’s new offering, Elastic Compute Cloud (EC2), lets you obtain any number of servers at a moment’s notice and you pay for what you use.  You create a machine image and upload it to Amazon (I know this is not as easy as it sounds). If you need 5 servers running that image, it’s a bunch of web services and the servers are yours in 10 minutes.  You run your tests and give up the machines so you don’t pay for them.  It is finally cost effective to obtain identically-sized servers. Rightscale will help you manage your usage of EC2.
• LoadStorm and BrowserMob are two recent startups who provide the capability to create and execute scripts to stress the servers.  Both do their work external to the company’s infrastructure.  I have not had the opportunity to use their products. 

If you know of others, please let us know.

We have developed the mathematics for calculating the availability of systems. The mathematics is quite simple; the key to its usage is building a network representing the system.

The collection of the data on which to base the calculations can be a challenge. A system for collection of this data has to allow entry by members of a large organization, and consolidate that information into a query and reporting tool. The requirements for such a system have been identified. A system demonstrating an implementation of the proposed methods has been developed. We are seeking suitable partners to test the proposed methodology.

Here is the Availability Demo.  Two motivating examples:

GoodEats Catering Service takes delivery orders over the internet.  They have two system with 1 CPU, 2 Disks each and an internet connection.  You have devised a fail-over mechanism that works like this: one of the servers is designated the primary server.  This server handles the web traffic.  The database on this primary server uses log shipping to continuously send logs to the Secondary server.  When there is an outtage, the owner tries to reboot the primary server and hope that the problem goes away.  If it doesn’t, there is a manual procedure to apply the shipped logs to the Secondary server and route internet traffic to it.  The switch-back process can take a long time to accomplish.  Once the primary server has been repaired, the switch-back is done on Monday morning so it would cause minimum disruption.  The owner wants to know the availability of the system.

An Investment Advisor Company consolidates their clients’ account and position data and provides consolidated information about their portfolio via a set of reports delivered over a web site.  The information provided includes positions, trades, corporate actions and portfolio performance data.  The data presented is based on a series of feeds that arrive from the various custodians overnight.  The performance information is calculated from the data.  It is a time consuming process during which the web site is unavailable to the customers.  The challenge is to estimate the availability of the system for clients.

So log into the application.  You just need to provide your Google login ID.  Google will let you sign up right there on the page if you don’t already have one.

For the first example,

1. Create some base nodes:
   • CPU1, Disk11 and Disk12, with A-Ratings of 2.4, 3 and 3 respectively.
   • CPU2, Disk21 and Disk22, with A-Ratings of 2.4, 3 and 3 respectively.
   • Internet with A-Rating of 1.2.
2. Create 2 Aggregate nodes, Server1 and Server2, each with its CPU, disks and a (shared) internet.
3. Create a Fail-over node.  Assume that the fail-over process itself has an availability rating of 1.5 and it takes 1.0 hours to fail over and, once failed-over, takes on average 10 days (240 hrs) to switch back.

For the second example, based on a real world example, we entered this data representing the various failure modes of the application:

1. A CPU and disks that make up the server.
2. Two databases hosted on the server.  Each database is managed but of course can fail due to logs filling up, time-out conditions, errors in the application code creating an error condition in the database.
3. Timely arrival of feeds from custodians.
4. Processing of all data in the feeds.  This was a significant variable because the feeds sometimes contained back-dated events which caused the performance calculation engine to run a long time.
5. Processing of all data entered by the users the previous day.  This was a significant variable because the users sometimes entered back-dated events which caused the performance calculation engine to run a long time.
6. The web delivery infrastructure.

The results are shown in Availability Calculator Results (example 2).  SvcBatchDone is the node that represents availability of data in time for usage by the customers.  AppAvailability is the net availability of the system. Availability of 0.88 translates to 87%, which is close to what they experienced.

Feel free to utilize the system for your situation.  If you need assistance, or would like to see us enhance the application, please contact us.

Comments on this blog are closed but you may contact us through the company web site.

Thank you.

1. Nothing matches a real life system as the real life system itself! It has 100% congruence with itself.
   Of course, it would be unwise to wait till the time of install to discover any performance issues.
2. Most organizations use a performance test bed to measure the performance of a system before it is installed. Whereas the recommended best practice is to have the test bed be identical to the production machine, practical considerations can lead to one or more of these differences:
   1. Performance testing begins prior to all software defects being closed, so the code running in the performance test bed differs from what would eventually get installed.
   2. Test bed data may be different (mandated by privacy considerations).
   3. The upstream and downstream feeds of the test bed may not be configured the same way. Consider, for example, if the production system does ACH money transfers. It may not be practical to connect the test bed to ACH.
   4. The network configuration of the test bed will be different.
   5. To save cost, the hardware on which the test bed is hosted may not have the same capacity. Because of server virtualization, there may be interference from other applications running on the hardware.
   6. Synthetic workload, not real life.

   To compensate for these differences, the results obtained from performance testing often need to be scaled before any conclusions can be drawn about them.

3. A detailed quantitative model is created shortly after the analysis and technical design are completed. It is based on the technical design and key architectural characteristics of the various components. These will include purchased components as well characteristics of any technical integration that is required. An Architecture and Infrastructure Plan serves as the input to the quantitative model. The quantitative model is an approximation of the actual system in that it is a mathematical model based on empirical data about the components.
4. Even earlier in the life cycle, based on business drivers and knowledge of the architecture, a preliminary model is used to determine the key trade-offs that will be required in architecting a solution.

Assessing the options

The four types of data are compared in the accompanying bubble chart.
The size of the bubbles is proportional to the congruence between the model and the eventual system. The bubbles represent the 4 models discussed above; the x-axis represents the project time line and the y-axis represents the cost of getting results from the model.

The pre-install measurements cost about the same as the real system and yield results late — very close to the end of the project.

The model-based solutions provide a longer lead time but are less trustworthy, so their results need to be taken with a grain of salt.

There is one situation in which the model-based results can be quite reliable: when the system is in its 2^nd or 3^rd (or n^th revision). Under those circumstances, we may already have a model we trust.

Recommended Best Practice

The mathematical models should be built alongside the system during the first iteration. During subsequent iterations, they can be used to obtain an early view into system performance.

Once the model is built and validated, the model can be an adequate substitute for measurement, especially when the changes are minor. To maintain this trust, the model must be kept current and validated frequently.

The late surprises and additional costs can often be avoided through use of mathematical models.

What is an acceptable level of performance? It depends. If any UI takes longer than 0.25 seconds, it has been shown that it will cognitively interrupt the user’s thought process. Web sites don’t typically perform that fast — they can’t because of all the network delays — so our users have to put up with a level of slowness anyway.

After that very stringent threshold, it’s an expectations game. How complex does the user think the task is? If they think it is simple, they will be less tolerant of delays. If they think it is complex, more tolerant.

Around the 30 seconds mark, for web applications, you start to run into timeout values of the various components between the browser and the servers. If the operation will take longer than that, the UI should release the user from having to wait for the browser to refresh.

Scalability is a metric of how many clients you can service. The economics of the business are strongly influenced by scalability.

Scalability of a system is constrained by system bottlenecks. Every component of the system has a throughput rate — a rate it can not exceed. In a typical system, one component is operating at its maximum throughput rate and all others have a bit of slack.

If one increases the capacity of the bottleneck component, that component may no longer remain the bottleneck; the bottleneck will shift to another component.

Availability is best defined by visualizing its absence. When your system isn’t available, your business is stalled.

There are two ways to increase availability of a service: 1. reduce the down time of each component and 2. adopt a design that allows for redundant individual components that may fail but the overall system continues to function.

Dollar for dollar, systems are more robust with the second strategy.

Copyright © 2008, J Singh

Professor Spielman of Yale was recently awarded the 2008 Gödel Award.  The award was for his 2004 paper (I read a 2008 update) on the performance of the Simplex algorithm.  The algorithm has been around since 1947 and most people who went through an Engineering education in the last 50 years learned it.  The theoretical analysis suggests that in the worst case, it’s pretty bad but in practice it seems to work pretty well.  The reason it works so well, it turns out, is that the typical data it gets subjected to makes it perform reasonably well.  Alternative algorithms which have better worst-case performance don’t perform as well in practice.

The lesson from this post, and the paper, is this: You can’t study the performance of systems in a vacuum — you have to make assumptions about the data. Many a performance analysis has been invalidated by the fact that it was done with the wrong data.  We have 60 years of history to prove it!

Check it out.  It uses terminology that folks might not be familiar with, so here is a motivating example.  Say you have a small consulting company with 1 principal and 2 partners.  They could have been Servers and disk farms just as easily but a small company is easier for folks to relate to.  A bunch more assumptions:

• Every assignment involves the principal.  On an average, the assignment takes 20 days of her time.
• Every assignment also involves the partners with 50% probability.  They have different skills and the portion of assignments that go to partner A take 40 days of his time.  The portion of assignments that go to partner B take 30 days.

The question we need to ask is, how busy will the principal and the partners be when the firm is handling n assignments.  How much will clients end up waiting as the various tasks are done by the players?  Will adding a third partner to help out partner C improve the situation, etc.  It’s not my goal to list all the types of questions that could be asked.  Those have been documented elsewhere.  Rather, we want to introduce the problem so you can set up your own performance model and play with it.  If you have questions, please comment on this blog.

So log into the application.  You just need to provide your Google login ID.  They are so helpful over at Google, they will let you sign up right there on the page!

1. Create a model.  Call it something like ‘Partnership’ although ‘qwerty’ is a fine name too.
2. Click on the model name and enter stats for the 3 resources I listed above.  I know the UI is a little awkward.  I’ll improve on it.
   1. Resource 1: Service time 20, Visit Ratio 1 (meaning 100%).
   2. Resource 2: Service time 40, Visit Ratio 0.5.
   3. Resource 3: Service time 30, Visit Ratio 0.5.
3. Enter the number of customers on the right and see the results.

That’s all there is to it.  Enjoy.

In a previous blog entry, I have advocated for the urgency of getting an early view into system performance even if the conclusions are approximate.

In this entry, I want to describe a couple of successful techniques: Dimensional Analysis and Scaling. The two techniques are related but complementary. Scaling is the simpler of the techniques but Dimensional Analysis can be extremely powerful. They have been applied for analysis of physical systems in many branches of science. A particularly nice tutorial is available in this Applied Mathematics textbook. You don’t have to read the whole book — Chapter 1 is available here and that is all you need. The treatment of Dimensional Analysis still a bit dense but scaling is sufficient in many instances for the types of estimations we typically require.

A case study

Usage of the techniques is shown by example. Consider the operations of a hypothetical hedge fund company HFC. The hedge fund follows a strategy of examining the parameters of all stocks in its universe on a daily basis, making buy/sell decisions on those stocks and placing orders to the marketplace. At the end of every day, the data vendor compiles news announcements and other information from its sources, prepares files and sends them to the HFC’s computers. The expected arrival time for receipt of this daily file is 4am. The hedge fund managers expect the mathematical algorithms to complete their orders by 8:30 — that gives them 30 minutes to review the orders before the market opens.

Stocks and cash representing the assets of the fund are assumed to be kept at an external custodian — this assumption simplifies the technology infrastructure for record keeping and trade execution. To keep the example further contained, we will ignore other aspects of running the hedge fund: tracking execution at the custodian, corporate actions on the holdings, a user portal for clients to know the net worth of their portfolio, performance reporting, monthly statements, SEC compliance, handling investments and redemptions by clients. A crucial assumption we make here that would be untrue in practice and add significant performance overhead to the system should be noted: we assume that the parameters for a specific day would arrive that day and be acted upon the same day. In reality, parameters can get revised several days after, thus requiring a revisitation of the trade recommendations made several days prior.

The first step of the process is to identify all the relevant variables. In our case, the variable are: