Sometimes it’s not possible to use the JMeter Proxy to record your Thread Group.  Sometimes the client application does not support proxies or, maybe you want to replay a functional test script using Selenium Web Driver or another tool.  Whilst the the script is running you may need to record the HTTP/HTTPS traffic  to generate a Thread Group and perform tasks such as auto-insertion of Transaction Controllers.

One way to achieve these goals is to use Fiddler for the recording and then export the recorded sessions by writing a Fiddler extension.  This will give you full control over the generation of a JMeter Test Plan.

Creating a basic Fiddler export extension

In this article I will demonstrate how we can utilise Fiddler extension capabilities to export the sessions recorded by Fiddler to a JMeter .jmx file.

As your own requirements for exporting to JMeter may differ from mine I will offer only a basic example that merely creates a .jmx file containing only JMeter HTTP Sampler entries.  The ultimate aim of course is to create a complete Thread Group or, even a Test Plan.

A basic Fiddler export extension can be created very simply using Microsoft Visual Studio Professional or Visual C# Express.

1. Start Visual Studio.
2. Create a new Project of type Visual C# Class Library
3. Right-click the project’s References folder in the Solution Explorer
4. Choose the Browse tab and find Fiddler.exe in the C:\Program Files\Fiddler2 folder.
5. Click Ok to add the reference.

The main JMeterExporter class

First we must create the class that will be loaded by Fiddler when the Export Sessions option is selected.  Modify the default class1.cs (or create a new class) in your project as follows.

using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Windows.Forms;
using Fiddler;
// [1]
[assembly: Fiddler.RequiredVersion("4.4.0.0")]

namespace FiddlerExtensions
{
    /// <summary>
    /// Uses the Fiddler GUI to obtain the filename to export and writes to the
    /// Fiddler log tab, so it cannot be used with FiddlerCore.
    /// </summary>
    // [2]
    [ProfferFormat("JMeter", "JMeter .jmx Format")]
    public class JMeterExporter : ISessionExporter
    {
        public bool ExportSessions(string sFormat, Session[] oSessions, Dictionary<string, object> dictOptions,
                                    EventHandler<ProgressCallbackEventArgs> evtProgressNotifications)
        {
            bool bResult = true;
            string sFilename = null;

            // [3] Ask the Fiddler GUI to obtain the filename to export to
            sFilename = Fiddler.Utilities.ObtainSaveFilename("Export As " + sFormat, "JMeter Files (*.jmx)|*.jmx");

            if (String.IsNullOrEmpty(sFilename)) return false;

            if (!Path.HasExtension(sFilename)) sFilename = sFilename + ".jmx";

            try
            {

                Encoding encUTF8NoBOM = new UTF8Encoding(false);
                // [4]
                JMeterTestPlan jMeterTestPlan = new JMeterTestPlan(oSessions, sFilename);
                System.IO.StreamWriter sw = new StreamWriter(sFilename, false, encUTF8NoBOM);
                // [5]
                sw.Write(jMeterTestPlan.Jmx);
                sw.Close();

                Fiddler.FiddlerApplication.Log.LogString("Successfully exported sessions to JMeter Test Plan");
                Fiddler.FiddlerApplication.Log.LogString(String.Format("\t{0}", sFilename));
            }
            catch (Exception eX)
            {
                Fiddler.FiddlerApplication.Log.LogString(eX.Message);
                Fiddler.FiddlerApplication.Log.LogString(eX.StackTrace);
                bResult = false;
            }
            return bResult;
        }

        public void Dispose()
        {
        }
    }

}

[1] You must specify somewhere in your code the minimum version of Fiddler with which your code is compatible.

[2] Here is where you specify what will appear in the list of export formats when the user selects “File > Export Sessions > All Sessions…” from within Fiddler.

[3] After selecting the JMeter export option, Fiddler will prompt the user to enter a filename to which the export will be saved.

[4] Here we instantiate a class called JMeterTestPlan and pass to the constructor the output filename and the list of sessions recorded by Fiddler.

[5] Call the JMeterTestPlan to return the recorded sessions in jmx format.

The JMeterTestPlan class

The JMeter Test Plan .jmx format is XML, but as far as I can tell, it does not have a published definition.  I have therefore resorted to simply interrogating the format manually and then written code that outputs correctly formatted strings.

Create a new class called JMeterTestPlan.cs in your project as follows.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml.Serialization;
using System.Xml.Linq;
using System.IO;

namespace FiddlerExtensions
{
    /// <summary>
    /// A simple representation of a basic JMeterTestPlan
    /// </summary>
    public class JMeterTestPlan
    {
        // [1]
        private SessionList sessionList;
        private Fiddler.Session[] sessions; // A Fiddler session represents a request/response pair

        public JMeterTestPlan()
        {
            sessions = new Fiddler.Session[0];
            sessionList = new SessionList(sessions);
        }

        public JMeterTestPlan(Fiddler.Session[] oSessions, string outputFilename)
        {
            this.sessions = oSessions;
            sessionList = new SessionList(oSessions);
        }

        // [2] Return the final formatted JMX string
        public string Jmx
        {
            get
            {
                StringBuilder sb = new StringBuilder();
                sb.Append("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n");
                // [3] Linq used to prettify the XML output because it's
                // just a plain string with no formatting/indenting
                XDocument doc = XDocument.Parse(this.Xml);
                sb.Append(doc.ToString());
                return sb.ToString();
            }
        }

        // Traverse all objects and request their XML representations.
        private string Xml
        {
            get
            {
                StringBuilder sb = new StringBuilder();
                // based upon the version I was using at the time of development
                sb.Append("<jmeterTestPlan version=\"1.2\" properties=\"2.3\">");
                // [4]
                sb.Append(sessionList.Xml);
                sb.Append("</jmeterTestPlan>");
               return sb.ToString();
            }
        }
    }

    /// <summary>
    ///  [5] This class represents a list of HTTP Sampler nodes
    /// </summary>
    public class SessionList
    {
        private Fiddler.Session[] sessions;

        public SessionList()
        {
            sessions = new Fiddler.Session[0];
        }

        public SessionList(Fiddler.Session[] oSessions)
        {
            this.sessions = oSessions;
        }

        // Return the HTTP Sampler nodes
        public string Xml
        {
            get
            {
                StringBuilder sb = new StringBuilder();
                if (sessions.Length > 0)
                {
                    // [6]
                    sb.Append("<hashTree>");
                    foreach (Fiddler.Session session in sessions)
                    {
                        HTTPSamplerProxy httpSamplerProxy = new HTTPSamplerProxy(session);
                        sb.Append(httpSamplerProxy.Xml);
                    }
                    sb.Append("</hashTree>");
                }
                return sb.ToString();
            }
        }
    }

    /// <summary>
    /// This class represents a HTTP Sampler node
    /// </summary>
    public class HTTPSamplerProxy
    {
        Fiddler.Session session;

        public HTTPSamplerProxy(Fiddler.Session session)
        {
            this.session = session;
        }

        // [7] Return the HTTP Request node
        public string Xml
        {
            get
            {
                StringBuilder sb = new StringBuilder();
                sb.Append(String.Format("<HTTPSamplerProxy guiclass=\"HttpTestSampleGui\" "
                          + "testclass=\"HTTPSamplerProxy\" testname=\"{0}\" enabled=\"true\">"
                          , Path));
                sb.Append("<boolProp name=\"HTTPSampler.postBodyRaw\">true</boolProp>");
                sb.Append("<elementProp name=\"HTTPsampler.Arguments\" elementType=\"Arguments\">");
                sb.Append("<collectionProp name=\"Arguments.arguments\">");
                sb.Append("<elementProp name=\"\" elementType=\"HTTPArgument\">");
                sb.Append("<boolProp name=\"HTTPArgument.always_encode\">false</boolProp>");
                sb.Append(String.Format("<stringProp name=\"Argument.value\">{0}</stringProp>", RequestBody));
                sb.Append("<stringProp name=\"Argument.metadata\">=</stringProp>");
                sb.Append("</elementProp>");
                sb.Append("</collectionProp>");
                sb.Append("</elementProp>");
                sb.Append(String.Format("<stringProp name=\"HTTPSampler.domain\">{0}</stringProp>", session.host));
                sb.Append(String.Format("<stringProp name=\"HTTPSampler.port\">{0}</stringProp>", Port));
                sb.Append("<stringProp name=\"HTTPSampler.connect_timeout\"></stringProp>");
                sb.Append("<stringProp name=\"HTTPSampler.response_timeout\"></stringProp>");
                sb.Append(String.Format("<stringProp name=\"HTTPSampler.protocol\">{0}</stringProp>"
                          , session.oRequest.headers.UriScheme));
                sb.Append("<stringProp name=\"HTTPSampler.contentEncoding\"></stringProp>");
                sb.Append(String.Format("<stringProp name=\"HTTPSampler.path\">{0}</stringProp>", Path));
                sb.Append(String.Format("<stringProp name=\"HTTPSampler.method\">{0}</stringProp>"
                          , session.oRequest.headers.HTTPMethod.ToUpper()));
                sb.Append("<boolProp name=\"HTTPSampler.follow_redirects\">true</boolProp>");
                sb.Append("<boolProp name=\"HTTPSampler.auto_redirects\">false</boolProp>");
                sb.Append("<boolProp name=\"HTTPSampler.use_keepalive\">true</boolProp>");
                sb.Append("<boolProp name=\"HTTPSampler.DO_MULTIPART_POST\">false</boolProp>");
                sb.Append("<boolProp name=\"HTTPSampler.monitor\">false</boolProp>");
                sb.Append("<stringProp name=\"HTTPSampler.embedded_url_re\"></stringProp>");
                sb.Append("</HTTPSamplerProxy>");
                sb.Append("<hashTree/>");
                return sb.ToString();
            }
        }

        private string Path
        {
            get
            {
                return System.Net.WebUtility.HtmlEncode(session.PathAndQuery);
            }
        }

        private string getPort()
        {
            int port = session.port;
            string protocol = session.oRequest.headers.UriScheme; ;
            if (protocol.ToLower() == ("https") && port == 443)
            {
                return "";
            }
            if (protocol.ToLower() == ("http") && port == 80)
            {
                return "";
            }
            return port.ToString();
        }

        private string Port
        {
            get
            {
                return getPort();
            }
        }

        private string RequestBody
        {
            get
            {
                return System.Net.WebUtility.HtmlEncode(session.GetRequestBodyAsString());
            }
        }

    }
}

[1] The JMeterTestPlan class has a SessionList object that holds all the recorded sessions passed in by Fiddler.

[2] The Jmx method is called from the JMeterExporter [5] class.

[3] Call XDocument to prettify the xml output by JMeterTestPlan.

[4] Append the xml output from the SessionList object to the JMeterTestPlan xml.

[5] The SessionList class is a list of recorded sessions and returns constructed xml containing HTTPSamplerProxy nodes.

[6] A new indented node in JMeter jmx is defined by a <hashTree> entry.  The end of indentation is indicated with a </hashTree> entry.

[7] The xml for an individual HTTPSamplerProxy node is created here.

Building and deploying your extension

1. Close Fiddler.
2. Make sure you target AnyCPU when building your extension.
3. Build your extension in Debug mode and copy the outputs, JmeterExporter.dll and JmeterExporter.pdb, to either %ProgramFiles%\Fiddler2\ImportExport\ (available to all users) or, %USERPROFILE%\Documents\Fiddler2\ImportExport\ (available only to the current user).  Deploying the debug version with the .pdb file ensures that any error generated by your extension will report the line numbers of the offending code.
4. Start Fiddler and capture some traffic.
5. Stop capturing and select “File >> Export Session >> All Sessions…” and export to JMeter.
6. Open a blank test plan in JMeter and create a Thread Group.
7. Right-click the Thread Group and select Merge.
8. Navigate to the exported Jmx file and merge into your test plan.  The recorded requests will be merged into your test plan as child requests of the Thread Group.

Debugging your extension

As mentioned  earlier you should build your extension in Debug mode.  You must also set a number of Fiddler preferences in order to capture and log any exceptions thrown by your extension.  Using Fiddler’s QuickExec box, execute the following two commands:

prefs set fiddler.debug.extensions.showerrors True
prefs set fiddler.debug.extensions.verbose True

Any exceptions will now be logged  to Fiddler’s Log tab.

Extending the basic extension

There are many ways in which you could extend the functionality of this extension.

Export a complete Thread Group or Test Plan

This would involve viewing the jmx file of an existing test plan containing a Thread Group and then working out what xml needs generating to create a Thread Group entry.  It’s then just a matter of adding to your code to create these extra elements.  You could name the Thread Group based upon the output filename of the jmx.

Keep a skeleton JMeter Test Plan containing the basic configuration elements, listeners etc and just merge in your new or updated Thread Groups after you record them.

Assign more meaningful names to the HTTP Samplers

For instance, if the HTTP requests recorded were RESTful or Web Service calls then you could easily give them shorter names, based upon the service call, rather than the URL’s full path.

Generate Transaction Controllers for your test plan

If you are recording manually using Fiddler then you could use Fiddler’s Comment feature to mark Transactions as you record.  The comment feature allow you to add a comment to a selected session.  Your export code can detect these comments by accessing the following property of a session:

    foreach (Fiddler.Session session in sessions)
    {
        if (session.oFlags.ContainsKey("ui-comments"))
        {
            // create a Transaction Controller entry with name =
            // session.oFlags.ContainsKey("ui-comments")
        }
        // continue with processing session
    }

If you are recording a replaying functional test script e.g. Selenium Web Driver or Windows UI Automation (WPF) then you can insert dummy HTTP calls into your code that will be captured by Fiddler.  Simply ensure that your exporter code processes these dummy calls and turns them into Transaction Controllers.  Here is an example of a dummy call I use in my Windows UI Automation scripts:

public static void StartTransaction(string transaction)
{
    HttpWebRequest transactionRequest = (HttpWebRequest)WebRequest.Create("http://auto-transaction/" + transaction);
    transactionRequest.Method = "GET";
    try
    {
        HttpWebResponse transactionResponse = (HttpWebResponse)transactionRequest.GetResponse();
    }
    catch { }
}

I then process these in my exporter code:

    foreach (Fiddler.Session session in sessions)
    {
        if (session.host.StartsWith("auto-transaction"))
        {
            transactionName = session.PathAndQuery.Remove(0, 1); // remove "/" from start of path
            // create a Transaction Controller entry with name =
            // transactionName
        }
        // continue with processing session
    }

Important Notes

The hardest thing to get right when constructing the jmx file is the placement of <hashTree> tags.

In the above image the following <hashTree> tags are created:

<hashTree> <!-- indent the next node -->
<!-- Test Plan node -->
    <hashTree> <!-- indent the next node -->
    <!-- Thread Group node -->
        <hashTree> <!-- indent the next node -->
        <!-- HTTP Sampler node -->
        <hashTree/> <!-- end HTTP Sampler node but no de-dent -->
        <!-- HTTP Sampler node -->
        <hashTree/> <!-- end HTTP Sampler node but no de-dent -->
        <!-- HTTP Sampler node -->
        <hashTree/> <!-- end HTTP Sampler node but no de-dent -->
        </hashTree> <!-- de-dent the next node – end of HTTP Samplers -->
    </hashTree> <!-- de-dent the next node – end of Thread Group -->
</hashTree> <!-- de-dent the next node – end of Test Plan -->

I’m a big fan of performance reliability, and it isn’t the expensive & ‘cheap thinking’ type that is failover and resilience.  I’m talking about ‘functional performance resilience’ – the one where your system doesn’t blow up because someone somewhere has entered something silly or there has been a series of system defects that causes the system to slow or crash under load.  When I look for ways to make a system more performant (see my definition of performant here), one of the ways I look to increase reliability is to identify items that are not entirely obvious and have a low-risk threshold per item when looked at in isolation.

Removing Performance Reliability Risks – Don’t Ignore the FLAGS!!!

The risks I’m particularly seeking to address are unhandled exceptions, errors, messages telling you something you didn’t expect and warnings – I’m going to call these Flags.  I’ve been on a number of sites where system logs (including DB) are not monitored in live.  How a system behaves in live will ALWAYS be different from test.  If these Flags are not monitored, then this increases reliability risk over time.  Each of these Flags becomes a potential single point of failure (SPOF) for performance.  These Flags are very different from functional issues – as functional issues get noticed and prioritised.  Flags tend not to, they are invisible and don’t affect ‘the business’ in an obvious way, so get brushed under the carpet.

Wood Lice and Your House

A single woodlice is not a problem in your house, but if you see one you step on it and remove.  If left unchecked you become infested, the many small problems will become a much larger issue – I think of  Flags in a similar way.

An Avalanche of Flags

No monitoring of flags encourages an undisciplined approach, new releases will add to the number of Flags and before anyone realises there are a high number of Flags on system … and then its too late.  Simply looking at something and accepting it as a ‘valid error’ is common. In my book – the whole point of Flags are they are there not to be ignored – the system is behaving in a way that wasn’t expected.

Wood From Trees

This is best illustrated when something bad happens on the live environment – people take a look at the logs, realize there are so many false Flags that they can’t easily piece together the crime scene.  Its littered with other debris.  A hard job has suddenly become a lot harder.  I’ve seen companies just ‘hope’ that that thing doesn’t just happen again (it failed over) because it too time consuming to the bottom of it.  This is a bad experience for end users and its also the result of poor Flag management.     This can result in expensive & avoidable solutions such as Splunk.

Causes of poor Flag Management:

Here are what I think are the underlying causes:

Distorted Low Risk: We have a tendency to trivialize the small risks we feel we can control (e.g. Flags or functional issues) and exaggerate the large ones we feel we cannot (large influx of traffic, hardware failure).  But over time, these low risks can easily add up to a much larger risk than the infrequent.   This is perhaps the underlying cause people are most guilty of.  Over time a flag or set of flags becoming a Single Point Of Failure is inevitable.

Usability: Visibility & usability.  If all the flags are consolidated into a single place, easily accessible and easy to drill into – then people become much more proactive about them.  If they are hidden, dissipate and take time to access then who on earth is going to poke about in them unless there is an issue? … and by the time there is an issue its usually too late…..  Particularly relevant if a system has a high number of servers.

Technical Management: Of lack of.  Communicating this risk and ensuring adequate processes and tools are in place is down to a lack of technical oversight (or Technical PM).  I view this as essential infrastructure & plumbing that should be in place – If I commission a house, I expect the guttering, drainage and basic infrastructure to be built in. Not stated after the fact.

Risk Management

So an effective “Performance Reliability” strategy is one that is a disciplined and a steady slog approach. Monitoring Flags on a daily basis, investigating and introducing fixes.  Its also a virtuous approach – the system behavior is better understood and reliability and overall health of the system is increased leading to less down time, more efficient use of dev time and results in less time on unnecessary remedial solutions.  Reliability is an intrinsic part of a Performant system. Keep the woodlice at bay.   Consolidate your logs, monitor and keep your house clean.

Jason

Performant Anti-Patterns 

Monitoring Cache Performance 

Creating a Performant System

“Hmmm, how can I make a system more Performant….?” .  I was tasked recently with making a fairly large system “More Performant” and I thought –

Here are some definitions I found:  “Meeting system needs adequately”,  ”Scales well and is efficient”,   “Performs well under load” …  most seemed loose, subjective and open to interpretation, particularly when different stakeholders have different objectives.  The underbelly of most definitions seems to be “Make it quicker, we are concerned it’s going to hack off our customers”.

Defining a Performant Score:

How can I decide that a system is more Performant than another and objectively compare to another system that may be completely different?  How do we factor in the reliability of a system?  In short, how can I define and measure if a system is Performant? Here’s a first stab…

For me it’s more than the conventional definition of speed, A Performant system boils down to this:

Performant Value = function: (Responsiveness x Reliability x Usability)

A perfectly Performant system will score 1, which is impossible as there is always an element that can be improved.

Responsiveness = function: (User Speed x User Speed under Load x … other load related activities)  (Max Score 1)

Reliability = function: (Functional Defects x Uptime x resilience x failover ability x …..  ) (Max Score 1)

Usability = function: (Ability of User to complete a desired task in the shortest amount of time x Any other relevant factors)  (Max Score 1)

Note: Attribute’s are scored on a scale of 0 -> 1. A value of 1 means an attribute is impossible to improve.

Each of the main attributes will consist of different weightings and variables for different type of systems e.g. Weight & battery life may be added as attributes for Usability for hardware devices such as tablets.

The only software system that I can think of that may approach a Performant Score of 1 are Aircraft Control systems.

Here is my take on the three main attributes:

A system that scores highly only in Responsiveness will result in high user engagement.

A system that scores highly in both Responsiveness & Reliability will endear loyalty.

A system that scores highly in Responsiveness & Reliability & Usability with result in Commitment & Respect aka LOVE (yes I said that).

Here’s a thought: When MS Vista was released, it was considered ‘Performant’ by some -  Imagine if they used the above formula to measure?  I’m sure there is a high chance it would have been less performant than XP.   This to me perfectly captures why a more rigid definition is important. It creates a common language and communicates in a clear and measurable way.

The above definition of Performant looks pretty powerful, it can now be an objective measure of how we would emotionally impact an end user.  e.g. Are Apple devices more performant than MS devices?

Does this definition make sense? Lets look at this in reverse “My system is not performant” generally means the system is beginning to hack people off.  So I think I’m in the right area.   We can also use the Performant score to measure the competitions weaknesses and concentrate on making a product better in those areas.

So back to reality, when a client asks me to make a system more performant – I don’t just think about increasing speed, scalability and capacity pinch points.  I also look at reliability.  This is often overlooked – and I can have difficulty making a customer view reliability as a cost worth paying to make a system more performant. (Link here about  Reliability and Performant).  The problem with reliability is sometimes it just isn’t visible enough until its becomes a peformaant issue.

Usability is something I rarely get directly involved in (But would like to) – this is one of the most overlooked aspects of systems, I’ve witnessed many great technologies fail/lose to inferior ones because of usability aspects.  A problem solved is not effective if it isn’t easily accessible to a the target user.

Making A System More Performant

So that’s my first stab at a most robust definition of a “Performant System”.   With the above definition a Performant system can now:

1. Objectively be measured
2. Compared to rivals in a less subjective way
3. Be used as a generic measure to compare different systems
4. Can be a measure of user stickiness or commitment
5. Produce comparisons against competitors and provide weaknesses 

…and with a stronger definition of Performant I can now meaningfully answer the question “How can I make a system more Perfomant?”

Jason

You may also like:

Performant Anti-Patterns 

Monitoring Cache Performance 

Performance Testing is Hitting the Wall 

Performance By Design – an Agile Approach 

Caches are all about performance.  Performance Engineers need to have an appreciation of caches to be effective – they are used everywhere and directly have an impact on smooth running of an application.  This is one area where a live system can quickly fall over because time hasn’t been taken to understand the behaviour of caches during load testing.

I’ve been using Memcache with WordPress, its so simple it’s beautiful.  I quickly starting caching frequency requested pages that requested expensive SQL.  Its so simple its scary and I was tempted to find different ways I could use it as ‘an easy way out’ … (and therein the path to abuse lies).  WordPress is a little different in its approach – it caches to disk, not to memory.  Which actually isn’t a bad thing – not everything needs to be cached to memory, expensive op’s are OK to cache to disk, and disk is cheaper than memory.    Anyway, I drift …. 

Here’s a list of things to consider with real world caching:

Load Testing Vs Real World testing:  If the test data you use for load testing is less than the cache size, but the real world data is greater than cache size  you are always going to get faster results – because the system will never see thrashing/cache expires and dramatically lower DB activity.  Sometimes you just can’t model the breath of data a live system has against a load model – that OK, just assess this risk and document.

Cache Lemming Visibility:  Caches are live but not monitored!? You are blind and are going the way of the lemming.  How big is the cache size?  How many elements does it have? How many cache hits/misses have you got?  What are the sizes of each of the elements – are any too big? Eviction policy and rate …  If you aren’t monitoring, trending and keeping tabs its all going to go horribly wrong.

Here are some essential stats: 

Percent Used:	=(Objects)/(Size)
Effectiveness	=(Hits)/(Hits + Misses)
Objects / Size:	The number of entries in the cache / the number of total possible entries allowed (configurable).
Hit / Miss / Expiry (Counts):	Reads accessing cache where required content was found / Reads accessing cache where required content was not found / Objects evicted from the cache.

Always work out the total size of the cache – not all elements belonging to a cache are necessarily the same size.

Transient Visibility: The nature of the above stats will change over a busy and non-busy period.  So I take these states every X mins intervals and analyze in Excel.   It lets me know the behavior and also allows me to work out if cache sizes are going to need resizing (essential if you are up to your limit in memory).  Busy periods are generally the most useful (but not necessarily).

Session Caching:  Are caches heavy on user sessions.  This makes a difference if load-testing and you are attempting to cheat by upping the transaction rate (See Cheating – User or high transaction rates).  Everyone cheats (or should) – but when you do, know the risks.

Large Cache Objects: If the cache holds objects that are large in size be suspicious – is a small attribute of that object required, or is it needed in its entirety? Consider the processing/serialization overhead.  If we don’t cache hit big objects often, why cache?- This will have an impact on memory consumption and create excessive GC.  Don’t be afraid to ask questions.  Not everything is best in cache e.g. Large SQL result-sets. .

Evidence based Cache profiling: A good Software Engineer will evidence where the application is spending the majority of its time, particularly on the DB.   If these are frequent then they should be good candidates for cache.

Sharing: Does the Cache scale and share memory across servers? (Memcache can).  Do you have a dedicated cache server? A lot of implementations do not – which means there is a risk of cache duplication across servers.    

Cache Stampede: Under very high loads of concurrency caches can conspire to empty at the same time – causing a stampede effect on the DB and killing the system.

Phase Synchronization:   Highly concurrency and high usage systems have a tendency to stop being unordered and ‘phase shift’ into a pattern of behaviour – a pattern of order comes from relative chaos.  This can cause Cache stampedes and is a difficult one to identify.   A good analogy to describe this is the Millenium Bridge - people started walking in synchronisation, causing it to wobble.   Adding noise (aka Jitter) will help eliminate this.  This type of behaviour is only typical in large scale systems.  (ps I made this title up – as I couldn’t find a definition)

Papering the Cracks

I have mixed feelings about caches – Its an inevitable and required strategy, however I’ve often seen them misused by software engineers.  Its too easy to take a good idea a little too far – and then begin to lose site of the fundamental architectural problem.  “Lets put everything in cache…. It’ll make everything faster”  ….  Its almost as if the DB and tuning is then forgotten (See Costs of Encapsulation).  Whats more I’ve seen SE’s then get their knickers in a twist attempting to solve locking and contention issues (you know everything a DB does really really well) through excessive use of a cache.  So my general observation about caches is that operations that require non time critical reads (i.e. data that isn’t likely to change soon) are generally good candidates. Highly volatile time critical data isn’t.

Throwing the golden goose over the fence and then forgetting

The golden rule for effective cache performance is to monitor, analyse and actively measure, measure measure. No visibility means you are blind and likely to find trouble

Jason

You might also be interested in:

Being Proactive about Performance 

Anti-Performant Patterns 

Performance Benchmarking Part1

Managing Identifying & Assessing Performance Requirements

“Any gain in bandwidth capacity through efficiency techniques will be short lived. Increases in functionality, bad practice or abstracted inefficiency will absorb that initial saving quickly ” – me 

I was flicking through the IKEA catalogue the advert (see diagram) struck a cord with me. An  LED bulb had been created which was 85% more energy efficient than a traditional bulb. However the designer and retailer have seen fit to put 16 of them into a single housing.  I calculated that they had created a fixture which would use 140% more energy than that of a single incadensant bulb.  As we innovate and create a more efficient ways, the market appears to unconsciously find a way of utilising the spare capacity given.

I see many parallels within the IT industry.  I’ve seen chipsets get faster and the supporting software get bloated (MS for instance) – negating the increased and intended benefits.  Within the web industry we see that as the average bandwidth capability increases, the average size of web-pages increase.  Every low level improvement (SPDY, Broadband, CDN’s) seems to be met by a general tendancy to gobble up the spare capacity given. It’s almost as if there is a unconscious and self-regulating mechanism at play.

No matter how much we increase the available capacity the marketplace seems to find a way of utilising this, through bad practice or good.  I have a theory as to one reason why this may be the case and it’s one of the costs associated with encapsulation and abstraction.  The hardware, OS, software services, compiler, back, middle and presentation tier are essentially abstracted.  The people that implement (developers, web designers) are so specialised and siloed they have little regard or interest in these components until they realise the application is unacceptably slow; then improvements are made up until application performance becomes acceptable, thus filling all available capacity.

I’m going to make an attempt at a general industry Rule:

Buksh’s Web Capacity Rule:

Any gain in bandwidth capacity through efficiency techniques will be short lived, as increases in functionality, bad practice or abstracted inefficiency will absorb that initial saving quickly

Also by implication, the rule is saying that the marketplave will average a page to reach the target audience within an acceptable industry response time  (2-3 seconds)

Average Page Size = ALT * Average Bandwidth Capacity 

ALT = Average Acceptable Load Time, lets say this is 3 secs.

ABT = Average Bandwidth Capacity of target market.

This means we can probably work out the average bandwidth capacity of the world (western) by taking the average page size and dividing by 3 secs.

….looking around, the average page size is now 1024k (according to Strangeloop).  So:

Average Bandwidth Capacity of consumers is 341k == 0.3 of a MB ==2.5 Mbps  

This means if a page is to load in 1 sec, it should be approx. 0.3 MB in size.  (Of course this is a very course calculation, but it should provide a good guide)

This as a rule has some other implications.  It’s fairly safe to assume that the average bandwidth (to consumers) will continue to increase.  This means the industry is forever going to be in a continuous game of cat and mouse.  Pages will continue to gain in average size no matter what.  Client browsers will experience larger demands on their resources and continue to grow in complexity.

Dancing is better than waddling

So back to the lights – I suspect if we ever reach a point where too much performance capacity is cheaply given, then as a whole we will have a tendency to over utilise & become bloated.  Our nature and practice seems to dictate this.  Having performance and capacity restrictions ironically makes us leaner, meaner, inventive and more responsive.  I think if we get to a point where bandwidth capacity isn’t a restricting factor then this will have a tendency to make us lazy, fat and bloated with a host of unintentional and indirect consequences.  Thankfully those that are lean, dance a lot better than those that are not at present, I hope it remains that way.

Note: I’ve deliberately steered away from the delivery mechanisms that deal with the effects of latency (such as CDN’s and best practices for browser rendering), they have been covered elsewhere and are well understood. This piece will deal with the ‘inside the firewall’ anti-performant patterns.

Each of the below items should be considered summaries, they can easily be expanded into more expansive articles, but for readability they have been kept succinct:

Organic Inheritance and loss of control:  Applications are built over years, people and knowledge move on.  Code isn’t “in-line” documented, understood or owned. Temporary fixes become permanent.  No concerted effort is made to understand, refactor or reengineer old code as this ‘just works’; only the truth of the matter is no-one has the time to understand what ‘that bit does’ and they have become scared of changing it because it does something that no-one quite understands and is used by a lot of other code.

Prevention: Constant refactoring of existing code, peer reviews and strict in-line documentation and coding standards.  If there is code that isn’t understood this should be made a priority.

‘Over the fence’ to Production:  The system resides on hardware that is managed by an external vendor.  In some extreme cases I’ve seen clients think that once it’s live it’ll just work.  When it’s ‘thrown over fence’, they fail to see the need for continuous monitoring of both hardware, software, error messages and end user response times.

Prevention: Consolidated logging consoles e.g. Splunk, loglogic, graylog2  (Splunk is overpriced ) - particularly if there are a high number of web & application servers.  Active monitoring of log messages and removal of spurious error logs.  Monitoring and reporting of hardware performance counters such as CPU, I/O and heap allocation.  Instrumentation of key business flows – into CSV’s and graphed, this will vastly reduce the need for expensive APM’s or tools such as Gomez.  (See Creating Performance Reliability)

No dedicated DB expertise:  DB’s are complex, I think of them as operating systems in their own right. I’ve been surprised by how many DB’s have been constructed by developers – sometimes many developers contribute to a DB.  But without a holistic source of control and expertise the DB will become unmanageable and a source of application contention.

Prevention:   Employ a DB developer (not a DBA).. all tables changes, PL/SQL and SQL should at least be QA’ed by this role before being allowed live onto the system.  A good DB developer will constantly measure the live system performance and tune, tune, tune.

No Live Measures for Continuous Improvement (CI): The performance characteristics of a system are constantly subject to change (such as load) and variables that are in a constant state of flux (such as user profiles).  How the system behaves in test will be extremely different to live.   If you don’t measure, monitor and feedback constantly from the live environment you are going the way of the Lemming.

Prevention: Measure! If you don’t measure how can you improve? The quality and depth of your measurements are directly related to CI improvements you can feedback.

The Encapsulation Jigsaw:  Many different services are pieced together with little oversight or communication.  A loss of respect for CPU cycles and understanding of internal system behaviour within these services becomes the normal.  More detail can be found here.

Unmeasured Cache Behaviour: The system utilizes a number of caches for commonly requested objects, only no-one has any data on how these caches are actually used in the real world.  The cache contains a 1000 objects – but no-one knows the size of each of these objects, how long they live for, how many times they are requested, how many times they are thrashed.  Is the cache too big or too small? Is the TTL too high or too small?  This all creates memory overhead and inefficient use of resources such as DB’s. (See Measuring Cache Performance)

Prevention: Measure and report on the caches everyday. Understand their behavior, refine and keep in check.  Caches are a good idea that become a drag if left unchecked.

Unrealistic Performance Expectations:  Not strictly an anti-pattern but a pattern I see that becomes a significant drag to projects.  A new system (or part) is due to go live, the business expects use that is going to turn out to be wildly above expectations – resulting in a system that is over engineered, resourced & over budget.  By default the business is going to be over optimistic and think this is the best thing since sliced bread.

Prevention:  Use current usage statistics, derive a usage profile that is derived from a mathematical formula that can be scrutinized and subject to rigor by all parties.  If variables aren’t known then guess and state.  Make sure this derived figure is reviewed through walk-throughs and classroom inspections. I’ve been in many places where figures are stated and not inspected until too late.  Months of effort can be wiped off a project by simple review.  

Non technical PM’s:  Project managers that concentrate on managing up, not down – have no interest or appreciation of development practices.  Developers are then not given enough steer and are allowed to work in isolation without an awareness of the overall system.   This is the one that irks me the most – would you employ a football manager that has little understanding of how the pieces fit together and flow? Or just fills in the gaps and reports on progress (or lack of)?

Prevention:  Employ a PM that has good development experience.

I’ve found each of the above are common patterns that accumulate into a system becoming none performant. More over, the above patterns make quickly identifying the root cause of poor performance almost impossible.  Putting simple measures and practices (such as performance by design approaches) in place during development and product lifecycle will mitigate performance and non performant risk.  I’ve seen many of the above implemented and they work extremely well.

See Also:

Proactive Performance Prevention

Performance By Design – an Agile Approach

My Top Tips For Performance Testing

Unforeseen Costs of an encapsulated approach:  

I’ve termed the act of encapsulating, blackboxing or providing a service as an ‘Island’ (aka information island).  I’m not fond of the term but its seems apt. Islanding occurs in reality and frequently within enterprise environments. Different ‘information islands’ are joined together with the expectation they will fit like a neat little jigsaw – only they don’t.  Here are some of the reasons I see the pieces not joining neatly together as one would expect:

Replication: Repeat checking, Performance

Often there is duplication replication of functions within separate islands.  A simple example of this is transferring information between islands through XML, also functions may be repeated to wrap up information into a format that is easier to pass around. Passing information between Islands means the Islands aren’t quite sure if its been syntactically correct or validated.  This creates overhead, as multiple same checks for the passage of data through the system is performed.

Demand: Disregard for execute time, Performance

Example:  If I require an array of objects from an Island, should I ask for one at a time or all at once?  In most cases all at once, however in the real world this is down to the experience/skill of the requester.  I’ve seen plenty of cases where a service has been provided and the external user to the island has placed unnecessary high demands on the island.  When you black box its easy for the requesters to lose respect for efficiency, resulting in a higher than necessary overall execution time.

Abstraction: Loose of Understanding, Maintenance

I think some items can be abstracted too much.  Too much abstraction can mean its difficult to understand what should be a simple function (e.g Date formatting).  I’ve lost count of the times I’ve had to open 20+ screens to understand a very simple process or function.  Anything with a complexity function depth of 5+ inherently starts to introduce code risk. There is a little irony here we need to encapsulate to build complexity, but the more we encapsulate (or abstract) the harder it is to understand and there is an associated risk of defects being introduced.

Deluge: Overhead, CPU cycles

If a service requires a small piece of information, but an Island returns that information with an associated large amount of ancillary information (which is swiftly disregarded),  this is wasteful. I’ve seen this happen because internal teams simply haven’t communicated well. Services are built and then requests for small pieces of information result in a large XML being returned, much of which is redundant.   This places a large amount of demand on run-time memory, the Island’s DB and unnecessary processing of the XML.  Islands do not know exactly what is going to be required (or how frequency), so they overengineering of return data which then become redundant

Policing:

If you open your island up to the external world, there is code overhead – you have to actively police the requests coming in. Throttling, authentication and ensuring the business as usual system isn’t being affected.  If some form of policing and control isn’t put in place someone can misuse unintentionally.

Loss of Control: Complexity: Execution

If information is sent to an essential external Island but and no response is received what should happen? If I get a response back that is outside my SLA, how do I prove? Is it my service?, the other service? or something in-between? How do I replicate?   Relying on external services by third parties (such as Credit Card) introduces a whole raft of difficult issues should it not perform as expected.   Black box islands are fine when they work however they can be incredibly tricky to isolate as a source of issues if you have no control or visibility.

There are other costs that are overlooked …  I’ll add as I think of more.

Wasted CPU cycles are BAD. 

Some of the above items result in wasted or duplication of CPU cycles, these quickly add up to a large performance overhead and an associated large financial cost, either in lost business, developer re-engineering time or associated hardware costs.

With ‘islanding’ an unintentional loss of respect for CPU cycles naturally occurs, system behavior and architectural respect is naturally lost.  High demands and a loss of performance are symptomatic of this.  This gap is why APM solutions have grown and become such large businesses – customers need help managing the OO jigsaw.

Many of the above instances can be prevented with oversight, having someone responsible for a system wide view (a performance by design role) – working across the project and communicating within all teams. However I’m seeing a tendency for teams to increasingly work in isolation – either because they have high demands on their time, they are geographically dispersed and the effort to communicate becomes difficult.

Lets look at this web request -

As we can see that an id generates for BS_Type parameter of the request. This id changes in every replay and requires parameterization for effective script replay and business flow completion.

To overcome the challenge, we add a regular expression extractor to extract the value of dynamic data during replay and use it in subsequent request. Regular exp extractor can be added by, Right Click on Webrequest -> Click ‘Add’ -> Click ‘Post Processors’ -> Click ‘Regular Expression Extractor’.

Some of the key elements of RegEx are -

• Reference Name – Name of the variable used for the dynamic id, here it is named as QATypeID. In subsequent requests in JMeter, we’ll reference this variable as ${QATypeId}.
• Regular Expression – This is the key statement and we will extract the data we’re looking for using this expression. Brackets are put around the value we want to extract. So in our example we have qatype=”(.+?)”     var sr_f = “mn_newsmartsearch.php”; or, Here we are looking for the value that is located between  qatype=” and “    var sr_f = “mn_newsmartsearch.php”; The expression will search for the strings and identify the right string to parameterize based on values of following elements.
• Template – Multiple occurance of LB and RB mentioned above is possible and thus multiple values could be extracted. In our case we have just one, and so we’re using the first template, or $1$.
• Match No. – In case of multiple matches, this element plays a key role. In our case, we want the first match, so we have mentioned ’1′.
• Default value – If no match for mentioned LB and RB combination is found, then one can parse a default value to the variable using this element. It can also be left blank.

Reg Ex text extraction requirements can be different in different scenarios. At times only one value (out of many available) may be required and at times multiple values. The below pasted solutions can be used to meet to different types of text / string requirements.

• *? – Match 0 or more times
• +? – Match 1 or more times
• ?? – Match 0 or 1 time
• {n}? – Match exactly n times
• {n,}? – Match at least n times
• {n,m}? – Match at least n but not more than m times

One can look at the below pasted perl script regex features that supported in Jmeter -

• (?#text) – An embedded comment that causes text to be ignored.
• (?:regexp) – Groups things like “()” but doesn’t cause the group match to be saved.
• (?=regexp) – A zero-width positive lookahead assertion. For example, \w+(?=\s) matches a word followed by whitespace, without including whitespace in the MatchResult.
• (?!regexp) – A zero-width negative lookahead assertion. For example foo(?!bar) matches any occurrence of “foo” that isn’t followed by “bar”. Remember that this is a zero-width assertion, which means that a(?!b)d will match ad because a is followed by a character that is not b (the d) and a d follows the zero-width assertion.
• (?imsx) – One or more embedded pattern-match modifier; ’i’ enables case insensitivity, ‘m’ enables multiline treatment of the input, ‘s’ enables single line treatment of the input, and ‘x’ enables extended whitespace comments.

My personal experience with Jmeter has been great.  Many performance test engineers face difficulty during their first attempts with the tool because of the terminology.Listeners, pre-processor elements, post processor elements can appear overly complicated. A list of such terms can be formed and many testers loose their enthusiasm and move to other tool before completing a POC on Jmeter.

Test Plan – A test plan provides a framework on which different web requests and operations pertaining to a test can be added. The tool will execute these operations during run-time with multiple users (threads) as planned. Some of the important elements of a test plan are – Thread Group, Listeners, Logic controllers, assertions, etc.

Samplers – Samplers are the components that are used to send requests to the server. The requests can be HTTP, FTP, SOAP, XML, JDBC and LDAP requests. For ‘Post requests’, post data can be found added in the parameter section of the samplers.

Controllers - You can group your Samplers (Requests) in controller. Controller thus helps to set order in which you wish the samplers to be processed. You can add a controller in another. Different controllers are – Simple controllers, Interleave controllers, While controller, If controller, Runtime controller, etc.

Assertions – Assertions are added to the samplers as child elements and they perform the task of checkpoints. There are different types of assertions. Some of them are – compare assertion, duration assertion, html assertion, etc.

Timers – Timers are added to samplers to make a vuser wait at an after / before a sampler . We can add think times at a step using timers. Some types of timers that are most commonly used are – Constant timers, uniform random timer, etc.

Pre-processor elements – Preprocessors are added to modify samplers. They are usually used to update variables not extracted from response text or to modify settings of sample request just before it runs.
Some preprocessor elements are -

• HTML Link Parser
• HTTP URL Re writing modifier
• HTML Parameter Mask
• User parameters
• Counter

Postprocessor elements – Postprocessors execute once the request is made from the samplers. They are added as child to the samplers and this ensures that a particular postprocessor executes after a specific sampler. These elements are usually used to process response data and to extract a particular value from a response.
Some widely used postprocessor elements are -

• Regular Expression Extractor
• XPath Extractor
• Save Responses to a file
• Result Status Action Handler

HTTP Proxy Server – The Proxy Server keeps a watch and record your actions while you browse your web application with normal browser. As one moves performing steps in the browser, corresponding web requests get recorded as samplers in the test plan and those can directly be added to the test plan when one plans to execute a test.

HTTP Cookie Manager – This is another very important element of Jmeter. While replaying, Cookie manager stores cookies in the initial requests and sends them to the web server whenever required. It helps Jmeter function like a normal web browser.

Listeners - During replay or test run, results of a sampler get saved in listeners in the form of tables, graphs, trees or simple text. Each listeners display response information in specific way. Eg – ‘View Resupts in table’ enables one to view results in Table manner, with sample number, response time, response status, bytes, etc all in tabular format.

Some widely used Listeners are -

• Aggregate Graph
• Aggregate Report
• Assertion Report
• Graph Results
• Monitor Results
• Summary report
• View Results in Tree
• View Results in Table

The Performance of web sites is increasingly becoming important. Companies are realising the importance of snappy performance. Losses caused to businesses due to poor performance issues can be huge (including the company’s image). Tracking the performance issues, analyzing and resolving those before the code moves to production is increasingly important. This increase in the need for performance across major verticals in the market is causing a spur in the industry. With such a rapid movement in the domain, many functional testers are dipping into  performance testing. Whilst learning these new technologies and tools many of the basics are missed; often it leads to a lot of re-work or an incorrect deliverable. To minimize this gap, I am writing this article.

Protocol –

A communications protocol is a set of procedures or rules to be followed for exchanging messages (communicating) between devices on a network. Protocols are structured using a layering scheme as a basis. Instead of using a single universal protocol to handle all transmission tasks, a set of cooperating protocols is used. These different protocols are governed by different rules and thus serve different purpose each.

Examples –

• File Transfer Protocol (FTP) is used to transfer files from one host to another host over network.
• HTTP (Hypertext Transfer Protocol) servers as a request- response protocol for distributed, collaborative, hypermedia information systems. It is the foundation of data communication for world wide web (www). It functions in the client-server computing model. In HTTP, a web-browser, acts as a client, while an application running on a computer hosting a web site functions as a server. The client submits an HTTP request message to the server. The server, which stores content, or provides resources, such as HTML files, returns a response message to the client.

• SMTP (Simple Mail Transfer Protocol) is a protocol for sending e-mail messages between two servers. Most e-mail systems that send mail over the Internet use SMTP to send messages from one server to another; the messages can then be retrieved with an e-mail client using either POP or IMAP.

TCP/IP protocol Suite -

The Internet protocol suite, commonly known as TCP/IP, is the set of communications protocols used for the Internet and is the most popular protocol stack for wide area networks. TCP stands for ‘Transmission Control Protocol’ and IP stands for Internet Protocol. TCP/IP suite falls into 4 different layers -

Application layer: Process to process: On this layer, applications create user data and communicate this data to process (peers) on same or another host. Higher level protocols such as SMTP, FTP, SSH, HTTP, etc. operate on this layer.

Transport layer: Host-to-host: This layer deals with opening and maintaining connections between Internet hosts and constitutes the communication between two network hosts, either on the local network or remote networks. This is where flow-control, error-correction, and connection protocols, such as TCP, exist.

Internet layer: Internetworking: The internet layer has the task of exchanging data and establishing connections across network boundaries. Internet Protocol (IP) that defines IP addresses is a primary protocol in this layer.

Link layer: This layer defines the networking methods within the scope of the local network link on which hosts communicate without intervening routers.

Browser Cookie –

Browser cookie also referred as web cookie is a piece of data stored within a browser by a website and then subsequently sent back to the website by the browser. Cookies help the websites (servers) remember actions that the browser (client) performed in the past – such as logging in. Cookies also serve the purpose of user authentication and if not implemented correctly, a cookie’s data can be intercepted by a hacker to gain unapproved access to the user’s data and possibly to the originating website.

In modern performance test tools, cookies are managed by the tool and hence a performance test engineer doesn’t have to manage the cookie information while automating an end user’s action. The tool maintains the cookie information within itself and provides the web server with the cookie information during run time. Tools like Loadrunner and SilkPerformer also store cache information for each vuser and help performance test engineers to replicate– first time user (no cache) or revisiting user (use stored cache) scenario.

Bandwidth –

In networking, bandwidth is the maximum channel capacity of a network communication path. This path is the channel connecting computing devices. In performance tests, when aspiration is to emulate test more close to real time situation, bandwidth restrictions are also taken into consideration. Modern performance test tools provide us with the feature of putting network / bandwidth related restrictions. In loadrunner, this can be done using ‘speed simulation in ‘run time settings’. In SilkPerformer, this can be done by changing ‘Internet settings’ in ‘Active profile settings’.

LoadRunner Network Bandwidth Settings - 

SilkPerformer Network Settings - 

Bandwidth simulation can be achieved more effectively by using WAN emulators.

WAN emulation –

WAN emulation is creating a network scenario in the test environment, exactly similar to production environment (real-time, end user scenario). This can be achieved using WAN emulation tools. Popular amongst those are – WANem, Shunra, etc.

WAN emulation thus allows the performance test team to setup a transparent application gateway which can be used to simulate WAN characteristics like Network delay, Packet loss, Packet corruption, Disconnections, Packet re-ordering, Jitter, etc.

IP Spoofing –

IP spoofing in performance test domain is assigning different IP addresses to different VUsers during a performance test. This helps the web server identify requests coming from different vusers as the ones coming from different machines, thus taking the test to more real time scenario. IP Spoofing in performance center can be done in scenarios section.

Most application development team have moved onto an AGILE environment. This Agility in the software development process is opening more doors within performance testing domain. The aspiration in past was ‘resolving performance issues post app development’. The target now is ‘tracking performance issues right from the first day of development’. The need of performance test experts both conceptually and practically (tool knowledge) is going to increase. The questions that remains is – How will you leverage this opportunity?