In most organisations, you will find that while they have a process, nobody seems to know it exactly, or even where to go to find it. The problem, it seems is with the way in which processes are documented. Process documents are usually lamented over at the time of their writing, then shelved without much thought at all. The reason for this I believe is that there is primarily only two times when a process document is actually referenced:

1. When a new employee joins the team, and is shown how things are done
2. When a higher manager asks “how does your team operate”

In my mind, I would much prefer a simpler process flow that is actually used by staff, even if it doesn’t cover every possible eventuality along the way. The visual process document provides the most effective way of presenting the flow of how we go about completing our tasks. Its typically printable on one page (though it might have to be A3), it’s pinnable to your office cubicle, and sometimes as importantly, can be pasted into powerpoint presentations for the business.

So how do you present your testing process?

While I might have gone a little bit overboard with this one, keep Rainmeter in mind if you are trying to display the system information without having to worry if your users have wallpapers turned on or off.

So how can we use Analytics to help build a realistic load profile for performance testing?

A post by Suraj Sundarrajan (@perfengineering) highlighted some of the current uses of GA in modelling user behaviour for performance testing. These include the calculation of user idle time, 3rd party web components like Flash, Silverlight & AJAX and a current area of interest for me, cache.



Industry reports suggest that ‘Cached pages’ can account for up to one-third of all page views. Due to its obvious performance benefits, browser caching mechanism is being extensively used by application developers. Web server logs do not (and will not be able to) capture user activity metrics for cached pages as no request is made to the Web server. On the contrary, Web Analytics tools track visits to cached pages (as they track usage from users’ browser) and thus provide a more accurate picture of “browser caching” on the web application. Performance testers can use this information to determine what percentage of total application usage is being cached and emulate this browser behavior in the load test scenario. – Suraj Sundarrajan

Suraj does a great job of describing some of the elements of Analytics that will help performance testers build a realistic load profile. Though I hear you saying “Analytics? So what, Google Analytics have been publicly available since 2006, and there were many options before then”. This is exactly where I was headed. Though I am impressed with the new features of GA, one of the most valuable additions isn’t actually a feature at all. It’s the fact that they have been around so long, and that we now have enough history to start to evaluate trends. Combining this data with projected business usage provides another invaluable source of information to help ensure a successful performance test.

ldapmodify -h ldap.headwired.com -p 389 -D 'cn=Directory Manager' -w PASSWORD <<EOF
 
dn: uid=$USERNAME,ou=people,dc=headwired,dc=com
changetype: modify
delete: passwordHistory
 
EOF

As the action is actually a delete, this translates to the mldap_delete function in LoadRunner.

// Clear Password History
mldap_delete("LDAP Delete", 
      "DN=uid={USERNAME},ou=people,dc=headwired,dc=com ", 
      "AttrName=passwordHistory", 
      LAST );

The next step was reseting the actual password, and here is the script again. Note that as the script must authenticate every time it calls ldapmodify.

ldapmodify -h ldap.headwired.com -p 389 -D 'cn=Directory Manager' -w PASSWORD <<EOF
 
dn: uid=$USERNAME,ou=people,dc=headwired,dc=com
changetype: modify
replace: userpassword
userpassword: $PASSWORD
 
EOF

This time, we will use the mldap_modify function to change the userpassword value

// Update Password
mldap_modify("LDAP Modify", 
      "DN=uid={USERNAME},ou=people,dc=headwired,dc=com", 
      "AttrName=userpassword",   // Name of attribute 
      "AttrValue={PASSWORD}",    	// New Value 
      LAST );

While the task to be performed here is very trivial, it’s a simple demonstration of the mldap_modify and mldap_delete functions. The full script is below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

Action()
{
  // LDAP Bind
  mldap_logon("TRANSACTION_NAME",
        "URL=ldap://cn=Directory Manager:PASSWORD@ldap.headwired.com:389/ ",
        "Version=3",
        "Mode=Async",
        LAST);
 
  // Clear Password History
  mldap_delete("LDAP Delete", 
        "DN=uid={USERNAME},ou=people,dc=headwired,dc=com ", 
        "AttrName=passwordHistory", 
        LAST );
 
  // Update Password
  mldap_modify("LDAP Modify", 
        "DN=uid={USERNAME},ou=people,dc=headwired,dc=com", 
        "AttrName=userpassword",   // Name of attribute 
        "AttrValue={PASSWORD}",    	// New Value 
        LAST );
 
  return 0;
}

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Xml.XmlException: 'xsi' is an undeclared namespace.

Using the custom web request (as suggested by Wilson Mar, Stuart Moncrieff and many others) bypasses the XML validation steps of LoadRunner, which is a good thing. This is possible because SOAP is just XML over HTTP, so lets take a look.

Understanding SOAP

A typical SOAP message is made up of header and body encapsulated in a SOAP envelope. The SOAP header contains application-specific information (like authentication, payment, etc) about the SOAP message. The header is optional, yet if present, must be the first child element of the Envelope. The SOAP body element contains the actual message.
As SOAP uses HTTP as an application protocol, it utilises a request / response model. A SOAP request is sent to the endpoint via a HTTP POST and the returned SOAP message uses the same response codes for HTTP (200 OK, etc). This request is to get the stock price for the ticker symbol HWB (Headwired Bank), with the response returning a price of $460.75 as the last traded price.

SOAP Request

POST /StockQuote HTTP/1.1
Host: headwired.com
Content-Type: text/xml; charset="utf-8"
Content-Length: 1452
 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
    <soapenv:Body>
        <m:GetLastTradePrice xmlns:m="headwired.com">
            <m:tickerSymbol>HWB</m:tickerSymbol>
        </m:GetLastTradePrice>
    </soapenv:Body>
</soapenv:Envelope>

SOAP Response

HTTP/1.1 200 OK
Content-Type: text/xml; charset="utf-8"
Content-Length: 1298
 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
    <soapenv:Body>
        <m:GetLastTradePriceResponse xmlns:m="Some-URI">
            <m:price>460.75</m:price>
        </m:GetLastTradePriceResponse>
    </soapenv:Body>
</soapenv:Envelope>

SOAP as a Web Service

As a Web Service, the structure of all supported SOAP requests and responses are provided in a WSDL (Web Service Definitions Language). A WSDL provides the following information:

• The interactions service provided
• The arguments and results are involved in the interactions
• The network addresses are used to locate the service
• The communication protocol that should be used
• The data formats that messages are represented in

In our example SOAP web have a structure for our request in the SOAP body. We have a GetLastTradePrice tag, with a child tickerSymbol . How can we determine that this is what we need to submit by looking at the WSDL? The following WSDL definition declares what elements should be in GetLastTradePrice input request.

<message name="GetLastTradePriceInput">
  <part name="body" element="xsd1:TradePriceRequest"/>
</message>

This specifies that our input should contain the “ TradePriceRequest” element which is also defined in the WSDL. The definition is for one element “tickerSymbol” that must be a string value.

<element name="TradePriceRequest">
  <complexType>
    <all>
      <element name="tickerSymbol" type="string"/>
    </all>
  </complexType>
</element>

The WSDL will also specify the response that we should expect from this request.

<message name="GetLastTradePriceOutput">
  <part name="body" element="xsd1:TradePrice"/>
</message>

This specifies that our input should contain the “TradePrice” element which is also defined in the WSDL. The definition is for one element “price” that will contain a float value.

<element name="TradePrice">
  <complexType>
    <all>
      <element name="price" type="float"/>
    </all>
  </complexType>
</element>

The WSDL below is a rough example of the stock service that would be used by the example SOAP.

<?xml version="1.0"?>
<definitions name="Stock" targetNamespace="http://headwired.com/stock.wsdl"
       xmlns:tns="http://headwired.com/stock.wsdl"
       xmlns:xsd1="http://headwired.com/stock.xsd"
       xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
       xmlns="http://schemas.xmlsoap.org/wsdl/">
  <types>
    <schema targetNamespace="http://headwired.com/stock.xsd"
            xmlns="http://www.w3.org/2000/10/XMLSchema">
      <element name="TradePriceRequest">
        <complexType>
          <all><element name="tickerSymbol" type="string"/></all>
        </complexType>
      </element>
      <element name="TradePrice">
        <complexType>
          <all><element name="price" type="float"/></all>
        </complexType>
      </element>
    </schema>
  </types>
  <message name="GetLastTradePriceInput">
    <part name="body" element="xsd1:TradePriceRequest"/>
  </message>
  <message name="GetLastTradePriceOutput">
    <part name="body" element="xsd1:TradePrice"/>
  </message>
  <portType name="StockPortType">
    <operation name="GetLastTradePrice">
      <input message="tns:GetLastTradePriceInput"/>
      <output message="tns:GetLastTradePriceOutput"/>
    </operation>
  </portType>
  <binding name="StockSoapBinding" type="tns:StockPortType">
  <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
        <operation name="GetLastTradePrice">
           <soap:operation soapAction="http://headwired.com/GetLastTradePrice"/>
           <input><soap:body use="literal"/></input>
           <output><soap:body use="literal"/></output>
        </operation>
    </binding>
    <service name="StockService">
        <documentation>My first service</documentation>
        <port name="StockPort" binding="tns:StockBinding">
           <soap:address location="http://headwired.com/stock"/>
        </port>
    </service>
</definitions>

SOAP With LoadRunner Custom Requests

SOAP in its most basic form is XML + HTTP. Therefore, we can use the LoadRunner web_custom_request function to manually specify our SOAP message. This method is more verbose than a web_service_call script, yet as it is a web request, there is no XML parsing performed by LoadRunner. This can be very useful if your web_service_call scripts are returning errors.

Scripting a web service is different a web page. Firstly, unless you have a client application, you will not be able to record and replay, instead you will have to recreate the web service calls based on logs, interface specification documents or a WSDL. It is also important to understand how the operations of the web service will be used in production.

Setup

This section discusses the steps to setup your LoadRunner script to commence scripting.

Script Type

A web_custom_request script for SOAP should be created as a “Web (HTTP/HTML)” script. This allows for Internet Protocol Run Time settings to be configured for handling authentication in a more effective way (read: web_set_sockets_option).

Authentication

The web service you are using may require HTTP level Authentication in order to access the web service operations. This can be performed using a simple web_set_user command in LoadRunner. To avoid a Authentication Error on your first request, the web_set_sockets_option can be configured along with the “Enable Integrated Authentication” run-time setting to avoid this issue.

// Requires "Enable Integrated Authentication" to be set to "yes"
web_set_sockets_option("INITIAL_BASIC_AUTH","1");
// Authenticate
web_set_user("USERNAME","PASSWORD","www.headwired.com:443");

To enable integrated authentication, open “Run-time Settings > Internet Protocol > Preferences > Set Advanced Options” and select yes for “Enable Integrated Authentication”. More information about authentication can be found here.

There may also be the requirement to login using multiple authentication pairs throughout the test. This is possible through the use of the web_cleanup_cookies function. This function will remove all cookies related to the VUser, including NTLM authentication cookies.

// Clear previous logins
web_cleanup_cookies();

Scripting

Making the Request

In order to use the web_custom_request function for SOAP, we must specify that we will be submitting text/xml. The web_add_header function is used to specify this.

// Set Headers to plain text / xml
web_add_header("Content-Type", "text/xml; charset=utf8");

The web_custom_request is pretty straightforward. You specify your URL, Method and Body of the request. The SOAP message, including XML specification and Envelope is placed in the Body section of the request.

web_custom_request("Transaction_Name", 
	"URL=https://www.headwired.com/StockQuote", 
	"Method=POST", "TargetFrame=", "Resource=0", "Referer=", 
	"Body=<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>" 
	<SOAP Envelope Here>
	"Snapshot=t1.inf",
	LAST);

Verifying the Response

The simplest way to verify the response of a SOAP request is to use the web_reg_find function. This allows the specification of text that must be matched within the response text.

// Ensure the response contains correct text
web_reg_find("Text=GetLastTradePriceResponse",LAST);

To extract data from the response and save it as a parameter, you may also use the web_reg_save_param function. In the example below, the entire XML response, including HTTP headers is saved in the “response” variable.

// Save entire response as a string
web_reg_save_param("response","LB=","RB=",LAST);

Debugging Tips

Perhaps the easiest way to debug a web services script that utilises the web_custom_request function is to enable “Visual test results”. This feature can be enabled via the VuGen General Options, Replay Tab

The visual test results will display a report after replaying a script and provide you a script tree view, and allow you to view the XML response from the SOAP server.

Example Script

The following is a LoadRunner script that will send our sample SOAP request to the server. It will ensure that the response contains the term “GetLastTradePriceResponse”. As SOAP uses HTTP, any HTTP level errors (401, 500) will also automatically be failed by LoadRunner.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Ensure the response contains correct text
web_reg_find("Text=GetLastTradePriceResponse",LAST);
 
// Set Headers to plain text / xml
web_add_header("Content-Type", "text/xml; charset=ISO-8859-1"); 
 
// Submit SOAP Request
lr_start_transaction("SOAP_GetLastTradePrice");
web_custom_request("SOAP_GetLastTradePrice", 
	"URL=http://www.headwired.com/stock", 
	"Method=POST", "TargetFrame=", "Resource=0", "Referer=", 
	"Body=<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>" 
"<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">"
  "<soapenv:Body>"
    "<m:GetLastTradePrice xmlns:m=\"Some-URI\">"
      "<m:tickerSymbol>HWB</m:tickerSymbol>"
    "</m:GetLastTradePrice>"
  "</soapenv:Body>"
"</soapenv:Envelope>"
	"Snapshot=t1.inf",
	LAST); 
lr_end_transaction("SOAP_GetLastTradePrice", LR_AUTO);

One interesting outcome of this change was a nasty C compile error.

Error: C interpreter run time error: Action.c (39):  Error -- memory violation : Exception ACCESS_VIOLATION received.

As it turns out, as we were using a proxy auto-config (PAC) file and this error occurs if your download filters are set to block your pac files. Luckily, LoadRunner does report that the PAC file was “Not downloaded due to a filter”, yet on the very next line reports that it was “downloaded and evaluated successfully”

Starting to download the proxy automatic configuration script, URL="http://www.headwired.com/proxy.pac"  	[MsgId: MMSG-27097]
Warning -26554: Not downloaded due to a filter, URL="http://www.headwired.com/proxy.pac"  	[MsgId: MWAR-26554]
Proxy automatic configuration script (URL="http://www.headwired.com/proxy.pac") downloaded and evaluated successfully  	[MsgId: MMSG-27096]

Surely if iframes are considered a critical resource, then the PAC file should cause an error and not just a warning, especially when it causes such a poorly handled memory violation error.

Luckily adding the pac file location to your download filters include list fixes this problem.

Error -26627: HTTP Status-Code=404 (Not Found) for "http://tracking.headwired.com/track.asp"  	[MsgId: MERR-26627]

So why would a 404 cause an error and not just a warning? Turns out that iframes are treated as a critical resource. By default, images, scripts, css are all treated as non-critical resources resulting in a warning only, however iframes are considered to be critical and hence cause an error. It makes sense for iframes to be a critical resource… in 1999 when the virtual mall was a good idea, yet in 2011, iframes are used for tracking services only. Rants aside, I could find the option to treat images, scripts and css as critical (pictured below), however I could not find the option to treat iframes as non-critical.

By turning this option off, all 404′ed resources are treated as errors as well, as you can see in the following output log

Found resource "http://localhost/notfound.css" in HTML "http://localhost/iframe.html"  	[MsgId: MMSG-26659]
Found resource "http://localhost/notfound.js" in HTML "http://localhost/iframe.html"  	[MsgId: MMSG-26659]
Found resource "http://localhost/notfound.png" in HTML "http://localhost/iframe.html"  	[MsgId: MMSG-26659]
Detected non-resource "http://localhost/notfound-iframe.html" in "http://localhost/iframe.html"  	[MsgId: MMSG-26574]
Found resource "http://localhost/notfound-object.html" in HTML "http://localhost/iframe.html"  	[MsgId: MMSG-26659]
Error -26627: HTTP Status-Code=404 (Not Found) for "http://localhost/notfound.js"  	[MsgId: MERR-26627]
Error -26627: HTTP Status-Code=404 (Not Found) for "http://localhost/notfound.css"  	[MsgId: MERR-26627]
Error -26627: HTTP Status-Code=404 (Not Found) for "http://localhost/notfound.png"  	[MsgId: MERR-26627]
Error -26627: HTTP Status-Code=404 (Not Found) for "http://localhost/notfound-iframe.html"  	[MsgId: MERR-26627]
Error -26627: HTTP Status-Code=404 (Not Found) for "http://localhost/notfound-object.html"  	[MsgId: MERR-26627]
web_url("web_url") highest severity level was "ERROR", 1627 body bytes, 1438 header bytes  	[MsgId: MMSG-26388]

So back to the problem, a new domain that I needed to block had surfaced and if the domain name resolved, then it would have been included in the tests without my knowledge. While there are two schools of thought here, in the particular testing that I was doing, the customer did not want their tracking sites being hit from the performance tests, it was time to change approach and move to the include only list.

Why was this important? Well now I don’t have to catch every new tracking feature in order to block it, and my ever expanding list of blocked hostnames was now a thing of the past and everything was coming up Milhouse.

My test HTML page is as follows for anyone who is interested.

<html>
  <head>
	<link rel="shortcut icon" href="notfound.ico" />
	<link rel="stylesheet" href="notfound.css" type="text/css" media="all" />
	<script src="notfound.js" type="text/javascript"></script>
  </head>
  <body>
	<h1>Testing IFRAME</h1>
	<img src="notfound.png" />
	<!-- iframe -->
	<iframe src="notfound-iframe.html" width="100%" height="300">
	  <p>Your browser does not support iframes.</p>
	</iframe>
	<!-- object -->
	<object data="notfound-object.html" type="text/html" width="300" height="200">
	</object>
  </body>
</html>

So what is this magical Wordle you ask?

Wordle is a toy for generating “word clouds” from text that you provide. The clouds give greater prominence to words that appear more frequently in the source text. You can tweak your clouds with different fonts, layouts, and color schemes. The images you create with Wordle are yours to use however you like. You can print them out, or save them to the Wordle gallery to share with your friends

So get out there an create your own!

In my 10 years of performance test consulting, I have seen reports that are one-pagers, outputted directly by the tool with very little user interaction. I have also seen reports that go into significant detail and require hours of data manipulation to produce. Personally, I prefer the later.

Its not that I like am a stickler for documentation, its more that I believe that an execution report should tell me all of the interesting parts of a test, from response times to resource utilization, I want to know as much as I can about injecting a particular load level against an application. When I walk in to work in the morning, I want to be able to answer that question “How did last nights test go?”.

While working at a major bank, I had the opportunity of defining the test execution report templates for a major application. I worked closely with the production availability team to create a report template that provided everything that was required to determine if changes could result in performance degradation. The sign-off process involved a pile of execution reports that were analysed top to bottom and signed-off one by one. Needless to say, I learned a lot about which content was vital for these reports, and which could be ommitted.

Unfortunately, despite a very happy production support team, and a nomination by the client for a customer service award, there was one problem. The reports took too long to produce. Few elements of the report could be simply copied out of the customers performance tool (LoadRunner), and many took parsing raw results through excel to get the desired result.

After a successful release, I left the client and moved onto a new challenge. I wanted to find a way to maintain the quality of the reports while reducing the amount of effort required to produce them. I heard that after I left, the client had employed some automation for the excel processing parts of the report. While this had reduced their time, it was still a very hands on process. I knew there must be a better way.

6 months on, and I now recieve 95% of the elements of this original report in my inbox automatically after each test execution. I had reached a point where my team can now spend more time on analysis, and less time on producing graphs. It is now clear within minutes of an execution if there is a major problem or not. At last, I had my cake and was eating it too.

Over the next couple of weeks, I will document the journey from looking at the individual elements of an execution report to having an automated report in my inbox after each test.

If you have ever wondered what is going on, it turns out the culprit is mmdrv.exe. This executable is the Multithreaded Driver Command Line application that is used to simulate virtual users, and killing all instances from the task manager will remove any locks on your results directory, and you can go back to normal vugenning.

While most of us don’t mind using multiple directories, it seems that even in Vugen 11, the replay log window will always show the log of the first results directory after playback. This can be very confusing when debugging scripts.

This exe can also be used manually to execute vuser scripts from the command line. This can be handy for environment smoke tests. I will write more about some alternate uses for mmdrv.exe in a later post.