Mordechai speaks

Hacking weeds with Fog

2013-08-01T15:30:00.000-07:00

If you're working on HPCloud you may want to monitor your assets, and know where your instances are running, and what state there in. Additionally, you might be concerned with cost, which like many cloud providers, is determined by the size of the instance [read amount of allocated resources] and instance up time. HPCloud provides you with 3 different ways of working with their OpenStack API.

Python Nova Client
Windows Shell
Ruby Fog - yes please!

If your reading this post you are likely already familiar with the awesomeness of Fog, if not you'll definitely want to take a gander at Fog.io

Fog works with different cloud providers and receives contributions of code changes which they push into their fog gem. If your working with HPCloud its worth it to work to go ahead and grab HPCloud's HP Fog gem. For more information check out: https://docs.hpcloud.com/bindings/fog

Assuming you are working with HP Fog version 20, you can connect like so:

Ok - thtats cool so we get a connection back that will allow us to work with the compute service...yawn.

So now lets do something a little more useful. Lets imagine that every team member has a ppk or security key which is labeled by either the team or instance owners name. We can group the number of instances using this ppk type for identifying assets or even use it to execute specific tasks on those labeled instances, maybe even throw in some Chef'ness to map specific roles.

Lastly I did mention earlier that I wanted to understand cost. Well if you have multiple instances of different sizes, on different avalibilty zones your not going to want to do this by hand.

Even if working with Ruby isn't your thing™, anyone can easily see the value that Fog is giving you. Script-able infrastructure changes, and asset querying that spans the Compute, Storage, DNS, and CDN services is some crazy useful stuff.

Happy Bootstrapping!

Performance Testing Strategy in 5 steps

2010-01-03T07:33:00.000-08:00

When I get put on a new project I'll make certain adjustments but overall my goal is to get to 'Performance Ready' as quick as possible. This allows me to divide my time on things that really matter. I can focus about 70% of time on exploratory performance testing and syncing on serious defects with developers; while 20% is spent on analysis, and 10% spent on fun things like front end performance or sleeping.

Below are a few notes on how I get to 'performance ready' in a few stages.

Performance testing either excites, or annoys. Its a specialized field which requires above average knowledge in several areas not limited to testing, development, architecture, networking, and social engineering. One can have a pretty solid performance approach in 5 very simples stages.

Stage 1: Define what you need to do
This is pretty simple. Identify which individuals are your real stakeholders and have the information you need. These individuals may range from obvious roles like: architects, managers, team leads, developers, testers, and of course the product owner. Don't overlook your individual developers or testers-in-the-know who can sometimes yield much more valuable information then leads or managers. Once you have an idea of who really holds the information you need to know about, next keep in mind whom out the group really affects decisions. Another words understand who calls the shots...in some groups for example a development lead may have a larger impact on resolving issues or authorizing [read: demanding] changes to improve performance then a QA manager.

So now we've identified who really matters. Next lets shmooze..

Identify and get wide agreement on suggested test scenarios.
These scenarios should be representative of types of user activities, and groups of users.
When translating assumed customer activities into test scenarios and later into workloads you'll need to consider average assumed use, and extremes cases.

What's the difference between test scenarios and workloads?

A test scenario is a specific scenario in which a feature or part of the application is targeted with a specific functional objective.
A workload is collection of multiple test scripts which represent one or more test scenarios; and are executed to simulate specific application use on mass level.

If your lucky enough to merit logs from customers' servers, or anything that provides analytical details to construct workloads then by all means don't be a donkey...use it. Let's say that your not that lucky...you'll need to sit with your product architect, and development lead and understand the products pain points. Next sit with the Product Owner who deals directly with customers and ask him/her for feed back of which parts of the paths through the application are most common. Ask the Product Owner what features are customers asking for or expecting in this release; what are the most common user cases that customers are using now and will be using in the next release. It's also critical at this point to understand from the product owner how licensing for your application works. All the user scenarios that you have gathered and discussed should then match the licensing structure that your actually selling or believe will be sold. Meaning if your selling licenses for 40 application users, don't test for 20.

Define your data set.
One more time...define that data set and do it well. Without realistic test data, or good customer intel all your hard work will benefit each every customer that resides in imaginary land.
Your work is useless if its not measured against real test data. If you don't know what your customers are doing then ask them. If you can't ask them, then speak with the customer service agents and product managers who are on the front line. If your working on a product where the product owner and manager have no idea what the customers will be doing or what types of environments that they'll have - then you've got bigger issues then performance testing.

Define bug criteria.
If your a performance tester with QA background this step should be self evident. All non-functional testing (in this case performance testing) requires objectives set clearly upfront; otherwise prepare yourself for never ending testing. Break up your performance bug discussion into a few general categories such as but not limited to: Scalability, Optimization, Functional, Degradation.

It's important for scalability and optimizations issues to be based directly off your dataset with assumed growth factored in; which has been agreed to grudgingly by your stakeholders in earlier discussions. Otherwise any good finds will be subject to constant debate. Functional - does it work? - NO? Throw your shoe at the nearest developer of course...just kidding...ok not really. My personal peeve is to begin actual scripting of performance scripts only once I have some small bit of feature confidence meaning the feature is mostly flushed out and might partially work, but the structure itself is not changing by much.

Stage 2: Scripting
At this point most features are now working, the data set has been created, and the test scenarios have been defined in high level. Pull out your favorite test tool and goto work. Through out this stage scripts are being created, and tweaked. Don't miss the opportunity to include juicy functional assertions in your test such that other testers can re-use your scripts for functional testing. I remember once sitting in a meeting hearing "But his scripts focus on performance, we need functional tests not performance tests". I responded by explaining that all performance tests contain functional assertions if they don't then your testing is useless. Functional assertions in performance tests are necessary to validate the actions your executing are causing results that your expecting otherwise whats the point? Silo'ed Testers (who own a specific area or responsibility in a team) sometimes forget that others can really benefit from their scripts. If written in a 'drop in and run' style there's no reason why a simple Jmeter script can't contain numerous functional assertions and be created dynamic enough to run against different environments to benefit others.

Stage 3: Defining Workloads

Didn't we just discuss this? -- RULER SLAP - no...the focus of this stage is to consider your previously discussed test scenarios from a top down approach. Scripts should be grouped into flows. These flows are assigned activity frequency, concurrency, and think times. Designing workloads is sometimes overlooked and rushed. Running misleading workloads which are not accurate produce results that aren't accurate either.

Designing complex workloads is really a larger discussion. For those interested see this piece on UCML or just about anything from Scott Barber http://www.ibm.com/developerworks/rational/library/5219.html

Keep in mind this is where you organize, specify concurrency and type of work that should be accomplished in a set period of time. You might have workloads that represent fail over testing, stress, and different types of loads. I prefer to organize my scripts to set specific http requests that consist of a user activity as a transaction. This means if a user performs a "Log in" then all relevant http requests that are executed during a "Log In" are collected as a 'transaction'. The following is an example of http requests organized as a 'transaction' in the script level.

When I'm organizing a workload I'll keep in mind how many 'transactions' are intended on the following levels:

Script level: How many transactions per single execution
Expected Workload level = transaction number of single instance X total V_users per hour

Then in Stage 4 you'll actually create the workload and test your expected vs intended results.
This is really key...below I'll explain why.

Stage 4: Scripting Workload

The purpose of this stage is to 100% verify that the workload and scripts contained within are executing exactly as intended. Meaning the number of transactions, functional assertions, and test results are all going as planned. It's necessary to run simple scenarios limited by duration, amounts of VUsers or both to verify that the tests are really going as planned. If your running data driven tests where randomization can be removed, meaning your reading values from a csv; make sure to run 1 VU on the script level to check to make sure all your data is being used as intended. Why? Lets say your searching for "I stopped reading the crappy post 10 mins ago!" well if your regular expression or assertion is not expecting to be passed a non-alpha-numeric character eg "!" then you maybe in for unintended behavior.

This step might seem obvious but if skipped you might find yourself doing the walk of shame back to development to explain that the P1 level defect that you were screaming about was caused by your scripts not the product.

Stage 5: Performance Ready

Performance ready consists of executing workloads, analyzing results, and communicating results out...then repeating. I like using this simple name because managers understand that the testing cycle is now in a stage which implies that the necessary performance ground work is in place and testing is ready to be executed.

Executing Workloads
In order to squeeze every drop of value out of your performance scripts its necessary to have a framework in place that allows for flexible execution and result management. Several exist depending on your tooling.

If you want to know if your framework is sufficient enough or whether to move to something else ask yourself the following:

Can I push a button and go home to my family and come back to work the next day with results?
Are my results persistent, organized, and in an analyzable (thats definitely not a word) without much background work required before or after runs?
If my tests fail in execution or processing results, can I still recover results the following day or later? Meaning your work isn't lost but recoverable.

If you have a larger team you might consider:

Is my process visible, understandable, and usable by all team members?
Is result/issue notification clear enough that anyone on the team can resolve it besides the weird guy with stank breath?
How happy is everyone within the team with the framework? Are there any steps outside of analysis once in a 'performance ready state' that are performed manually and increase frustration and result turn around time?

Remember keep 'stank breath' guy happy....he's got feelings too, and may be your best resource..

Analyzing Results
Outside of scripting most of your time is probably spent here. You'll need different reports for different stakeholders. Your ability to efficiently analyze results will be dependent on the maturity of the framework in which your results are managed. Generating reports on the fly for specific stake holders is important but not as important as generating results that are useful to YOU.

Most likely your average developer, team lead and manager could give 2 hoots about what a specific transaction's median, 90% value,max and min may be for a certain transaction. But you my dear performance tester you do care...it will make your life easier when you have the metrics you need at your finger tips.

The point is...automate the creation of reports that you need to make quick decisions first. Leave automating cute charts for later until you have what you need. It's possible in your analysis of results the the numbers show the workload's aren't achieving the intended level of stress, or could be overshooting it. If that's the case go back to redefining your workload to make it accurate. Involve the necessary folks to think things out so that what your running and what people think your executing match up. The worst possible situation is to find yourself testing things that don't matter, while things that do go overlooked.

Communicating Results
Communicating results in timely manner is critical. Don't get bogged down on generating pretty graphs, save yourself time by creating templates for sending out the information. Additionally it may be helpful to gather all the stakeholders and give a brief demo of how to interpret performance results from the templates that you send out. You may find out that each technical and non technical audience per project is different. There also maybe easier ways to communicate more effectively by understanding what your audience immediately understands vs what they need to have explained. Managers may want high level graphs, developers specific response times, architects may want everything - your ability to present clean, accurate results will build your cred' - if you need further convincing check out Mike Kelly's article on "What's your credit score" http://www.testingreflections.com/node/view/4945 'nuff said eh?

If your looking for a thorough book on performance testing check out this online book Pattern and Practices: Performance Test Guidance for Web Applications This book contains a ton of great information that makes it an easy read, in addition to a nice reference. Many of the stages and ideas discussed in this post are covered in detail, with varying chapters on implementation per methodology.

Testing Restful services with Jmeter

2009-11-09T03:26:00.001-08:00

Outside of Transaction Controllers, regular expressions and xpath extractors are probably used frequently in your scripts. Sometimes you have the need to collect multiple variables from one or more responses and store them in one variable that will become a data parameter in a subsequent request.
Generally if you read through Jmeter User Group threads you’ll see lots of people trying to do this or at least asking how do I store variables in an array type of data structure.
In order to do this

lets review a few concepts about regex, and Xpath extractors. When you use these types of extractors your parsing a response because your interested in one more pieces of information. This information your interested in matches a pattern. All matches to the pattern are stored in capture groups. These groups are referenced by there variable name and an underscore with a number for the specific capture group.
If you’ve used regex or Xpath before its fairly easy to specify a specific value by applying a pattern, and then using its resulting capture group when needed.
But what do you do when you need to collect all the capture groups and put them together to create a specific data parameter.
Enter Beanshell
Beanshell is a light weight java like scripting language that allows you to do work with Jmeter variables. In the following scenario let’s assume I have to test a rest API service which returns an XML response. In the next request I will need to take specific response data from the 1st API request and collect the data by including a comma in between data values which will generate from our capture groups to be used in the next out going request.
Our first API response is the following:

<user>
<name>Adam Rowe</name>
<uri>myapi/user/+Adam+Rowe</uri>
<id> Adam Rowe</id>
</user>
−
<user>
<name>Albert Cruise</name>
<uri>myapi/user/Albert+Cruise</uri>
<id>Albert Cruise</id>
</user>
−
<user>
I’ve highlighted the parts of the response that I”m interested in in blue. We want to be able to collect the users id values which are shown by <id> tag in blue.
We then need to append them in in one variable and store a comma value in between each of the capture groups. The following screen shot shows how we would then use this newly created variable that contains a list of ids shown here as ${list_of_users}

Our test plan:

In the response of ListUsers Transaction /datamanager/Collection/all we can apply a xpath extractor to collect the different <id> tag values which will be returned in the response. In Requesting Users Transaction specifically the /user request we apply a BeanShell Pre-processor which is applied before the request. In this Beanshell preprocess element we will access different variables that are set in the Jmeter scope in order to reference them in the beanshell scope. Below is a sample of basic code used to do this.

This simple code snippet does the following:
a) gets the total number of Xpath capture groups another words the total times id’s was collected
b) Loops through the total number of capture groups appending the value and a comma into the “list_of_users” variable.

c) it removes any extra unneeded characters
d) places the newly constructed variable back in jmeter scope.

You can download the template or if your fancy pants use git
git://github.com/mordechai/Sample_Code.git

Ruby Sparklines

2009-07-13T13:15:00.001-07:00

My very bright younger sister was visiting me a few weeks ago. While the rest of the family was enjoying some relax time we decided to mess around with some Ruby.

I wanted to show her how easy it would be to open a file, parse it, and then create a sparkline. The code below is actually parsing a native_stderr.log created by Websphere during my test runs.

Jmeter remote execution, log collection via STAF / STAX

2009-07-11T14:43:00.001-07:00

One of the biggest challenges after learning your way around Jmeter is figuring out a way to organize and retrieve results in an automated way. There are different documented ways of doing this via Ant scripts or even neat graphing results like Performance Bling. Though my needs were different and after much looking around, coding and testing my performance environment is made up of 3 major areas.

i) Jmeter
ii) STAF / STAX
iii) Ruby
My focus will be on integrating STAF / STAX why I choose STAX as the glue between the different parts in my performance environment..
So what's Staf?
Staf can be thought of as a pseudo command shell that allows for specific commands of work to be executed on local or remote host machines depending upon permission settings of the caller and receiver of the requesting action. STAF can perform specific functions or tasks that are generally related to process automation such as file manipulations, command line actions, or logging functions. The focus moves away from code and OS specifics to completing pre-defined tasks of work to be called more efficiently.
A STAF action can be broken up into processes or commands. Both contain specific parameters and have pre-defined actions for what they do. In order to really benefit from combining multiple STAF commands into one script we'll need to look at STAX.
STAX is a service plug-in of STAF that allows for STAF commands to be written in XML. In STAX files STAF calls can interact with python using the <script> tags. The resulting action of a STAF call is returned by the STAXResult variable which is really a python list object. Inside this object two values are stored. The first indicates the STAFResult or value of the STAF call's execution status 1 or 0, the second value contains the standard out or error message if its set to store it.
So lets look at our requirements for our Jmeter scenario.
We want to be able to:

a) start Jmeter and execute a script
b) collect result logs including error logs
c) combine files where necessary.

So lets begin by creating a function that will understand this:

jmeter -n -t myLoadTestScript.jmx

and turn it into this

   1: <call function="'runJmeter'">'apu-testmachine',LinuxjmeterPath,'testscript_Linux.jmx','/opt/Jmeter/bin/scripts/GA',testDuration,'/opt/Jmeter/bin/scripts/GA','/opt/Jmeter/bin/scripts/GA' + '/staxRun.log'</call>

The idea here is that we ask STAF to make a STAF Command call using STAX code. We pass our Jmeter executable location, the test script, working directory, duration and other needed values.
This command line call will invoke Jmeter for the length of the test. Once the timer duration value has reached the "testDuration" value then the STAF process executing on the machine will be ended abruptly. This is important to keep in mind because this is the same as killing Jmeter's process during a test run.
Fortunately on Linux this is enough to stop the test by killing the Jmeter process which is a child process of the STAF command executing on that machine. For Windows machines executing this code the Jmeter process will not be killed. Unfortunately you will need to use a separate function to kill Jmeter on windows.

<script>
   
   LinuxjmeterPath = '/opt/Jmeter/bin/jmeter'
   WinjmeterPath = 'C:/Jmeter/jakarta-jmeter-2.3.2/bin/jmeter'
   winKill = "taskkill /f /fi \"imagename ne explorer.exe\" /fi \"memusage gt 40000\""
 
</script>

The basic idea is that taskkill is being performed via a command process through STAF which will kill the Jmeter process. This isn't the prettiest way to kill Jmeter but it's been effective for my testing needs in the past, so I've had no pressing need to change it or improve it.

Additionally one might consider performing some type of list process to verify that Jmeter is not running after the duration time has passed on Linux. This is something nice to have, but something I have not had a need for as stated above.
So the significance of what we've discussed so far is that we are able to have 1 function to run on windows, or linux to call and execute a Jmeter test plan. The advantage is that we don't worry about OS specifics and focus just on the parameters that we are passing. This is very useful if you have several load generator slave machines that need to fire off tests, and each machine may have a slightly different set up. This is also useful for queuing slaves to start tests within the same time frame. Generally the delays between starting the tests are usually insignificant and are impacted more by the processing speed on the machine then in possible bottlenecks of invoking the tests remotely through STAF.

Ok this is neat but why not just manage remote machines through Jmeter master/ slave set up?

There are several reasons for using separate slaves to run separate Jmeter instances. The most critical issue for myself was that I was unable to set up (1) Jmeter master to control multiple Jmeter (slaves) with one main log file for response and error logs. It seemed like I would be creating much work and network traffic to transfer such logs in real time. My solution was to run separate instances of Jmeter per machine and store the logs locally. After the test completes STAX is called to collect the respective logs to one computer, I then use a linux command to cat *.extension > new_combined file.extension I then move this file to another machine for parsing, processing and archiving. For some Jmeter users this wont be as useful though I tend to work with very large result files varying from 25 Mb to 250 Mb that are just csv's; so there's a lot of information that is being trapped and parsed.

So here is another simple example of using a simple command line call that will allow us to something very useful such as combing files. to one file. Before I run this code I'm able to identify the directory and testrun to copy the files from by the naming convention used by the folder name. Every test run creates a timestamp as a part of the directory name. I then scan the directories and identify the last modified directory and am able to construct a relevant path from the last modified folder and then copy the files to be combined.

Here I'll show how to identify the last modified folder or really the last test result written to disk, and how to copy that file over for processing

I've shown how to start Jmeter using STAX, how to collect and copy files when necessary and lastly how to combine different result files that come from Jmeter.

In subsequent posts I will show how to collect additional results such as server side metrics, active heap memory usage for Websphere applications, and parse the raw results to create some nice pretty summary reports using Ruby, Win32ole, FasterCsv, and Excel and even sparklines.

Getting Started in Jmeter

2009-04-16T15:16:00.001-07:00

A while back feeling increasingly frustrated by commercial load testing tools I decided to take Goranka Bjedov's advice in her presentation Using Open Source Tools for Performance testing.

I made the switch to
Jmeter and since I've been a much happier tester. I had used Jmeter in the past, but I had never really used it for projects involving complicated flows and correlations. Generally when I say correlations I'm referring to the process of matching up needed information for further requests by parsing the response of earlier requests. I shy-ed away from attempting to make the correlations myself because I felt that it required a lot of time up front to get the test working, and also required a higher level of maintenance for each script. Additionally I was concerned about how the added effort needed to analyze this large csv result that was full of all types of noise. While I found that my concerns were valid, it does take a longer time to create a script, and it does require some maintenance as do all scripts, but what I hadn't expected was the efficiency and forced process that creating Jmeter scripts lead me to adapt that had been absent while using the commercial tools.
<rant>
I'd like to discuss how to effectively tackle Jmeter scripts, and Jmeter results in a further post and focus more on the basics for those just getting stated with Jmeter.
</rant>
If you're considering switching to Jmeter I think Goranka makes the point very clearly. Jmeter can do mostly what any other commercial load testing tool can do there are very slim differences. The key points can be summed up on the following:

read the manual
become familiar with the Jmeter digest using Marc Mail
identify a list of key blogs who sum up the ..less easy to read/follow parts of Jmeter
find the process of script creation that works for you.
find someone who uses Jmeter tips from these fellow users are invaluable and can save tons of time..

3 posts that I feel are worth bookmaking and reviewing are:
Tim's post on using regular expressions in Jmeter
http://90kts.com/blog/2008/jmeter-tips-and-tricks-correlating-with-regular-expressions/
on representing transactions as states from a simulation model
http://90kts.com/blog/2008/jmeter-tips-and-tricks-application-simulation-models/
Coreplex Jmeter variables, parameters,and UDV's
http://coreplex.blogspot.com/2008/10/jmeter-variables-vs-properties-vs.html

Load STAF quietly- How To

2009-02-26T06:21:00.001-08:00

Problem: Staf doesn't start unless someone logs into the computer or STAFProc window is closed accidentally.

Well don't fret there's a very easy solution to get STAF to start up; and get rid of that pesky window.

You can read....
the details found here

If you don't feel like editing registries, and searching how to create a service try the following solution.
Add Staf Proc as a scheduled windows task.
Start -> Control Panel -> Scheduled Tasks -> Add Scheduled Task
Then select your Staf version

Set the task to run "when the computer starts"

Click Next. ( Yes password remains blank - D'oh make sure you have Admin rights when doing this..)
Now the last step is to delete that pesky Start up process..

And that's it. Now when you restart your computer go ahead and give STAF < target machine > ping ping a try.