First up are some file related functions and today is reading a UTF-8 file. We care little for the file size so you better have enough memory. If you are trying to load in a large file then you need to consider techniques like paging the data in.

void ReadFileUTF8 (const char* const Filename, const char* TargetParameter)
{

The TargetParameter is the string name of a HP Parameter where we will store the contents of the file.

    const unsigned int SEEK_END = 2 ;

    long FileHandle = 0 ;
    long FileSizeInBytes = 0 ;

    char * DataBuffer = 0;

    int shifted = 0 ;

    size_t result = 0;
    long index = 0 ;

    //Set our TargetParameter to Failed incase we don't make it
    lr_set("FAILED", TargetParameter);

    FileHandle = fopen (Filename, "rb" );
    if (!FileHandle)
    {
        lr_error_message ("Failed to open the file \"%s\"", Filename) ;
        return ;
    }

    //Get the file size. We use this to allocate our data buffer.
    fseek (FileHandle , 0 , SEEK_END);
    FileSizeInBytes = ftell (FileHandle);
    rewind (FileHandle);

    // allocate memory to contain the whole file
    DataBuffer = (char*) malloc ((sizeof(char) * FileSizeInBytes) + 1) ;
    if (!DataBuffer)
    {
        lr_error_message ("Could not allocate memory for buffer") ;
        return ;
    }

    // copy the file into the DataBuffer:
    result = fread (DataBuffer, sizeof(char), FileSizeInBytes, FileHandle) ;
    if (result != FileSizeInBytes)
    {
        lr_error_message ("Could not copy file data into buffer") ;
        if (DataBuffer)
        {
            free (DataBuffer) ;
        }
    }

    //add null-terminating char
    DataBuffer[FileSizeInBytes] = '\0' ;

    //Skip BOM (if there is one)
    if (DataBuffer[0] == '\xff' && DataBuffer[1] == '\xfe')
    {
        DataBuffer++ ;
        DataBuffer++ ;
        shifted = 1 ;
    }

If you don’t have HP Service Test then you can change this next line and at the end of the function return your DataBuffer pointer to the caller without freeing it. I prefer to use HP Service Test parameters where possible because they allow you to do away with memory management for strings.

    lr_set (DataBuffer, TargetParameter);

    //Move back to start of string before freeing memory
    if (shifted == 1)
    {
        DataBuffer-- ;
        DataBuffer-- ;
    }

    fclose (FileHandle) ;
    free (DataBuffer) ;
}

For now here is what I covered. If you’re doing work with HP Service Test then I highly recommend you read these and make the necessary changes. Send me an email if you get stuck.

1. one small step for writing less code
2. focus on the delta
3. configuring services for use
4. surfing the pipeline
5. great fault expectations
6. validating fault behaviour
7. fixtures for validation
8. being executed as

I hope this has helped.

When we are designing user interfaces we should be mindful of this and help the user make the correct decision rather than punishing them when they make the wrong one.

If you leave it too late you could end up in the intersection; if you keep going then you may run the red-light.

To make it more interesting there is a red-light and speed camera; so you can’t speed up and you get fined if you don’t make it.

What do you do?

The issue is uncertainty there are too many variables to make an easy judgement call. Not all intersections are the same length, not all roads are at the same speed, you are not always going exactly the speed limit, you’re not always the same distance from the intersection when the light changes.

If you add additional factors like poor vision from eyesight, poor vision from bad weather or poor spatial skills it becomes even harder.

If you look at this problem as “should I keep going?” we can make a simple change that I think would remove the issue.

If you’re travelling at the speed limit there is a finite distance you can travel in the five seconds from when the light goes orange to when the light goes red.

If you travelling 80km/h then are moving at 22.2metres per second and if you’re travelling at 60km/h then you are moving at 16.6m/s.

If you have 5 seconds to cross the intersection then you are going to travel 111.1 or 83.3 metres respectively.

Now consider the example story and add a single orange line drawn across the lane at 110m from the far end of the intersection. If the light goes orange and you haven’t crossed the orange line then you are never going to make it. The decision is made for you. If you’ve crossed the line then you will make it and can continue at the speed limit.

The caveat is that all of these equations are based on the maximum speed of the road. If you’re travelling slower than the speed limit or there are poor conditions and you are outside the line you have more time to stop. If you’re inside the line then it comes back to the judgement call.

Some may argue that because it doesn’t solve the problem for all scenarios it won’t work or the government would get sued. The judgement call is the backup and is the current process. The orange line solves the problem in all scenarios where the driver is travelling at speed. In rainy or icy conditions it may even help drivers who are unsure whether their car will stop in time.

To me; all of this relates back to designing the User Experience and challenging the assumptions about what we think the user knows at the decision point. I feel we are often too focused on the outcome of a decision (user invokes behaviour) rather than the decision process.

If we’re in the mindset of “if the user wants to do X they can click here and if they want to do Y they can click there” or “if the driver wants to make it through they keep driving and if they want to stop they slow down” we need to take a step back and make sure the user can make the decision in time.

When you are next giving a user choice consider why and how the decision is made. It may be possible to better plan the flow so that no decision is ever necessary.

1. if you’re turning left you will be travelling at a lower speed and better able to stop

“We’ve been testing Buzz internally at Google for a while. Of course, getting feedback from 20,000 Googlers isn’t quite the same as letting Gmail users play with Buzz in the wild.”

Recently I’ve noticed a trend, and it goes with the whole crowd-sourcing thing, where through sheer weight of ‘tester’ numbers, all major bugs would be found. This isn’t a rigorous process, this is luck. 20,000 user’s using a system will give you information about the product but it is entirely based on the demographic of the users.

What is the demographic of a Google employee? I don’t know but a guess would be technically minded, open-minded, friends with other Google employees? From this Q&A with Patrick Copeland the Senior Engineering Director he lists some advice on aspiring tech execs:

“Be technical: At Google all of our engineering execs have very strong technical backgrounds. They were programmers and many of them can – and do – still program today. Advice: remember your computer science and stay sharp.”

If your demographic isn’t varied enough you will end up having 20,000 of the same user testing the system.

In the rest of the BBC article, the author mentions the ‘Google Trusted Tester’ program:

Many of the firm’s new services are tested by the so-called Google Trusted Tester program, a network of friends and family of Google employees who are given confidential access to products before they launch.

Buzz was not tested by this program.

There is a better way.

When I test a system I have a list of items that can impact the outcome of a test from including, user, invocation method, input data, state data and the environment. The first of these is the user. This is what needs to be thought about before the requirements are defined:

• How does the user’s cultural background impact the user of the system?

• How does the user’s psychological state impact the user of the system?
• How does the user’s education impact the system?
• What is the user’s motivation and how does it impact the system?

I include more traditional testing impacts here as well:

• Mobility
• Accessibility
• Vision
• Geography

This is qualitative testing and this is where the bugs like the ones that plagued Buzz are found.

Throwing large numbers of users at a system will not find all the issues that a competent skilled tester will find. There is a chance that this mass of people will find no bugs. Asking users about a system is great way to get feedback; but don’t be fooled into thinking this is testing.

If I fill out a form about the service at my local restaurant am I testing or am I providing feedback? At Google you are testing the restaurant in the real world you a providing feedback to someone that may or may not care.

What irks me about the news post is that Google stated that they didn’t test it properly and the implication is that more users providing feedback will solve the issue. This isn’t a failure of testing; it’s a failure of management to think that user feedback replaces quality testing over the duration of the project.

When HP Service Test starts execution it picks up the protocols.config and uses it for your service call. If you have multiple services to hit in your test that are invoked by your test you can write a mapping file that maps each service to the appropriate protocols configuration file. If you are only hitting one service and that service calls other services on your behalf (an orchestration service) you only need the configuration for the service you directly consume.

Once that protocols.config is bound to the service, HP Service Test will not refresh the configuration until the iteration has complete. This is a particularly useless feature especially when you are trying to test multiple user configurations in the one project. As I outlined there are many ways to structure your service test projects and having each action be a test is an issue here. The solution is one project file per configuration. Not a pleasant experience if you have a lot of user configurations to run and lots of services.

For any non-trivial service testing instillation you are may have to support multiple service bindings and authentication mechanisms. Multiple these factors across services and endpoints and you start getting a configuration nightmare.

The solution of course is to generate the protocols.config file dynamically. I’m not going to provide a tutorial on how to generate this file. It’s dependant on your services and the environment you work in. I’ve only generated configurations for the three or four bindings we cater for. I’ll assume you can do the same.

What I am going to talk about is getting your actual test script so that it reads better when you combine security.

I like the fact that our tests look like this:

ExecuteTestAsUser (MyServiceUser) ;

SetDefaultParameters () ;

CallService () ;

It’s a simple setup that makes your test readable and more robust. ExecuteTestAsUser is responsible for taking the identifier for a service user. We allocate one user per service. We do this because you are going to iterate through multiple security combinations (has access, no access, super user, banned, etc) if you had users set up in those security combinations and you attached identity specific data to the user you have the potential to disrupt other tests running at the same time.

So it’s better to give each service a specific user and at the start of each test have code like so:

GrantSuperUserToUser (MyServiceUser);

The implementation of such a function is dependent on the authorisation and authentication mechanisms in your environment.

Your users should be setup in an enumeration like so:

enum UserScenarios
{
    DoNotSetSecurity,
    MyService1,
    MyService2
}

This helps to keep your user accounts type safe. You won’t misspell the username in your tests. Use a look up table to get to your username and passwords:

char* usernames[] =
{
    "User001",
    "User002"
}

char* passwords[] =
{
    "PASSWORD",
    "PASSWORD"
}

The point of all this is that when you call ExecuteTestAsUser you can query the lookup tables using your identifier to resolve the username and password. This information is required for both web_set_security and protocols.config.

There is one caveat with protocols.config but I’ll discuss that tomorrow as there is no workaround as yet.

why would we want such an abstraction?

The data store abstraction is useful because as I discussed here on my article about what to focus on in service tests we need to go back to the original source to validate information. If that code is in every test and the data store changes you need to recode every test. A better solution is to get the information you need (as supplied by the service) and pass that to the validation fixture.

If the data store changes then you can rewrite the validation fixture without having to change a single test.

If the service response changes then you are going to change every test anyway but, the scope of your changes are reduced. If you’re using the pipeline then you change your ExtractResult function and the code to pass those values into the validation fixture. This is a usually one-line change in each test (find and replace anyone?) and a pipeline change you had to make anyway.

what will our code look like?

//Result Validation
{
    ExtractResult () ;

    CheckResultCountEquals (1) ;

    CheckAwesomePersonAgainstServiceResponse
    (
        lr_get ("{Result_AwesomePerson_1_Id}"),
        lr_get ("{Result_AwesomePerson_1_Name}"),
        lr_get ("{Result_AwesomePerson_1_Address}"),
        lr_get ("{Result_AwesomePerson_1_MobileNumber}")
    ) ;
}

As you can see we have an ExtractResult call as we always do. We verify the result count; this is always a useful check to put in every test that returns a collection. Sometimes you get more than you expect.

Finally, we have our function CheckAwesomePerson. It takes each of the properties from the service request as parameters. This is where we separate the service response from our test. We are just using properties now. If the service gets another property then you will have to update the tests that are impacted but tests that are not impacted won’t need to change.

If for example our additional property was a date/time of the request then we don’t need to validate that in this test; we’ll have it in a separate test focusing on that attribute. Here we are focussing on awesome people.

Next up is the implementation of our CheckAwesomePerson method and this is where we separate our test from the data store implementation. In this case it is d a database.

void CheckAwesomePerson
(
    const char* const Id,
    const char* const Name,
    const char* const Address,
    const char* const MobileNumber
)
{
    lr_set (Id, "__CheckAwesomePerson_Id") ;

    GetRow ("AWESOME_PEOPLE", "ID", lr_get ("{__CheckAwesomePerson_Id}"), "__CheckAwesomePerson") ;

    VerifyRowValueEquals ("__CheckAwesomePerson", "ID", Id) ;
    VerifyRowValueEquals ("__CheckAwesomePerson", "FOLDER_NAME", Name) ;
    VerifyRowValueEquals ("__CheckAwesomePerson", "PARENT_ID", Address) ;
    VerifyRowValueEquals ("__CheckAwesomePerson", "FOLDER_TYPE", MobileNumber) ;
}

As you can see we get the row and validate it against the parameters. If the Awesome_People table ended up being moved to an xml-document or a web resource then we can change this implementation and the test themselves won’t need to change.

The goal is to have your service test code look like this:

SetDefaultParameters () ;

FaultExpected () ;

CallService () ;

ExpectFaultWithCode ("ServiceEpicFailFault") ;

The above snippet is of course assuming that your services through fault codes with each fault.

What we are going for is a one-line validation of fault behaviour. If your faults have an additional information that you want to validate then pass it into the function. If you have different types of faults (UserFault, BusinessFault, SystemFault, ServiceFault, etc) then you will want to change the name of the function for each one.

It is better to have the following code:

ExpectServiceFaultWithCode ("ServiceEpicFailFault") ;

Rather than

ExpectFaultWithCode (“ServiceFault”, "ServiceEpicFailFault") ;

The reason being is that your fault type is now strongly typed (you could achieve the same with an enumeration) and spelling mistakes will not cause incorrect fault validation.

The basics of your fault validation code will be something like this:

void ExpectGenericFault (void)
{
    int result = 0 ;

    result = lr_xml_extract
    (
        "XML={response}",
        "FastQuery=Envelope/Body/Fault",
        "XMLFragmentParam=Fault",
        "NotFound=Continue",
        LAST
    );

    if (!result)
    {
        lr_error_message("A generic fault was expected and not found") ;
    }
}

In this snippet we are expecting a generic fault; a SOAP fault with no additional information. We report an error if no fault is found.

If you have fault codes to validate then you will want a method signature like so:

void ExpectGenericFault (const char* const FaultCode)
{
    //...
}

the response document

In our code we are dependency on the response document. We have done this all through our [pipeline] code. The reason why we do this is that if every service stores its results in a HP parameter called “response” we can take it for granted that when you want the current service response you use that parameter. This saves effort giving each response object a unique name and makes your code more readable. You have to be careful but in most cases when the response variable is not set HP Service Test will halt execution anyway.

what fault code?

Where do you get the expected fault codes from? If your developers are good then you can get documentation from them that include the faults and when they are expected. This is the kind of information they would provide another developer that is going to consume their service. If they don’t have this information then they are doing shoddy work.

If you’re waiting for information you can try the test and see. This is what I do when I’m waiting for information; test anyway.

• If you get a fault, did you expect it?
• What about the fault code; does it make sense?
• What about other fault information; is it useful to the consumer?

Ponder that information and if it makes sense then move onto the next test. If it doesn’t make sense; raise a defect questioning the validity of the fault information. If you feel a defect is too strong then ask for a ‘request for clarification’.

where fault code?

I’m not going to provide code on how to validate a fault code response because I don’t know how your fault codes are stored. It’s basic XPATH validation like this:

success = lr_xml_find
(
    "XML={source}",
    "FastQuery={xpath}",
    "Value={value}",
    "NotFound=Continue",
    LAST
);

But if you have multiple fault codes then it may be more like this:

success = lr_xml_find
(
    "XML={source}",
    "Query={xpath}",
    "Value={value}",
    "NotFound=continue",
    "SelectAll=yes",
LAST
);

I’ve got the solution and it makes our tests a bit more explicit about our expectations. This is a good thing; the more someone can gather about the expected behaviour of the test, from the test, the better.

Rather than hard coding our expectations into the web request we want to get a setup that looks a bit like this:

SetDefaultParameters () ;

FaultExpected () ;

CallService () ;

The benefit is that anyone reading the test will see that we are expecting a fault. If we don’t want a fault then we use the following code. Note that we don’t need any coding effort for services requests that don’t fault.

SetDefaultParameters () ;

CallService () ;

To achieve this setup we need to make some small changes to our pipeline.

The first change is in SetDefaultParameters; here we setup the default state for our service; which is a “SoapResult”. We store this in the ServiceFaultState parameter

void SetDefaultParameters (void)
{
    lr_set ("SoapResult", "ServiceFaultState") ;

From here we write a new method called FaultExpected that sets the ServiceFaultState parameter to SoapFault. We don’t ever use the setting AnySoap. This allows anything through the gate and our services should behave in a deterministic way.

void FaultExpected (void)
{
    lr_save_string ("SoapFault", "ServiceFaultState") ;
}

Finally with all that written we need to change each and every service request to date. Thankfully this can be done using a find and replace. We want to go from any of the following setups:

"ExpectedResponse=SoapFault",

"ExpectedResponse=SoapResult",

"ExpectedResponse=AnySoap",

To

"ExpectedResponse={ServiceFaultState}",

And in the context of a full web service call:

web_service_call
(
    "StepName={Service}::{Method}",
    "SOAPMethod={Service}|{Binding}|{Method}",
    "ResponseParam=response",
    "Service={Service}",
    "ExpectedResponse={ServiceFaultState}",
    BEGIN_ARGUMENTS,
    END_ARGUMENTS,
    BEGIN_RESULT,
    END_RESULT,
    LAST
);

Tomorrow, I’ll talk a bit about how to validate service faults.