XQual Blog

Release of XStudio 1.6a6

2012-04-04T13:34:00.000+02:00

This has been released yesterday.

It includes many bug fixes as well as the major new feature that add to your SUT and campaign tree a "Quality" column indicating an index of quality of the SUT.

Details of the 1.6a6 add'ins:

   * Quality metrics on the SUT tree
   * Quality metrics on the campaign tree
   * New unified colors for progress bars
   * New areas selector on the left-hand side
   * Agent not responding timeout set to 30 seconds
   * Bug-fixes:
     - Incorrect UI indicators when a session is queued too long on
       an agent
     - Session not starting when queued more than 15 minutes on a remote
       agent busy with some other sessions
     - Refresh the progress at the end of a session execution
     - Too many stats recorded: XStudio frozen when selecting the integrated
       bug-tracking root node
     - Wrong color returned for the graph in the iPhone/iPad app.
     - Incorrect statistics reported in email notifications when re-executing
       a manual session
     - Exception while selecting some folder in the Campaign's content tab in
       the upper test tree
     - Main tree refresh issues (ordering) if defect tree is not in the user
       profile
     - Exception and inconsistent behaviour when modifying sprints and backlogs
       (due to velocity chart refresh)
     - Incorrect Bamboo report generated in case of test failure with messages
       (typo in the CDATA opening tag)
     - No more timeout on the sessions that have been queued for a long time.
       The 30s timeout for "not responding agents" now applies only starting
       from the point where the session has really been started by (at least)
       one agent.
     - Wrong background in the campaign tree in some cells
     - Force the stop even when a session did not really started (in ARMED
       state)
     - setCampaignSessionAgentStatus() called only the first time
     - Error while trying t get the ascending/descending icons
     - "generate spec" vs "generate test" button depending on if the
       specifications are enabled in the PLC
     - Coverage label different in the form depending on if the specifications
       are enabled in the PLC
     - Generate specifications does not update coverage on the requirement if
       necessary
     - Generate tests does not update coverage on the requirement if necessary
     - Same width on progress bars in the same tree
     - Wrong new SQL query to retrieve latest results (for quality)

Enjoy!

XStudio 1.6: New outstanding quality metrics!

2012-04-01T23:13:00.000+02:00

When it's about test management, you're looking for some simple ways to know the "quality" of the system under test. It's often something impossible to obtain easily.

You can get it without a single click with XStudio Business or XStudio Enterprise 1.6a6.

The new indicators appear first on the SUT tree so that you have for each SUT its quality. The quality is computed from:

- its functional coverage percentage

- the latest results of each meaningful test cases (test cases covering the SUT):

Mixed with the recently added Coverage column, this is just amazingly powerful. You can also get the details of the meaningful test cases results on the next column on the right.

You will also get the same indicators on the campaign tree. In this case for each session, the quality is computed from the test case results obtained (only test cases functionally covering the SUT are retained) vs the total number of test cases covering the SUT (associated with the session):

Simple isn't it?

XStudio 1.6: New progress metrics

2012-03-17T15:06:00.000+01:00

One of the most important question you need to figure out when you're a test manager is "when my test campaigns execution will be completed".

You can get the answer to this question without a single click starting with XStudio 1.6a5.

There are 2 different types of progress metric you can get:

Progress in terms of test cases executed versus test cases left
Progress in terms of timing: how much time has already been spent but more importantly how much time will be necessar to complete the execution. When you create a test in XStudio you will be asked to give an estimated time for execution. But each time this test is executed, XStudio measures the effective time spent and correct the original estimation. The time necessary to execute a test is calculated from the ten last execution and allows estimating the time necessary to complete the session execution.

The metrics are available directly from the general campaign tree.

When you select "Test cases executed" in the selector at the top of the tree you 'll get the progress in terms of test cases executed:

If you select "Estimated time" in the selector at the top of the tree you 'll get the progress in terms of time spent vs time still needed to complte the session:

The even better news is that If you organize your campaigns in folders that represent something (a functional area, a complete release, a package etc.), the system will consolidate the progress and recursively propagate the metric to parent folders.

This way you know how much time is necessary to finish all the sessions included within a folder for instance.

This feature is available in XStudio Professional, XStudio Business and XStudio Enterprise.

Release of XStudio 1.5sp4

2011-12-31T13:34:00.000+01:00

XStudio 1.5sp4 has been released and is available for download here: http://www.xqual.com/support/download.html

This is a minor maintenance release.

No schema update or configuration changes are necessary.

Changes:

   New web package html home page
   Bug-fixes:
     - CReqTreeModel used in specification's traceability matrix
     - Cannot delete bug attachments
     - Test cases are not automatically created
     - User can assign user group's member only when he has the relevant rights
     - Typos in rights management tree
     - Do not truncate the test and test case path in the tabular manual launcher
     - Auto-wrap long test and testcase names in the right panel
     - When a test name is very long, running the test directly can produce sql failure
     - Params: additional checks on the number of combinations
     - Pairwise algo: negative number of combinations displayed in the preview panel
     - Specification priority and status not imported

XStudio 1.6a2: Oracle support completed!

2011-12-24T01:17:00.001+01:00

It has been tough! very tough!!!

Much more time-consuming than we planned.

We saw here the limits of SQL standard ;)

But anyway, here we are :)

XStudio now supports Oracle 11g.

A new "Database server type" setting has been added where the user can select either MySql or Oracle from the drop-down menu. Then, the user needs to provide appropriate information to connect to the Oracle instance:

That's the only change visible...
This will be available with XStudio 1.6a2+ (as a plugin)

Atom feed created

2011-12-08T14:03:00.001+01:00

... so that it should be easier to automatically import feeds in LinkedIn groups etc.

http://feeds.feedburner.com/xqualblog

XStudio 1.6a1: Requirement coverage on sessions

2011-12-07T16:45:00.001+01:00

Some new coverage metrics will be added in XStudio 1.6a1 including probably the most interesting one everybody is looking for: requirements and specifications coverage.

Here is how it looks like:

What you can observe in this new "Requirements" tab design:

- individual results of requirements/specifications (not covered, success, failure, unknown)

- for each container, the nb of requirement with a specific result

- for each container, some visual indication on the percentage of each

Hope you like it!

XStudio 1.6a1: Mixed test campaigns

2011-12-01T00:03:00.001+01:00

Mixed campaign is a long story. This feature has been requested for long but it was technically requiring so many changes in the code architecture that it had been delayed several times.

But now it's done!

In terms of GUI, the changes are minimal though. you will notice only a couple of new radio buttons in the campaign's "Order" panel:

When you create a new test campaign, just select all the tests you want to be part of that campaign. These tests can be of different kinds (i.e. 3 Selenium scripts + 2 JUnit tests + 4 manual tests). You can reorder those tests manually by using the order toolbar but you can also use one of the two automatic ordering buttons (this will reorder automatically your tests by dependencies or by priorities).

Here comes the news: you have now 2 new options:

1) execute tests from different categories in parallel

In this case, tests from each category will be gathered and executed in a separate and independant thread. Within a single category/thread, all the tests will be executed following the order globally defined but each category will be running on its own and only its set of tests. In this case, there is NO synchronization between categories/launchers.

2) execute tests from different categories in sequence exactly as defined in the order you just set

In this case, the system will execute the tests exactly in the order you defined. At one time there will be only one launcher working.

This second option makes it possible to create sequences of tests mixing completely automated tests with manual tests. For instance, this could allow to manually configure a system then run all the automated tests and complete the campaign by performing some additional manual cleaning operations. This also allows mixing different types of automated tests in one single sequence.

I Cannot think yet about all this brings but it definitely brings a lot of opportunities in your way to test!

XStudio 1.5: New tools on Tabular Manual Launcher

2011-09-05T22:15:00.003+02:00

As planned some new tools have been implemented in the Tabular Manual Launcher.

They include:

Reset all testcases with success/failure/relative/not_executed status. On pressing the Submit button a dialog pops up and asks if it is needed to clear all the comments as well
List & open the attachments for tests and test cases
List & edit bugs (integrated, generic, mantis, bugzilla, jira trac) associated to tests
Auto-submission of the comments on focus lost (at least ! :)). In addition, it's now easy to identify the comments being edited (not yet submitted) and those already submitted

What may be done only for 1.6:

Edit/delete tests
Edit/delete test case

What will have to wait for sure for 1.6:

Create new test
Create new test case

XStudio 1.5: User Profiles

2011-07-20T23:40:00.005+02:00

Starting with XStudio 1.5a5, instead of setting the rights to the users, it will be possible to create some custom user profiles and for each user select one in the list.

By default, 6 user profiles will be present. It will be possible to delete all of them except the "Administrator" one and of course it will be possible to add some new ones. User profiles can be created, copied, edited and deleted. If the user profile associated to one or several users is deleted, the users will loose all their rights until they are associated to another user profile.

The good thing is that if at one point you edit a user profile, the changes will apply to all users associated to this profile. When you create a new user, it will also be easier to set his rights as he will probably need some rights that you already defined in one user profile.

WARNING: By default, when migrating to 1.5a5 all the users will be associated to the "Administrator" user profile. You'll have to associate your users to either the default profiles that fit or create you own user profiles and individually associate your users to appropriate ones.

XStudio 1.5: Custom fields

2011-04-29T00:53:00.005+02:00

One of the major addition to 1.5 will be the abilty to customize the details forms for the requirements, specifications, test cases and bugs.

If you have the appropriate rights, adding a new custom field will be easy enough:

Different types of fields will be available:

integer (text field)
string (text field)
boolean (checkbox)
formatted tring (wiki-style field) - see example above
drop-down menu (combo box)

The user will have the possibility to

select a position where to insert its custom field
specify if the field is mandatory or not
specify a default value
list all of the possible values (in case of the drop-down menu)

The list of custom fields for one type of object will be available on the root node of its corresponding tree and it will look like:

With those example settings, the details tab of a bug would look like:

Note: this feature will be probably delivered through a commercial plugin (very cheap - around 30€ per user). It will be also delivered freely to all the users who have formerly (or are going to) contracted a support license.

XStudio 1.5: Images and URLs in formatted text widgets

2011-04-24T14:16:00.004+02:00

One feature that has been requested by many customers was the possibility to embed images and anchors in formatted text fields. This will be possible in 1.5.
Images are physically stored in the database so that the user does not have to worry about maintaining an external repository.

The GUI is very simple and is as presented below:

To embed an image, just click the "image" button and select an image on your hard-disk:

The wiki tags for Images and anchors are the following:

For images, the name of the image is automatically prefixed with a unique id corresponding to the id of the image file in the database.

When previewing a formatted text embedding an image, the image will be of course rendered. The image is also rendered in reports.

XStudio 1.5: IDs in trees

2011-03-13T01:04:00.004+01:00

Certain people prefer to see the Ids of any object in the trees.

In XStudio 1.5, an option in the settings will allow to activate this feature.
Hence the display can look like:

XStudio 1.5: Customizable email notifications

2011-03-13T00:55:00.007+01:00

Starting with XStudio 1.5, email notification will be generated using the same mechanism than for reports. This means that the content of the email will be the results of an XML (generated by XStudio and including all the required data) transformed through an XSLT.

Since you can edit and modify the XSLT, you can customize your email notification presentation (for instance the logo but also the data you want to include, their orders, the colors etc. etc.).

In addition to this, the number of data included in the bug notification are bigger. With the default transform provided, the results is:

XStudio 1.5: New settings to better control notifications

2011-03-13T00:34:00.005+01:00

In XStudio 1.5, new settings will appear to better control email notification mechanisms:

Send additional notification per failed test case:
When you're running non-regression campaigns, this option can be useful as it will send you a specific notification as soon as one test case is failing (regression). The notification is sent immediately while the session is still running. One notification per failed test case is sent.

Send bug notification only on status change:
This option will limit the email notification to the case where the status of the bug has been changed.

Also, for the integrated bug-tracking database, a new butto allows sending a reminder to all people involved in the bug:

XStudio 1.5: User-groups, projects and access-rights

2011-03-13T00:05:00.007+01:00

XStudio already has a great set of options to authorize users to execute certain actions or not depending on their profile. This is what we call "Actions rights".

With XStudio 1.5, a new feature will allow restricting users to view only a subset of the objects in the Data-Model. For instance, you will want some people to work on a specific project and you want them to be isolated from the rest (you don't want them to see what the others are working on).

To do so, you will be able to create groups of users. Each user group will be granted to access certain folders in ANY XStudio tree. This is what we call "Access rights". If a user is member of several groups, all the rights will be cumulated and given to the user.

Here are a few screenshots:

Members of a user group

Selection of Access Rights for a user group

User group members

XStudio 1.5: Microsoft Word document parser

2011-03-12T23:50:00.007+01:00

Before using a real Test Management system, people often use Microsoft Word to write their requirements or specifications.

With XStudio 1.5, a tool allowing to read all your Word documents and generate an XML file will be delivered. Hence, you will be able to import very simply all your requirements/specifications in XStudio:

Install and run Export from Word v2.0

Indicate a type of export (Flat or Deep)
Indicate the root folder containing all the Word documents
Press "Generate XML"
Open XStudio and import the generated file

1) the "Deep" mode reads the Word document and will interpret any "leaf" tree node as a requirement. All its parent chapters will be imported as folders in the requirement tree
2) the "Flat" mode reads on the summary of the Word document and import any entry (inluding the head numbering)

XStudio 1.5: Import requirements from Word documents

2011-02-21T01:10:00.007+01:00

We are working at the moment on a Word exporter.
The goal is to be able to read/parse some Word document to export requirements in an XML file that XStudio can import.

2 modes are available:
1) the "deep" mode reads the Word document and will interpret any "leaf" tree node as a requirement. All its parent chapters will be imported as folders in the requirement tree
2) the "flat" mode reads on the summary of the Word document and import any entry (inluding the head numbering)

The programs takes 2 arguments:
ExportFromWord.exe inputFolder outputXML

All the Word documents in the input folder are parsed but only one single output XML is generated.

Example:
ExportFromWord.exe "C:\Documents and Settings\egavaldo\Desktop" "C:\exportFromWord.xml"

Study: Risk-based design

2011-02-21T00:52:00.012+01:00

I'm seeing many ways to "implement" risk-based concepts but I have the following requirements and constraints:

- make it simple enough so that people actually use it

- provide a certain level of customization

- must be mapped on XStudio's data-model which is basically:

Now, here is what I’m thinking of:

Business Requirements:

For each requirement, the user will have to provide:

a) a "probability/frequency" P value (between 1 and 5)

b) a "consequence/cost" C value (between 1 and 5)

Then each requirement have a computed "business risk" Br=P*C (between 1 and 25)

Technical Specifications:

For each specification, the user will have to provide:

a) a "technical risk/complexity" R value (between 1 and 10)

Tests:

The system will parse the graph linking each test to its related specifications then its related requirements and will compute a "criticality" Cr value based on some calculation:

Not sure what is the best formula yet though.

(A and B are user-customizable constants.)

Formula#1: no requirement duplicate

This is the simple formula where there is no duplicate.

Formula#2: all path

We could also recursively parse the graph and authorize duplicate requirements:

Output for the user:

This will allow providing many metrics to the user including 2 major ones:

1) a graph showing the distribution of tests (risk / number of test cases)

This is useful to check the correlation between risk and nb of test cases implemented per test (you do want that the more risky tests are tested the more in details so have the more test cases).

2) more importantly, some "risk-weighted-coverage" statistics

The normal coverage metrics give basic information about how much the requirements are covered but it does consider any requirement/specification with the same “weigth”. the "risk-weighted-coverage" would introduce in the calculation the criticality notion. If a highly critical test is succeeding and a low-criticity test is failing, the risk-weighted-coverage will be much better that the opposite way even though there are only 2 test.

Presentation will be as the standard coverage currently displayed.

Many other metrics might be deliverable to the user (need to think more about it).

Release of XStudio 1.4sp5

2011-02-21T00:49:00.003+01:00

XStudio 1.4sp5 has been released today.
This is a maintenane version fixing a couple of bugs releated to JIRA and TRAC connection.

Website: User's Manual being converted to online version

2010-06-30T02:00:00.002+02:00

Good news! Manon started working on the online documentation. This will avoid including a large PDF (more than 100 pages and a few MBytes) in the distribution package. But more importantly, this will allow keeping the documentation maintained/updated much easily and often!
The complete User's Manual should be available in 3 weeks.

Release of XStudio 1.3b2

2010-06-30T01:57:00.001+02:00

Beta 2 of XStudio 1.3 has been released today !

Release of XStudio 1.3b1

2010-06-21T17:00:00.002+02:00

Beta 1 of XStudio 1.3 has been released today !

XStudio 1.3: New test dependencies graph

2010-06-19T01:41:00.006+02:00

In XStudio 1.3, a new test dependencies graph will be released. This is using a new hierarchical layout much clearer than the former circular layout:

When some cycles do exist they are indicated in red/orange so that you can fix them. And again the new layout does a tremendous job:

XStudio 1.3: New manual launcher

2010-06-08T23:04:00.008+02:00

One comment that many people made about the 2 manual launchers already available was: "It's nice but how can I execute some tests, then stop, close XStudio and continue the same session the day after?".
Another question was: "what if I want to run again one individual test already executed a long time ago in a campaign session?".

In XStudio 1.3, we would like to introduce a new manual launcher and a new feature to answer these 2 questions.

1) A third manual launcher
This launcher will be aimed at giving the more flexibility as possible to the operator so that:

he can execute the tests in the order he wants
he can re-run the same tests several times
it's trivial and fast to run the tests

The goal being: "as simple as executing an Excel/Word test plan writing down the result manually with a pen"

2 options:

Option1:
Here is a mock-up of what could be the design of the new launcher:

This solution is pretty elegant and provides a good overview of the current status to the test operator but it's not exactly as simple as running test cases "on paper": The test operator has to:

select a test in the test tree (A)
select a test case in the sub tree (B)
description of the test and test case are updated
click on Succeeded or Failed buttons
use the Previous test case and Next test case buttons
optionally submit a bug or link to an already existing bug, post a comment on the fly

At any time the test operator can select a specific test (A) and test case (B) to execute it (even if already executed). The clock and progress bar are still triggered only if the test has the timeout attribute (no change from the current launchers).

Option2:
The launcher would just display tests and test cases in a table.
Columns would be:

Id
Path
Name
Priority
Result
Comment

The Result column of test case rows would contain a combo-box to manually select the results. This is much simpler but the test operator would have to rely on a printed copy of the test plan. All the results would be updated when the test operator clicks on the Submit button.

Option3:
Maybe something in the middle?

I'm interested in your views about it...

2) An option to re-execute stopped campaign session.
This is a delicate point as cheating must be avoided. Indeed, who never got to the following situation:
A tester runs a very time consuming test campaign (several days) on a product, getting all tests succeeding... except one. Really unfortunate isn't it? Hopefully the day after the tester receives a new version of the SUT and (because he's running out of time) re-runs only the failed test overwriting a failure with a success (hence ignoring all potential risk of regression included in the new version o the SUT) :( It is very common unfortunately and should be authorized only in very specific cases.

So, a new right will be introduced so that the users who are granted with this right are the only ones having the ability to re-run a stopped campaign session.

In a next version, the ability the select a specific version of the test to run will be added. This will be part of the flagging system.