studios.thoughtworks.com - Blog

Announcing the ThoughtWorks Studios Community

2009-03-16T20:28:00Z

It is our pleasure to announce that earlier this month, we launched the new ThoughtWorks Studios Community site, as a home on the web for our growing base of users. Our goals for the site are to allow our users to: get help, share best practices, contribute to the future of the product. It is our pleasure to announce that earlier this month, we launched the new ThoughtWorks Studios Community site, as a home on the web for our growing base of users.

Even before our products became publicly available, we had a vision for facilitating a community space where our customers could showcase how our tools were utilized in their environment. As with everything we do, however, we did not want the creation of the space to precede the need, and we focused on having the most basic tools available for our users on day one. We have outgrown our forum toolset and needed a more interactive platform to allow our community members to exchange information.

Our new HiveLive platform has provided us with a number of new tools for enriching the community experience, including the ability to include screenshots, videos, code samples and templates in your posts. We believe that this will lead to many more opportunities for our team and your teams to gain common understanding of each others challenges and accomplishments.

Ultimately, our goals for the site are to allow our users to: get help, share best practices, contribute to the future of the product. We hope that this is the first step towards accomplishing those goals, while allowing our users to show off some of their cool creations.

We are always looking for ways to improve our online experience, so please let us know any ideas you have for making this experience better for you.

Announcing Cruise 1.1

2008-11-27T04:58:00Z

The Cruise team is proud to announce the release of Cruise 1.1, an upgrade to our Continuous Integration and Release Management tool.

This release builds on the unique features of 1.0: deployment pipelines to model your build, deploy, test and release cycle, and the most scalable and easy-to-set-up build cloud on the market. The Cruise team is proud to announce the release of Cruise 1.1, an upgrade to our Continuous Integration and Release Management tool.

This release builds on the unique features of 1.0: deployment pipelines to model your build, deploy, test and release cycle, and the most scalable and easy-to-set-up build cloud on the market.

This release includes a number of features requested by our users, including:

Support for Perforce and Git;
Role-based security;
The ability to lock down who can trigger manually approved stages;
The ability to re-run a stage so you can re-deploy known good versions of your software to your environments, and resolve environmental issues;
Better reporting of server errors;
Logging of source control and publishing activities that occur on the Agent to the Agent console output;
Improvements to the UI;
Significant performance improvements: you can now run thousands of builds a day using our build cloud on commodity hardware with unparalleled performance and reliability.

We've also updated the licensing process so that you don't need to answer so many questions to get your hands on Cruise, and so you get issued a one year 2 agent license right off the bat. Of course if you want a trial license with unlimited agents, you can still email studios@thoughtworks.com to get one.

Please go to our download page to get your free copy of Cruise 1.1. You can read the full release notes on our discussion forum. In our online documentation you'll find a rundown of our new features and an upgrade guide, and on our site you'll find detailed information on how to use pipelines and more on our build cloud.

Thank you for choosing Cruise, and I look forward to your feedback and suggestions.

Phasing out support for Internet Explorer 6 (IE6)

2008-11-24T00:08:00Z

As of January 1st, 2009, we will begin phasing out support for Internet Explorer 6 within our Mingle and Cruise products. What this means is that although we will not intentionally break IE 6 compatibility, we will also not invest significant time or resources into making sure these products are backward compatible with IE 6 after January 1, 2009. As of January 1st, 2009, we will begin phasing out support for Internet Explorer 6 within our Mingle and Cruise products. What this means is that although we will not intentionally break IE 6 compatibility, we will also not invest significant time or resources into making sure these products are backward compatible with IE 6 after January 1, 2009.

The rationale behind this is that IE 6 is a last-generation browser, and Mingle and Cruise are implemented using very modern technology. We have spent a significant investment over the course of the last year trying to make the Mingle and Cruise experience as good for IE 6 users as it is for users of other browsers, however, there are limits to what is possible with this technology without inhibiting progress in advancing the tool for the majority of the customer base. Given that IE 8 is due out next year and IE 7 is over two years old – is proven to have great performance, and is already in the process of roll-out at many of our customer sites that have managed desktops – we feel that it is a prudent time to make this announcement. To clarify, Twist will continue to support the testing of IE6 applications, even though it is not a browser based application itself.

Our efforts for the remainder of this year will be to provide best efforts to resolve IE 6 issues, but we will request that our customers begin looking at preparing to use one of our “recommended” browsers as soon as possible, to reap the benefits of better performance:

Thanks again for your continued support. We look forward to the boost that phasing this technology out of our supported stack will give us. Please feel free to contact us at support@thoughtworks.com should you have any questions about making the transition.

Announcing Mingle 2.1

2008-10-21T12:08:00Z

Hello Friends,

The Mingle team is today pleased to announce the release of Mingle 2.1 an upgrade to our Agile Project Management tool.

In our mission to continuously improve the experience for our customers, we are pleased to bring forward a number of improvements to some of the core functionality that we delivered in our 2.0 release based upon your feedback. In the few short months since Mingle 2.0 was released, we have seen an enormous uptake in the number of organizations using Mingle to collaborate day-to-day in the quest to create better software and find more effective ways to work together. Hello Friends,

The Mingle team is today pleased to announce the release of Mingle 2.1, an upgrade to our Agile Project Management tool.

In our mission to continuously improve the experience for our customers, we are pleased to bring forward a number of improvements to some of the core functionality that we delivered in our 2.0 release based upon your feedback. In the few short months since Mingle 2.0 was released, we have seen an enormous uptake in the number of organizations using Mingle to collaborate day-to-day in the quest to create better software and find more effective ways to work together.

Among the things you can expect in this exciting new release:

New user roles to support Read-Only Project Membership, Light User access, and Anonymous Project access.
Automated transition capabilities to enable workflow by simple gestures on our grid view.
Full support for Card properties to enable greater traceability and flexibility in reporting.
Improved usability and performance in Tree and Hierarchy views to maximize their effectiveness in managing large programs of work.
A full list of What's new in Mingle 2.1 can be found here.

New users are welcome to join us for a free evaluation, and existing customers can enjoy the benefits of this upgrade for their Mingle instance. Just make your way to our download page and get started. Or you can evaluate Mingle without downloading it by choosing from the Test-drive Online and Request a Mingle Hosted Trial options.

One last note to our users: With your support, Mingle has within just over one year forged a global community of users from nearly 200 organizations in Australia, Brazil, Canada, Denmark, France, Germany, India, Ireland, Israel, Japan, New Zealand, Norway, South Africa, Turkey, the United Kingdom and the United States. We say thank you.

Announcing the Twist Public Beta

2008-09-29T10:29:00Z

Today I'm happy to announce that Twist, our exciting new functional and acceptance testing tool, is available for download through our public beta program.

Twist gives you the tool support you need to take your application testing to the next level. ThoughtWorks built Twist to help solve some fundamental issues in test automation. Today I'm happy to announce that Twist, our exciting new functional and acceptance testing tool, is available for download through our public beta program.

Twist gives you the tool support you need to take your application testing to the next level. ThoughtWorks built Twist to help solve some fundamental issues in test automation.

Automated tests are often brittle, especially in rapidly evolving applications. Twist allows you to make controlled changes to your test suite, helping you evolve it with your application. It also provides the tool support you need to make your test suites more modular.
With most existing automated testing frameworks, you need to spend substantial effort to build your own additional frameworks to make managing and writing tests easier. Twist by design helps you build a robust framework in the language of your domain.
Business users are typically disconnected from automated testing efforts. Twist allows you to write tests in English-like rich text specifications, which are readable by your whole team.
Running automated tests in different contexts can be challenging. Twist organizes your tests with a simple yet powerful tagging mechanism.

Twist builds on our legacy of creating tools for functional testing, and it uses popular open source tools like Selenium and Frankenstein under the covers.

Twist is Beta software, so we're working on improving it everyday. We're working on even more features for the commercial release. You can use Twist for the entire period of the beta, with no strings attached. Pricing for the commercial release, which we be available before the end of the year, is here. Please give us your feedback, bug reports and ideas in the forums.

Mingle, Automated Testing, and Cruise

2008-07-30T09:29:00Z

The Mingle team is absolutely crazy about testing and automation. Outside of testing installer integrity and production data migrations, we've totally replaced the manual regression test phase of the release cycle with a massive suite of automated acceptance tests. I'm mostly going to write about how the Mingle team has overcome some of the difficulties teams encounter when working with large acceptance suites, but, before I do that, I want to talk about our attitude towards testing. Why? Because without the right approach to testing, automation is pretty much useless.

Mingle testers start working with new features even before they are built. They participate in the requirement and scope discussions. When development starts, they sit right there with the developers, analysts, and product manager. They do not go up to the 2nd floor to work on 327 Excel worksheets full of test matrices based upon some pre-written specification. Rather, they start testing the actual feature as soon as development begins. This allows our testers to be extremely informed in their testing of features. They're actually talking to the team and they're using the development builds. They have tremendous insight into what's most important and what's most likely to break. They explore the right areas of the application in all the right ways. This allows them to write the correct automated tests. It's one thing to say you automate tests, it's quite another to build a robust, well-organized, well-targeted automated suite that actually gives the entire team confidence. We've done that.

So, back to Mingle's acceptance suite... We use Selenium RC, with tests written in Ruby. At last check, the suite was over 31,000 LOC and required over 12 hours of execution time for a given platform. Multiply that by the number of platforms we support (database, operating system, Java version, ...) and you see a real problem. How do we get feedback to the team in a timely fashion. If it takes a day to get feedback on a commit, the CI (Continuous Integration) cycle is completely broken. The team will stop authoring and maintaining tests. This is always the rub with a large end-to-end acceptance suite. It is slow. The team stops caring. CI practices break down.

The first step for us to get faster feedback was to break the tests up into smaller suites that could be run in parallel. This presented two immediate problems. First, how would we maintain the suites? Second, what CI tool would be able to run the suites in parallel and then roll them up into a single result?

The suite maintenance problem we knew how to tackle. One of our testers had already implemented a means of tagging tests such that we could associate our tests with specific parts of the domain. This made it possible for a developer or tester to run small, targeted sets of tests that correlated to a specific feature. So long as we appropriately tag tests we will always have nice options for grouping tests. To extend this concept to our CI build, we just wrote a little bit of rake code that would take advantage of those same tags to build sensible suites at runtime.

Now, how to run these suites in a CI build? Initially, for each test platform, we setup 8 CC (open source CruiseControl) boxes, each running a different acceptance target. Each of those boxes was writing results to a shared CC Dashboard. 12 hours of tests split across 8 boxes was getting us feedback in approximately 90 minutes for the entire acceptance suite.

This was certainly a huge improvement, but, as you might imagine, this was far from ideal. The different CC projects were building independently and never really rolled up into a single result. There was also no guarantee that the different projects ever ran against the same source. There was quite a bit of detective work when determining a green revision or exactly when the application broke. And we were maintaining a separate CC project (along with its very own server) for each individual acceptance target. And all of this multiplied by the number of platforms we were targeting. Clearly, our CI infrastructure was not good enough.

Around 2 months ago, the Mingle team moved its CI build to new internal Studios build grid that was running early versions of our newest product, Cruise. Given the challenges of the existing infrastructure, you can imagine that the team was quite excited to move to the new product. Today, I can confidently state that Cruise changes the game in a big way for teams running large acceptance suites. Cruise's concepts of pipelines, stages, and jobs allowed us to implement our CI build exactly as we see it. There are no kludges.

To start, we defined a pipeline where our 'Acceptance' stage only runs after the 'Units and Functionals' stage has passed. Cruise makes this part quite easy. We do not need to hack around with special build targets and tasks outside of the tool to implement the pipeline concept. Cruise supports the pipeline as a first-class concept. Better yet, in our 'Acceptance' stage, we can specify any number of the smaller, split-out acceptance suites as the jobs from which that stage is composed. Cruise manages executing those jobs in parallel and rolls up the results to a single result at build plan level. For a given pipeline run, everything builds against the same source. We're no longer wasting time manually rolling up a bunch of loosely-tied results. Cruise is managing and reporting on our build exactly as we need.

Pipelines in Cruise

We've also placed a manually triggered 'Installers' stage after the 'Acceptance' stage in our pipeline. Any team member can look for a label where the 'Acceptance' stage has passed and kick-off the 'Installers' stage. This will make those installers available to the entire team.

The Cruise Grid is a huge advantage as well. Agent resource tagging allows us to more efficiently share the agent hardware across target platforms, e.g., if an agent is tagged as providing 'Ubuntu' and 'Firefox 3' it can pick up any scheduled job that requires those resources. And when the suites grow larger, or we want to test more platforms, it's fairly trivial for us to add agents to the grid and increase our capacity.

The Mingle team is extremely excited about the new Cruise product. This post only begins to dive into what Cruise offers. As you move forward with the pipeline concept and manual gates it's quite easy to see how Cruise can extend typical CI activities all the way into the deployment phase. And the dynamic build properties and RESTful API will allow emergent usages we've yet to even envision.

Announcing Cruise - Continuous Integration and Release Management system

2008-07-28T03:15:00Z

ThoughtWorks Studios is proud to announce that Cruise, our continuous integration and release management product, is available for download and purchase.

Since ThoughtWorks' creation of , the first open source continuous integration server, our consultants have been helping our clients implement the practice of continuous integration. Over the years we've developed a catalogue of principles and practices around build and deployment management to deliver fast, high quality, low risk software releases. All of this collected experience has gone into the development of Cruise. ThoughtWorks Studios is proud to announce that Cruise, our continuous integration and release management product, is available for download and purchase.

Since ThoughtWorks' creation of CruiseControl, the first open source continuous integration server, our consultants have been helping our clients implement the practice of continuous integration. Over the years we've developed a catalogue of principles and practices around build and deployment management to deliver fast, high quality, low risk software releases. All of this collected experience has gone into the development of Cruise.

If you want to just get started with continuous integration, you can get up and running in just a few minutes.

If you're already a continuous integration power user, you'll appreciate features like:

Our zero-configuration build cloud, which means you only configure Cruise in one place. Install Cruise Agents on other computers and they'll pick up configuration and check out code automatically. You can even monitor the status of every Agent and see its live build output on the central dashboard.
Integrated support for build pipelines. Group your build into stages, and see your changes progress through the various tests and environments you define from check-in through to deployment. Manually control the progress of builds through your environments.
Powerful parallelisation. Split long-running builds into jobs, and idle Cruise Agents will pick them up and run them in parallel. Make your tests run faster, and test easily on multiple platforms.

much more

You can get a free trial version of Cruise, which is valid for a month, or buy a license. Academic institutions, non-profits and open source projects can email us for free licenses. Once you're done with the trial, if you don't need our high performance editions, you can still keep on building with our free edition.

We've been using Cruise internally within ThoughtWorks on some of our largest and most critical projects to get their feedback and make sure that we've created a product that addresses the problems that high-performance software delivery teams face today. We hope you will be delighted using Cruise, and we look forward to your feedback as we start work on 1.1.

Metric-driven management versus management-driven metrics

2008-06-24T08:44:00Z

The demand for IT metrics is outstripping IT’s ability to produce them. Part of this is due to the increase in quantitative management practices in business, the trend being to measure everything in sight in an effort to show change in performance over time as well as competitiveness (to the extent possible) relative to commercial peers. But part of this is also self-inflicted: long delivery horizons punctuated by abstract milestones make IT opaque. This puts IT under increased pressure to produce performance and quality metrics. More often than not, the metrics IT come up with contribute more noise than signal.

The root of the problem is how IT goes about its business. Traditional IT abstracts a business need into collections of narrowly defined technical tasks that are performed by technical specialists. The UI might be developed by one set of programmers, middleware by another, back office by yet another. In addition, some roles, notably QA and database administrators, are assigned to support multiple teams simultaneously. The net effect is that people in IT teams don’t work together, they work individually on fragments of business requirements.

To get control over the vast distribution of work, IT tries to measure to the smallest unit of work that it possibly can. This is done by creating project plans with highly detailed inventories of technical tasks, estimating the effort required to complete each, and assigning them to people in very specifically defined specialist roles. The assumption is that completion of all defined tasks will result in a complete business solution. “Progress” toward completion is measured as the percentage of time spent versus the time estimated.

Quality is similarly measured to a finely-grained level of precision. In traditional IT, QA is a discrete phase that follows development, during which very specific test scripts are executed against delivered code. The assumption here is that the software is of sufficient fitness once all test scripts pass. “Quality” is defined largely as the pass rate of test scripts.

Three factors undermine this pursuit of precision. The first is the tragedy of the commons: because everyone is responsible for delivering a project, nobody takes responsibility to see that it’s delivered. In this model, no one person is responsible for completing a business requirement from end-to-end; people are responsible for nothing more than working their particular task to completion and then passing it along. The second is a soft definition of what it means to get work done. “Completion” can be relative, interpretive, or simply a matter of working to a state where nobody can say that a technical task isn’t “done.” The third is an increased number of hand-offs. Handoffs create the risk of misunderstanding. Misunderstanding requires rework.

Thus we may be highly cost efficient at the unit-of-execution level, but horribly cost ineffective at delivering solutions, because the whole is always less than the sum of the parts.

Project management isn’t as precise as it hopes to be. IT is a business of problem solving, not executing to script. In any project, every day we will discover new things that weren’t in the original task order. We will also create work that isn’t on the plan through all the hand-offs. This means that we continuously create work that isn’t tracked. It ends up being “off-balance sheet” to our project, or the shadow matter of our project universe.

Quality management must deal with two problems. First, staging QA to happen only after a long development cycle means there is long latency in feedback. Second, that feedback is of questionable value as it comes as single points of potential failure, not comprehensive feedback on fitness or quality. That is, a test script could fail for any one of a number of reasons: environment, data, incorrect results interpretation or actual software defect being a few among many. At best, all we can say is that the test script did not pass; that is not the same as saying that a test script failed. Each failure report is of limited value and requires re-creation and investigation by somebody else. True product quality may be obfuscated. It also may be incomplete: the universe of test scripts may not in fact test how the software will actually be used; thus “percentage of test scripts passing” may be an inaccurate indicator of fitness.

Worst of all, we’re measuring hours. The business isn’t buying hours, its buying software. These metrics confuse effort for results. They are not the same thing. Ask any CEO which they’d rather be armed with when they go to talk to Wall Street.

Ironically, not only does this approach not unwind after a project failure, it intensifies. When projects fail, the tendency is to double-down: longer time horizons for phases of activity, more explicit role definitions, more precise task definition, more precise estimating, and above all more project data. As the saying goes, “there must be a pony in here somewhere!”

Caught in the middle of this is the project manager. While the increase in project data doesn’t lead to an increase in project information, it does create an increase in effort needed to collect the data. This relegates the project manager to a role of spreadsheet-pusher. The one person on the hook for results spends all of his or her time chasing after the metrics to feed the data collection beast, not doing what a manager is supposed to do: get things done through people. The metrics drive management.

Agile offers a materially different alternative. Agile teams don’t work to an abstract set of tasks, they work to deliver small, discreet chunks of functionality that are recognizable to the business. People in Agile teams don’t work independently, they work as an integrated team, collaborating and pairing to achieve team goals. They also work to a state where code delivered is functionally and technically complete. By making deliveries as often as daily, and immediately sharing those with both business partners and QA, functional completeness is certified as near the moment of creation as possible and in the full context of the business need.

Asset capability, certified to satisfy the business need, is a far more meaningful measure of progress than time spent, obviously because it’s a measure of results. It’s also beneficial to the project manager, who no longer has to translate from one set of abstract measures into another, but can simply report out team status in a common language with the business. Management is no longer at the mercy of metrics, but is master of them.

It also means that “effort” is not a proxy for “results.” This shifts the focus away from the cost of IT, toward an expression of speed. IT has an obsession with “cost per function point,” when it should be obsessed with time-to-live: the length of time it takes for a priority feature to go from idea to production. The shorter the time to get something live, the more competitive the organization. In a time when the cost of capital is rising, it is far more expensive for the business to not have the capability to do something than the cost to rapidly – and competently – deliver it.

This is a critical differentiating point for IT. Providing the lowest cost per function point describes an effective utility, no different from water, electricity or garbage collection. Utilities don’t create business impact. Providing the fastest time to market describes a strategic partner. Being fast to market is far more critical a factor if IT is to be a contributor to business competitiveness.

Usability and Mingle

2008-06-10T09:13:00Z

The usability of Mingle has been the team’ focus from the very beginning and with 2.0 released I thought it might be interesting to take a little time to look at some of the challenges and opportunities we've had along the way.

Firstly, having the usability hat on a team is always interesting - and I'm going to venture that it's doubly interesting on an Agile team. Where each point of effort must be justified in terms of business value, it can be tough to argue for work that isn't always easy to express in those terms. Don't get me wrong - I'm not saying that you can't analyse usability in terms of business value, I'm just saying that if the decision is between having functionality period or having usable functionality, it can be tempting to opt for just functionality. Winning that argument is a lot about the mindset of the team and the sponsors - so I wanted to talk about what we did on Mingle... The usability of Mingle has been the team’s focus from the very beginning and with 2.0 released I thought it might be interesting to take a little time to look at some of the challenges and opportunities we've had along the way.

Firstly, having the usability hat on a team is always interesting - and I have to say that it's doubly interesting when on an Agile team. When each point of effort must be justified in terms of business value, it can be tough to argue on behalf of work that isn't always easy to express in those terms. Don't get me wrong. I'm not saying that you can't analyse usability in terms of business value, I'm just saying that if the decision is between having functionality period or having usable functionality, it can be tempting to opt for just functionality. Winning that argument is a lot about the mindset of the team and the sponsors - so I wanted to talk about what we did on Mingle...

Usability as a team effort: Everyone on the Mingle team uses Mingle day in and day out. Because of the nature of this product, we use it to build it. Typically the development team itself can be one of the worst judges of what a product needs, because in most cases they aren't representative users. But with Mingle that's turned on its head to a certain extent (although not entirely). The upshot is that usability becomes a distributed, team effort. And since we're a pretty hard-to-please bunch, there's no shortage of immediate feedback. Team members spot, for example, where the interface requires them to traipse through multi-click interactions where one would do, and beat me up about it. Use of the forums means this extends beyond the team to our wider user base too.

Storyboards: There's nothing quite like feedback early and often. We make extensive use of MS PowerPoint-based boards to make mock up interaction and interface designs for new features. This doesn't mean we get it right the first time, but it does mean we have something cheap and tangible on the table to help the 'robust' discussions which usually follow. In the first case, the storyboards are reviewed and feedback given only by the immediate team - in the second case, and particularly for bigger features, they are also user tested. We probably go through several rounds of iterative design on the boards before an initial implementable design is settled on.

Guerrilla usability testing: As Nielsen pointed out many moons ago (well..in 1994) you can get tremendous value from a small number of depth interviews when trying to assess the usability of an application. This is the approach we adopt. Usually running for 45 minutes to an hour, each session involves volunteer users from an appropriate user group completing a number of pre-defined tasks set into an overall user scenario. As mentioned, depending on how mature the feature is, the users use either a development version of Mingle or simple MS PowerPoint screens. While having users interact with MS PowerPoint may sound a little bizarre (and I'm not talking about wiring the PPT up here - no hyperlinks, nothing clever, just a deck following a particular path through the app interaction by interaction) I'm always amazed at how quickly users settle into treating it like a real application. This is splendid news for the quality of feedback for such a relatively cheap investment. This generally leaves us with pages of feedback which we must then prioritise and decide what to act on.

We recognise we're never finished: Designs that seemed to work well as boards don't flow as hoped in reality; what was simple to do in PowerPoint is a significant and expensive technical challenge in practice; a new and more streamlined interaction in one area is now inconsistent with an older more clunky interaction in another.The above reasons and more mean that work to revise and tweak the design is an ongoing effort. In particular, as Mingle becomes richer and richer in terms of features, one of our most significant challenges will be avoiding the confusion many interfaces can become as they try to cram more and more options into ever-reducing real estate.

Watch this space to see how we do what we do.

Automated Testing Poll

2008-05-09T22:36:00Z

Our testers at ThoughtWorks use several different automation tools. A lot of people know us for starting the Selenium OSS project, but ThoughtWorkers have actually built several test automation tools over the years, including Sahi, Frankenstein, and White (and now we're building Twist). What tools does your company use? Tell us why and your likes/dislikes in the comments.

<embed src="http://www.polldaddy.com/poll.swf" height="659" width="252"></embed>

Deployment Pipelines: Revolutionizing Release Management

2008-04-30T11:53:00Z

Continuous integration was one of the original twelve XP practices introduced in Kent Beck's book Extreme Programming Explained, back in 1999. It was designed to solve a specific and acute problem in the software development process: making sure that when an individual developer completes new functionality, or makes a change to the code, the software as a whole still works. Continuous integration requires that every time a developer writes some code, he or she builds the software and runs some tests on a non-development environment to make sure that his or her changes haven't broken anything. If done manually this process can be error-prone and time consuming, so it needs to be automated. The upshot is that to do continuous integration well, you need an automated build and a set of automated tests, and once the team is large enough, a continuous integration server to run them for you every time a developer submits code. Thus, CruiseControl and its many siblings were born. Continuous integration was one of the original twelve XP practices introduced in Kent Beck's book Extreme Programming Explained, back in 1999. It was designed to solve a specific and acute problem in the software development process: making sure that when an individual developer completes new functionality, or makes a change to the code, the software as a whole still works. Continuous integration requires that every time a developer writes some code, he or she builds the software and runs some tests on a non-development environment to make sure that his or her changes haven't broken anything. If done manually this process can be error-prone and time consuming, so it needs to be automated. The upshot is that to do continuous integration well, you need an automated build and a set of automated tests, and once the team is large enough, a continuous integration server to run them for you every time a developer submits code. Thus, CruiseControl and its many siblings were born.

However, the problem that continuous integration solves is only a part, albeit an important one, of a greater problem: how to get software that satisfies its functional requirements actually working in production. The part of the software development process between an application being functionally complete and releasing it into a production environment is known as the "last mile" (the opening chapter of the ThoughtWorks Anthology covers this issue in more detail). If sufficient effort has been put into preparing for a production release during the development process, the last mile can take seconds or minutes. However at ThoughtWorks we have been called in to projects where it takes weeks or even months.

This phenomenon has several causes. First of all, any kind of deployment of a complex application — whether or not to a production environment — is difficult. When an application is first deployed to a non-development environment, or when there are significant changes in the application or its environment, it is usual for all kinds of bugs to be exposed. When the application undergoes user testing, further issues are often discovered, such as violations of non-functional requirements. These problems have to be fixed, and the whole process repeated until the application is stable enough to be deployed into production. If deployment to testing environments is hard, it's a good bet that production release will be risky. The options open to you when a production deployment goes wrong are usually very limited — in most cases, backing out is the best option — and downtime costs money. This leads most organizations to be extremely conservative about production releases. This in turn means that releases don't happen very often, which translates to big changes in process between releases, which means the risk doesn't decrease over time. These various factors combine to make releasing applications expensive and slow. Your release process can be the difference between being a market maker and a late entrant.

There is a solution to the problem of the last mile. It is regular, automated deployment and testing of your software in a production-like environment — and ultimately, into production itself. In this paradigm, which we call the deployment pipeline, software is built and then passes through a series of tests — unit tests, functional tests, performance and other non-functional tests and user acceptance testing — before being deployed into staging and then production. The benefits of putting in place an automated deployment pipeline are huge, and ThoughtWorks has been helping companies to solve their last mile problems with this technique for several years. In one organization we turned a deployment process that took days into one that took less than 15 minutes and resulted in split-second downtime, with full rollback, through automating the release process. This had positive knock-on effects throughout the delivery phase of the software lifecycle, making it a push-button process to prepare testing environments. This in turn reduced defects and proved out the deployment process, significantly reducing the risk of performing releases. As a result releases became more frequent and required less effort, freeing up the teams to concentrate on what they wanted to be doing: developing new features.

That's why we at ThoughtWorks Studios have been working to bring you Cruise. We wanted to create a tool that embodies the best practices we have learned through years of solving the last mile problem for our clients. Cruise provides you with an engine that lets you implement deployment pipelines simply and intuitively. It uses a robust, reliable architecture that allows you to scale pipelines across your whole operation by creating fault-tolerant grids of hardware so you can parallelize your long-running builds and test your software on a variety of different platforms. It includes an artifacts repository to manage binaries, reports and other artifacts. And because we are committed to making continuous integration and automated deployment an ubiquitous industry-best practice, we will be offering a one-agent free edition of Cruise which gives you all the features of the commercial edition, as well as free licenses to open source projects, non-profits, and academic institutions.

The other major roadblock to a low-risk deployment is writing a sufficiently comprehensive automated test suite. That's why ThoughtWorks Studios has created Twist, a testing IDE that makes creating functional tests a snap for analysts and business users. This, in combination with Cruise's ability to run test suites in parallel to minimize cycle time, will provide a powerful and simple solution to the problem of the last mile.

Introducing Mingle 2.0

2008-04-15T06:08:00Z

Greetings Minglers New and Old,

It is our pleasure to announce that Mingle 2.0 is now generally available.

The last 9 months have been a labor of love for us, as we have worked to develop a fine piece of software with your input. It makes us proud to say that a broad community of our colleagues and customers from around the world has influenced Mingle through their own project experiences. Greetings Minglers New and Old,

It is our pleasure to announce that Mingle 2.0 is now generally available.

The last 9 months have been a labor of love for us, as we have worked to develop a fine piece of software with your input. It makes us proud to say that a broad community of our colleagues and customers from around the world has influenced Mingle through their own project experiences. With Mingle 2.0 we believe that teams will come up with even more creative ways to interact with each other and change the way that they work.

Among the things you can expect in this exciting new release:

Card Trees, a powerful new mechanism for organizing complex projects.
Formulas and aggregates to derive your team's own custom metrics from your day-to-day work.
RESTful APIs that allow you to communicate with Mingle and create your own solutions.
A full list of What's new in Mingle 2.0 can be found here.

As we are always looking to welcome more people into the fold, we are welcoming anyone to enjoy a 30-day unlimited free trial of Mingle 2.0, even if you have already tried a prior version. Just make your way to our download page and get started.

While you are at it, please take a walk through the newly updated ThoughtWorks Studios website and read about what is new in Mingle 2.0, as well as, our sister products: Twist and Cruise.

We thank those of you that have come along on the journey and helped drive us forward. Because of your participation, Mingle gets better every day.

Forest for the Trees

2008-02-19T06:52:00Z

With Mingle's new feature, Card Trees, we are breaking new ground in how a team can visualize project complexity. We have built into Mingle the power to visually define parent-child relationships between cards on your project and tools to capture metrics at any level and aggregate them according to the hierarchies you design. Because Mingle does the work of crunching the data, you can spend time doing what you do best: working with your team rather than spreadsheets. How many of us have participated in a project in which the estimates made at the beginning of a project no longer hold weight after days or weeks of the project have gone by? Chances are that all of us have. This is generally because a number of things impact estimates, including the team's comfort with each other, familiarity with the technology and understanding of the business domain. We know from experience that trying to predict the size of the system during inception can be fraught with risk. Nonetheless, project stakeholders still ask the critical questions they need answered in order to budget and schedule projects early in the game. Namely: How big is it?

Traditional Agile radiators of team performance such as burn-up or finger charts are focused on progress against a backlog of work and the highlighting of process bottlenecks. What these visualizations do not do is provide us with a picture of the team's overall understanding of the problem space, and thus a feel for how reliable their estimates and progress are. For example, an increase in scope on a burn-up chart could easily be misinterpreted as out-of-control scope, whereas it might be a refinement of previous requirements.

This is where Story Trees become a useful visualization for external stakeholders to better understand scoping efforts.

Those that are familiar with Agile approaches to software development are also familiar with the idea of a user story being a deliverable unit of business value within the context of a system. Using a variety of approaches, teams can gather together a representative set of experts regarding a particular domain and hash out the relevant personas, goals and actions that the system needs to fulfill in order to meet the needs of a particular business function or user community. User stories have presented a very simple mechanism for breaking down a complex domain problem into simpler bits to understand, but they often mask many of the implementation details that contribute to the cost of delivery of that story and as a result, the entire project. Story Trees, whose benefits are also articulated here by Amit Rathore, provide a simple structure for showing how user stories relate in a broader solution.

With Mingle's new feature, Card Trees, we are breaking new ground in how a team can visualize project complexity. First, we have built into Mingle the power to visually define parent-child relationships between cards on your project. What is unique about our approach is that we allow you to have trees for varied perspectives and roles: a project manager may wish to have a planning tree that indicates a work breakdown structure while a business analyst may want to use Story Trees to demonstrate a functional decomposition. At the same time, a QA team may want to break down testing scenarios into more granular cases.

Second, we provide you with tools to capture metrics at any level and aggregate them according to the hierarchies you design. These could be simple metrics like estimates, or calculations like the Time-To-Life of a requirement or defect; in themselves they have meaning, however, totaled or averaged across a planning period, they take on new life.

With these tools, Story Trees become a reality as the team is able to break down complex concepts into manageable units of work. Since Card Trees are implemented in Mingle's flexible work model, they can be easily sorted and reported on as new uses for the data emerge. Because Mingle allows you to define and capture metrics at any point during the project, you add data only as you need it, rather than at the start of the project. Finally, because Mingle does the work of crunching the data, you can spend time doing what you do best: working with your team rather than spreadsheets.

In the next few weeks, with the release of Mingle 2.0, we will invite you to plant your own trees and let us know what takes root.

Bootstrap with a Bootstrapper

2008-01-24T10:32:00Z

CruiseControl Enterprise Best Practices, part 4 In my last post I demonstrated using a bootstrapper to make sure that the configuration for CruiseControl was up to date. I'd like to expand on that in this post. Here's the question that led to the creation of the Bootstrappers in CruiseControl: How do you fix the build when you have broken your build files? CruiseControl Enterprise Best Practices, part 4 In my last post I demonstrated using a bootstrapper to make sure that the configuration for CruiseControl was up to date. I'd like to expand on that in this post. Here's the question that led to the creation of the Bootstrappers in CruiseControl: How do you fix the build when you have broken your build files?

Picture this: you're cheerfully making a change to the build to support a new feature. Your colleague Bob walks up to your desk for a chat about the feature. But he hasn't burst your bubble yet. You were pretty much done anyway, so you check in quickly and turn around to engage Bob in a conversation about work before he can mention fishing.

So eventually Bob wanders over the water cooler to entrap someone else, and you get back to your work. You look over to the screen that displays your build status, and it's red! "Amateurs." you think. "Always breaking the build". You refresh the CruiseControl Dashboard to see which checkins were in that build (everybody checks in frequently here), to find out that the build failed in about a second because the build.xml is invalid.

Suddenly a bit more tolerant of your build breaking colleagues, you look at the build.xml file., which is still open in your IDE. There it is. A dirty great hash symbol at the end of the file. "Curse my keyboard layout!" you exclaim. When you typed the last line of the file, you missed the return key and hit '#'. You didn't see that in your haste to check in, and now the build script won't update the checkout. The build file won't run as it is invalid. The very first thing that it would normally do is update the working copy on the CruiseControl server, to get the latest changes. Looks like you'll need to check in the fix and then get on the CruiseControl machine and update the working copy yourself. D'oh.

Having been there (and sometimes not needing an imaginary colleague to break the build file), I can vouch for the bootstrapper approach:

<?xml version="1.0"?>

<cruisecontrol>

<project name="my_great_app">

<bootstrappers>
<svnbootstrapper localWorkingCopy="${working.dir}/${project.name}" file="build.xml">
<svnbootstrapper localWorkingCopy="${working.dir}/${project.name}" file="ccbuild.xml">
</bootstrappers>

<modificationset quietperiod="30">
<svn LocalWorkingCopy="${working.dir}/${project.name}"/>
</modificationset>

<schedule interval="60">
<ant antWorkingDir="${working.dir}/${project.name}" antscript="${working.dir}/tools/apache-ant-1.6.5/bin/ant" />
<<schedule>

</project>
</cruisecontrol>

In the example above, the bootstrapper will always fetch the latest files from Subversion. Bootstrappers are run before a build takes place, regardless of whether a build is necessary or not, but not if the build is paused. But why stop there? That will make sure that you get the most recent build files, which have responsibility for getting the latest changes from your VCS. But should your build need to do that? No. The bootstrapper can update all the working files, if you configure it like this:

<bootstrappers>
<svnbootstrapper localWorkingCopy="${working.dir}/${project.name}"/>
</bootstrappers>

Now CruiseControl will update all the recently changed files in the working copy on your server. There used to be an issue with this approach - Bob will explain: "It's like the time I caught a 10 pound marlin, but the line snapped. I checked in my change, saw a build get triggered shortly afterwards, and I waited for the build to complete. My change was there in the list of modifications, but the QA team said that change wasn't in that build!"

This problem used to happen if you checked in a change while CruiseControl was bootstrapping and fetching the latest revision. Because of CruiseControl's history with non-atomic version control systems, it internally refers to change sets by date. The CCE team at ThoughtWorks Studios have fixed this in CruiseControl 2.7.1 by adding the useLocalRevision to the Subversion modification set:

<modificationset quietperiod="0">
<svn LocalWorkingCopy="/etc/cruisecontrol" useLocalRevision="true"/>
</modificationset>

Once the bootstrapper has updated the project's working copy on the CruiseControl server, it makes the Subversion revision number of the working copy available to the ModificationSet. When the ModificationSet attempts to work out what changes are new in that build, it then constrains it's query to the revision number of the working copy. This makes CruiseControl accurately report changes. Its a huge boost to larger projects that use CruiseControl. This will also save you from more conversations with Bob. Another useful side effect of the useLocalRevision property with Subversion is that the Subversion reposistory revision number gets passed to the build as a property called${svnrevision}. This makes it trivial to tag the build later at the correct revision.

While we're on the subject of tweaks for CruiseControl under Subversion (or any truly atomic VCS): never use a QuietPeriod. The QuietPeriod is there to stop CruiseControl prematurely building when you're using CVS or another non-atomic VCS. It will only slow down your build.

There are more bootstrappers available to play with for fun and profit: CruiseControl supports many Version Control Systems and most of the VCS plugins have an optional bootstrapper. Bob and I have explained the big picture on VCS bootstrappers above, so I'll outline the rest of them here:

AntBootStrapper will execute an Apache Ant file before the project builds. Most of the time you shouldn't need it, but I have seen it used to fetch dependencies and then trigger a build if those dependencies have changed.
ExecBootStrapper will execute any script or command that you tell it to.
LockFileBootStrapper is an interesting one. Read on for more.

Do you have many projects on the same CruiseControl server? Good. Do you have two or more Builder Threads configured? Do you build two or more projects at once? Great. Multiple, simultaneous projects is something that CruiseControl has supported since the summer of 2004. Do 2 of those projects sometimes compete for the same resources?. Oh my, that's bad. Especially annoying if it only happens once a day or so. Here's what you do:
<cruisecontrol>
<system>
<configuration>
<threads count="2" />
</configuration>
</system>

<project name="test-1" >
<listeners>
<lockfilelistener lockFile="/tmp/lock" projectName="${project.name}"/>
</listeners>
<bootstrappers>
<lockfilebootstrapper lockFile="/tmp/lock" projectName="${project.name}" />
</bootstrappers>
<schedule interval="30">
<exec command="/bin/true" />
</schedule>
</project>

<project name="test-2" >
<listeners>
<lockfilelistener lockFile="/tmp/lock" projectName="${project.name}"/>
</listeners>
<bootstrappers>
<lockfilebootstrapper lockFile="/tmp/lock" projectName="${project.name}" />
</bootstrappers>
<schedule interval="30">
<exec command="/bin/true" />
</schedule>
</project>

</cruisecontrol>

The lockfilebootstrapper will run before the build, like a bootstrapper should. It will look for the lockfile for the name of the of the project. If the lockfile exists and the name of the project inside doesn't match the expectation set in the configuration, it will cancel the entire build attempt. This way, one project can warn all the others that it is building, so you never see the two projects that don't sit well together executing at the same time. However, you could have 5 other builds that don't share the same external dependencies that the first project does, and they can run just fine at the same time. The lockfilelistener will cleanup any old lock files left oiver after the build, so you don't prevent some builds running all the time.

"Hang on," says Bob: "I just had to wait 10 minutes for the FooLib project to build! What's going on?." "Now Bob." you say, not unkindly: "You know there's only one integration test server. Now, so does the CruiseControl build. Next time you go fishing with the CTO, you can tell him about the delays to the project and ask if we can spend some of next year's budget on for another test server."

©Julian Simpson 2007. All rights reserved.

The state of the art of Functional Testing

2007-11-28T09:26:00Z

At ThoughtWorks, we’ve had a strong tradition of innovation with functional testing. We’ve always viewed testing as a collaborative, team activity, involving domain experts, developers, and testers. However, we believe that we have a long way to go before we achieve testing nirvana. The current set of functional testing tools all have their own strengths, weaknesses, and gotchas. Over the past decade, development tools have got more and more sophisticated. In contrast, however, functional testing tools seem to have missed the party. We believe that the time is right for a next generation functional testing tool. A team at ThoughtWorks Studios has been hard at work, and we believe we do have something that pushes the envelope of functional testing.

At ThoughtWorks, we’ve had a strong tradition of innovation with functional testing, with tools like Selenium, Sahi and Frankenstein. A precursor to FIT was born on one of our projects (incidentally, this was also the project where CruiseControl was born). We’ve always viewed testing as a collaborative, team activity, involving domain experts, developers, and testers. However, we believe that we have a long way to go before we achieve testing nirvana.

The current set of functional testing tools all have their own strengths, weaknesses, and gotchas. With GUI drivers such as Selenium, for example, it is quite critical to build abstractions. Without abstraction, test suites tend to rapidly become unmaintainable. Test code is code, after all – and it needs the same tender loving care that production code does. (I will be going into more detail about this in my future posts). Doug Seller’s talk at the Google Testing Conference is a great example of what’s possible with the right level of abstraction.

Tools from the FIT family are great for expressing intent, which aids collaboration between domain experts and the development team. However, the cost of maintaining these tests can often be prohibitively high. Nat Pryce has a great post on the issues to consider while using FIT here. FIT often isn’t used appropriately – James Shore’s rant about misusing FIT goes into a lot more detail about how this tends to happen.

Over the past decade, development tools have got more and more sophisticated. With automated refactoring support, tool integration, and extensibility, IDEs such as IntelliJ and Eclipse make it possible to work with large code bases. In contrast, however, functional testing tools seem to have missed the party.

We believe that the time is right for a next generation functional testing tool. The Agile community recently held a workshop on next generation functional testing tools. Meanwhile, a team at ThoughtWorks Studios has been hard at work too, and we believe we do have something that pushes the envelope of functional testing. We want to make higher levels of abstraction easier to create. We want testers to be able to richly express their intent. And, we’d like to provide the tool support to help teams do all that while treating tests like code. Stay tuned, and fasten your seat belts!