A significant change in TeamCity licensing is worth mentioning. The change is targeted at the Professional Server license users. Starting from version 8.0, Professional Server users can increase the limit of build configurations by 10 with each additional Build Agent license. In other words, the maximum number of build configurations in a Professional Server is calculated as follows: the default 20 +10 * <N obtained Build Agent Licenses>.

Note that a Build Agent license does not give access to TeamCity Email support; for this you still need to buy the TeamCity Enterprise Server license.

Please try this RC and give us your feedback.

Happy building!

Please try it and tell us if some feature does not work for you the way it should or if something needs fixing.

Happy building!

This guest blog post is from Philipp Dolder, Software Architect and Continuous Integration Expert at bbv Software Services AG, a new JetBrains Training and Consulting Partner. Philipp is an enthusiastic software architect, agile coach, continuous integration expert and joint-owner of open source projects FakeItEasy and Appccelerate. Follow him on Twitter @philippdolder.

The views expressed in this blog post are those of the individual, and do not necessarily represent the views of JetBrains.

Do you work with build chains and artifact dependencies? Do you wonder why it takes so much more build time when your artifacts get bigger and bigger?

You probably have the same potential for improvement as I had in my current project. Read on to learn how you can get your builds faster and waste less time waiting for your builds to complete. A fast feedback loop is of major importance, especially in agile software development.

The issue

In the project I’m currently working on, we have a fairly complex build process with many artifact dependencies between the various stages in our build chain. At some time we felt the build started slowing down. In the next retrospective meeting my team decided that I should analyze the build process and implement improvements to get us back to fast build times.

In this post I’ll only write about how I implemented the improvements. Let’s have a look at our build chain first:

In the above diagram you see the 4 stages in our build chain: build, test, installer, publish. You can see that the artifacts of the Integration-build (.NET) are used in four dependent builds: 3 test runs and the installer. These artifacts are about 1.2 GB in size and consist of thousands of dll’s and other small files. Now, keep in mind that the TeamCity system has to copy those artifacts 5 times per build (e.g. 6 GB in small files). Why 5? – You got it. Once from the Integration-build (.NET) build agent to the TeamCity server (artifact publishing) and 4 times to distribute them to the 3 test run and the create installer agents (artifact resolving).

Remember what you learned about reading and writing very small files from and to a hard disk? It is very slow compared to writing one large file with the same total size. Let’s have some numbers to see the impact on our build times: Resolving the build artifacts takes about 20 minutes for each of the 4 builds. So, it takes 80 minutes of CPU time just to resolve the artifacts in our build, which is triggered on every commit. As we are running these 4 builds in parallel, we actually “only” lose 20 minutes with artifact resolving, but it’s still too much. And it causes a higher CPU consumption on our virtualized server environment.

How can we improve and get our build faster?

The approach

Obviously we can save CPU time when we change our artifacts to a single large file. Don’t start changing your build script and use 7-zip command line to achieve that, yet!

Let’s check out what TeamCity has to offer first. You probably read about it once you created your build configurations for the first time. At least that happened to me. I remember having read about zipping artifacts before but never gave a thought about it until now. Let’s get started.

The solution

In my project we build everything into a folder called results inside our working copy. For continuous integration builds we always build in Release mode; therefore, we also have a subfolder Release in our structure. Every C# project (.csproj) builds into its own subfolder which leads to the following folder structure:

How it was before

Our build artifacts for the Integration-build (.NET) should consist of everything inside the results folder. Initially our artifact paths in the General Settings looks like this:

That means everything inside the results folder and its subfolders will be published as build artifact recursively (excluding the results folder itself).

That’s how my build configurations looked like before I started improving build times.

How it is now

We simply need to change the artifact paths of our Integration-build (.NET) build configuration to the following:

This will put everything inside the results folder into the results.zip file.

And in the build configurations depending on the artifacts we simply change the Artifacts rules to the following:

And you’re done.

What is the impact?

When it comes to the build time, we improved the artifact resolving part of our build chain drastically. It was definitely more than I would have imagined. Having about 20 minutes to resolve artifacts for one dependent build before, we are down to about 6 to 7 minutes now. So we just cut the resolving time to a third by such a simple change.

At the same time our artifact zip file shrank to a third of its initial size resulting in about 400 MB now. We also saved space and network traffic on our server.

Conclusion

We saved a lot of CPU time and network bandwidth with such a simple change in our build configuration without having to change our build script at all. But this is only the beginning of improving our continuous integration builds regarding the time it takes us for a complete cycle.

There is even more

This is just an incomplete list of additional options you should consider when further optimizing your build times.

Chain your build configurations properly…
to make sure you benefit from maximum parallelization of builds to have the shortest possible feedback time. Add additional build agents if necessary.

Filter unnecessary files from your artifacts…
when publishing them to minimize overall artifact size. TeamCity documentation provides more details for the configuration of artifact paths and dependencies.

Ensure good network connectivity and use fast hard disks…
to have shortest file transfer times possible.

Ensure fast unit and integration tests…
to avoid unnecessary long build times caused by long running tests that connect to real databases, unit tests that access the file system, etc.

Use TeamCity performance monitor…
to find bottlenecks, understand possible hardware upgrade impact, discover suspicious periods in the build when nothing happens and so on. See Maarten Balliauw’s blog post post for more details.

New make in IDEA Project runner

We integrated a new IntelliJ IDEA external make bringing all of the goodies provided by this tool into TeamCity:

• support for different languages like Groovy, Scala, Closure, etc.
• ability to build Android projects and artifacts, as well as all other types of artifacts supported by IntelliJ IDEA itself
• faster compilation thanks to the highly optimized IntelliJ IDEA make tool, plus the ability to build a project incrementally

Meta-Runner

All meta-runners are now stored on the project level. In other words, you can create a meta-runner which will be available in your project and its sub-projects only and won’t be visible outside. Also we have made several important fixes in the Meta-runner itself.

Other enhancements

There are other improvements including:

• human assignable identifiers for VCS roots;
• new icons in the web interface and IntelliJ plugin;
• and so on.

You are most welcome to try new EAP and tell us about your experience. You can find the full list of changes in the release notes.

Happy building!

This post describes one of the new features in the upcoming TeamCity 8.0

Using TeamCity, we can easily set up a build farm using a central TeamCity server and several connected build agents. By having multiple build agents, we can run builds and build configurations in parallel. Different branches of the same project can build simultaneously on different build agents,  personal builds can be run on one build agent while the nightly build is running on another agent and so on.

But what if we have multiple build agents and we want to constrain parallelism for some projects? What if we do want to have parallelism but want to provide different build variables to each build? Good questions, and the Shared Resources plugin is the answer!

The Shared Resources plugin comes bundled with TeamCity 8.0 EAP. In TeamCity 7.1.x it has to be installed separately. Sources for this plugin are available on GitHub.

Limiting concurrent builds with simple named resources

The Shared Resources plugin works at the project level. Per project, we can specify one or more shared resources which we can use to decide on the number of concurrent builds for a given project and build configuration.

In the Project Home under the Shared Resources tab, we can specify the resources we want to be able to lock. A shared resource should be given a name which can be used later on in our builds. The resource type can be infinite (effectively not limiting build concurrency), a resource with quota and a resource with custom values. Let’s go with resource with quota.

The resource quota is the number of concurrent read locks that can be acquired on the resource. Or in other words: the maximum number of concurrent builds we want to run. Let’s be restrictive and set the quota to 1.

None of the Build Configurations will obey these quota: we have to add a Shared Resources Build Feature to every Build Configuration we want to subject to these quota. This can be done in the Build Configuration. Under Build Features, we can add a Shared Resources build feature which acquires a read or write lock on the shared resource created earlier.

Once added, the build configuration will only be run on one build agent at a time. Personal builds as well as regular builds will all be enqueued and wait for the lock on the shared resource to be released, even if one of the build agents is idle.

Note that it is possible to await multiple locks in a build configuration.

Working with custom resource values

While working with simple named resources may be useful, it may be worth having a custom resource value for every lock acquired. Imagine we have a build which has a custom build step running load tests on a server. Wouldn’t it be nice if we could limit the number of concurrent build based on the number of load test targets available and be able to get the URL to one of these servers in our build? We can using a set of custom resource values.

When creating a new Shared Resource, we can specify the type as a resource with custom values:

In our build, we have several options available to acquire a lock on this shared resource. We can lock on any value that is not locked, we can lock on all values or on a specific value. For our purpose of needing a dedicated load testing machine, we can opt to lock on any available value.

Once added, we will not be able to run more builds concurrently than the number of available values in our custom shared resource. If we need the value of the locked value during our build, we can read it using the %teamcity.locks.<lockType>.<lockName>% variable, %teamcity.locks.readLock.RunTestsAgainstServer% in our case. For example when working with build parameters, we can use the lock value as a custom parameter:

Where can I manage shared resources?

So far we’ve seen that shared resources can be added for every project separately and used in that project’s build configurations. With TeamCity 8.0, we’ve introduced project groups which allows us to group several projects under a project group. Shared resources can be added to both the project group as well as the individual projects. Locks can be acquired for every build configuration in a project and/or project group, making it possible to have shared resources across projects as well.

Do let us know your thoughts via the comments below. Dive in, experiment and… happy building!

An outline of the table of contents is:

• CI Process and TeamCity
• Web Interface Overview
• Build Configurations
• Build Results
• Working with Tests
• Code Analysis
• Working with Changes
• Personal Builds
• Notifications and User Profile

The entire course duration is under an hour. You can view it entirely for free either on JetBrainsTV or on our YouTube Channel. Enjoy!

Many of you have run a custom build at least once. If you’re not sure what a custom build is, you are welcome to visit our documentation for details. In short, a custom build is a build with modified parameters; it can be run from the Custom Build dialog:

Typed parameters is a simple user-friendly approach to customizing the Custom Build dialog behavior. Let’s go over a few scenarios to see how these parameters work.

Scenario 1: Accidental deployments

We’ve all had a few cases when we started a deployment process by mistake, accidentally clicking the Run button. It’s a good thing these deployments didn’t cause any errors or outages, but how do we prevent such behavior in the future? Just follow the steps below:

Go to the Build parameters step:

Click Add new parameter and then the Edit button:

Set Display to prompt. In this case TeamCity will always ask to review the value and choose the Checkbox type.

Next, modify your deployment script to execute only if the value is true and that’s it! No more space for mistakes! Now start a manual build by clicking Run Build.

Scenario 2: Authorization by password

When a third-party provider hosting our service doesn’t provide authentication by key, it means we have to enter the username and password every time we deploy a new version. That’s a pain, since our internal policies allow us to keep the username in the configuration files but not a password. Moreover, we don’t want the password from the server to appear anywhere in the user interface or build logs. Let’s see how we can we handle this.

Add a new parameter – Password, click Edit, choose the Password type:

and click Run.

Problem solved!

Note: TeamCity tries hard not to reveal the actual value of  the password type parameter by filtering it out from the build log and other places. Check our documentation for more details.

Scenario 3: Deployment to different servers

We deploy a new version of our service in three steps:

1. To internal users
2. To beta testers
3. To all users

Our deployment system accepts the stage parameter with the following values: internal, beta and all. If the value is not from the described set, the deployment process will fail. Certainly, a person starting the deployment step can change the parameter’s value in Properties tab, but what if they make a mistake? Or simply forget to change the value? Here is how we can prevent it:

Add a new parameter - stage, click Edit, choose the Select type and add options.

Click Save and start the build.

Select, checkbox and password are not the only possible options: Text field with Regexp pattern validation is also available. Read our documentation to find out the details.

Hopefully, I managed to demonstrate that a custom build backed by typed parameters is indeed a powerful way to set up the process you need. This post is just an introduction which should help you to get to know TeamCity better.

Dig deeper, experiment and … happy building!

]]> http://blogs.jetbrains.com/teamcity/2013/05/07/typed-parameters-and-continuous-deployment/feed/ 0 http://blogs.jetbrains.com/teamcity/2013/05/07/typed-parameters-and-continuous-deployment/ http://feedproxy.google.com/~r/teamcity/~3/Nyp4SUG_PiA/ http://blogs.jetbrains.com/teamcity/2013/04/24/teamcity-8-0-eap-build-27147/#comments Wed, 24 Apr 2013 13:32:13 +0000 Sergey Kotlov http://blogs.jetbrains.com/teamcity/?p=2794 More than a month passed from last TeamCity EAP and we’re glad to disclose what have been done. Multiple improvements were made to the functionality we introduced in previous EAPs.

Project groups

We continue to integrate groups of projects into TeamCity. When new version is out we promise you won’t believe that there was no allusion to the groups in TeamCity just a few months before. Check an incomplete list of the places which are aware of sub-projects now:

• Project hierarchy is shown on Administration|Projects page;
• Projects hierarchy is shown in all drop downs where a list of projects or configurations is displayed;
• Notification rule defined for a project will be effective for sub-projects too;
• It is possible to define cleanup rules for projects. These cleanup rules are default ones for build configurations from this project and its sub-projects.

Get a full list of changes in release notes.

Meta-runner

Meta Runner is a powerful feature providing a promising new way for customizing TeamCity and making it even more people-oriented. To understand it, let’s consider an example.

Imagine you have some repeated task used over and over again across different build configurations. The task is defined as one or more build steps, based on built-in runners, like Ant or command line. Most likely you’d want to reuse this task easily across several build configurations. You can’t do this with templates because template enforces you to share the same settings across different configurations which is not always possible. A solution which we propose is to extract Meta-runner out of these steps, see how it works in release notes.

External IDs and projects transfer

External IDs will help to simplify import/export of projects and build configurations. We introduced them one EAP ago made some improvements from last time:

• it’s possible to define external ID for build configuration and template;
• external ID can be used in HTTP and REST requests;
• external IDs are stored in project configuration files, not internal ones.

Read more about external IDs in release notes.

Disk Usage Report

Disk Usage is now integrated with Server Health. New report, based on disk usage, show configurations with large build log files which are rarely useful because it’s difficult to analyze them. Read more on this report in release notes.

Queued Build page

Dependencies progress has been added and you can see estimates for all dependencies in one place now.

Please, try new EAP and tell us about your experience. You can find a full list of changes in release notes.

Happy building!

First TeamCity 8.0 EAP brought a Mercurial bookmarks support. Let’s see how you can use them with TeamCity.

Mercurial bookmarks are lightweight branches like those you have in Git. While a regular Mercurial branch is a property of a commit and cannot be changed once commit is published, bookmarks can be changed in any way you want: you can add bookmarks to existing commit, you can change the commit a bookmark points to, you can easily delete a bookmark and most importantly you can share them with others. If you are new to bookmarks this tutorial might be useful.

In TeamCity bookmarks can be used in branch and branch specification fields of VCS root settings just like regular branches.

I can see the following uses of bookmarks:

• feature branches with bookmarks
• using bookmarks to narrow changes detected by TeamCity

Feature branches with bookmarks

You can share you work with others (and with TeamCity) by pushing bookmarks to a Mercurial server. There is one caveat though. Since every commit still has a branch, this branch becomes multiheaded. It doesn’t make sence to track such a branch in TeamCity since its current revision is the latest pushed head and it constantly changes. It’s better to create a bookmark, e.g. main, which points to the tip of mainline of development, use it in the branch field and exclude the regular branch in which you create bookmarks from branch specification.

Narrowing detected changes

Let’s say you use regular Mercurial branches to share work with your peers and you have several dozens of branches. Let’s also say that you don’t work on a single branch until it is completely finished, but frequently switch between branches. You have +:* in your branch specification and TeamCity detected all of your branches.

The problem is: your complete build chain, which tests the product on different platforms under different environments, takes half a day to run and you don’t want to waste resources by running build on not yet finished work. On the other hand, you want to publish your work so other can use it. Bookmarks to rescue! All you need is to modify your branch specification to something like +:(*)@ci and publish a bookmark with an appropriate name, such as new.feature@ci, that points to the branch tip once the branch is ready for the build. You can continue to share you work using regular branches and TeamCity will detect only changes you want it to detect.

As you can see, bookmarks can help to implement some useful workflows. All you need to try them is Mercurial 2.4 installed on TeamCity server and of course your Mercurial server should support bookmarks. If you have any other interesting workflows – feel free to share them in the comments. Happy branching!

Release notes
Download page

BTW, expect new 8.0 EAP build in a few days with some interesting features!

Happy building!