• Gradle Kotlin plugin allows to compile Kotlin sources using Gradle. It is also possible to compile Kotlin and Java/Groovy/Scala sources kept in the same module, see the documentation for more details.
• Gradle TeamCity plugin allows to build and archive TeamCity plugins using Gradle. Following the long-awaited need, TeamCity dependencies were deployed to a Maven repository at Artifactory Online. So working on TeamCity plugins should be much easier now, at least from the build point of view. To make sure it works I’ve ported builds of teamcity-artifactory-plugin and teamcity-nuget-support to Gradle.
• TeamCity Console plugin is a Groovy console for TeamCity admins. Think Jenkins Script Console or Groovy Web Console but for TeamCity. It allows you to run a Groovy script in TeamCity run-time environment and displays a content of all Spring contexts linked to Open API Javadocs, letting you see what’s in there and what can be injected into your Spring beans.

Try them out and let me know how it works. Bugs, feedbacks, complains, suggestions – I’m all ears.

Build Configuration “id”

Before you retrieve artifacts of any build configuration you need to know its "id" which can be seen in a browser when corresponding configuration is browsed. Let’s take IntelliJ IDEA Community Edition project hosted at teamcity.jetbrains.com as an example. Its “Community Dist” build configuration provides a number of artifacts which we’re going to play with. And as can be seen on the screenshot below, its "id" is "bt343".

HTTP

Anonymous HTTP access is probably the easiest way to fetch TeamCity artifacts, the URL to do so is:

http://server/guestAuth/repository/download/<btN>/<buildNumber>/<artifactName>

Fot this request to work 3 parameters need to be specified:

Ant + Ivy

All artifacts published to TeamCity are accompanied by "teamcity-ivy.xml" Ivy descriptor, effectively making TeamCity an Ivy repository. The code below downloads "core/annotations.jar" from IDEA distribution to "download/ivy" directory:

"ivyconf.xml"

<ivysettings>
    <settings defaultResolver='teamcity-repo'/>
    <resolvers>
        <url name='teamcity-repo' alwaysCheckExactRevision='yes' checkmodified='true'>
            <ivy      pattern='http://teamcity.jetbrains.com/guestAuth/repository/download/[module]/[revision]/teamcity-ivy.xml'/>
            <artifact pattern='http://teamcity.jetbrains.com/guestAuth/repository/download/[module]/[revision]/[artifact](.[ext])'/>
        </url>
    </resolvers>
</ivysettings>

"ivy.xml"

<ivy-module version="1.3">
    <info organisation="com.jetbrains" module="idea"/>
    <dependencies>
        <dependency org="org" name="bt343" rev="lastSuccessful">
            <include name="core/annotations" ext="jar"/>
        </dependency>
    </dependencies>
</ivy-module>

"build.xml"

<project name="teamcity-download" default="download" xmlns:ivy="antlib:org.apache.ivy.ant">
    <target name="download" xmlns:ivy="antlib:org.apache.ivy.ant">
        <taskdef uri="antlib:org.apache.ivy.ant" resource="org/apache/ivy/ant/antlib.xml"/>
        <ivy:configure file    = "${basedir}/ivyconf.xml"/>
        <ivy:resolve   file    = "${basedir}/ivy.xml"/>
        <ivy:retrieve  pattern = "${basedir}/download/ivy/[artifact].[ext]"/>
    </target>
</project>

Gradle

Identically to Ivy example above it is fairly easy to retrieve TeamCity artifacts with Gradle due to its built-in Ivy support. In addition to downloading the same jar file to "download/gradle" directory with a custom Gradle task let’s use it as "compile" dependency for our Java class, importing IDEA’s @NotNull annotation:

"Test.java"

import org.jetbrains.annotations.NotNull;

public class Test
{
    private final String data;
    public Test ( @NotNull String data ){ this.data = data; }
}

"build.gradle"

apply plugin: 'java'

repositories {
    ivy {
        ivyPattern      'http://teamcity.jetbrains.com/guestAuth/repository/download/[module]/[revision]/teamcity-ivy.xml'
        artifactPattern 'http://teamcity.jetbrains.com/guestAuth/repository/download/[module]/[revision]/[artifact](.[ext])'
    }
}

dependencies {
    compile ( 'org:bt343:lastSuccessful' ){
        artifact {
            name = 'core/annotations'
            type = 'jar'
        }
    }
}

task copyJar( type: Copy ) {
    from configurations.compile
    into "${ project.projectDir }/download/gradle"
}

Maven

The best way to use Maven with TeamCity is by setting up an Artifactory repository manager and its TeamCity plugin. This way artifacts produced by your builds are nicely deployed to Artifactory and can be served from there as from any other remote Maven repository.

However, you can still use TeamCity artifacts in Maven without any additional setups. "ivy-maven-plugin" bridges two worlds allowing you to plug Ivy resolvers into Maven’s runtime environment, download dependencies required and add them to corresponding "compile" or "test" scopes.

Let’s compile the same Java source from the Gradle example but using Maven this time.

"pom.xml"

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns              = "http://maven.apache.org/POM/4.0.0"
         xmlns:xsi          = "http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation = "http://maven.apache.org/POM/4.0.0

http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <groupId>com.test</groupId>
    <artifactId>maven</artifactId>
    <packaging>jar</packaging>
    <version>0.1-SNAPSHOT</version>
    <name>[${project.groupId}:${project.artifactId}:${project.version}]</name>
    <description>Ivy Maven plugin example</description>

    <build>
        <plugins>
            <plugin>
                <groupId>com.github.goldin</groupId>
                <artifactId>ivy-maven-plugin</artifactId>
                <version>0.2.5</version>
                <executions>
                    <execution>
                        <id>get-ivy-artifacts</id>
                        <goals>
                            <goal>ivy</goal>
                        </goals>
                        <phase>initialize</phase>
                        <configuration>
                            <ivyconf>${project.basedir}/ivyconf.xml</ivyconf>
                            <ivy>${project.basedir}/ivy.xml</ivy>
                            <dir>${project.basedir}/download/maven</dir>
                            <scope>compile</scope>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

When this plugin runs it resolves IDEA annotations artifact using the same "ivyconf.xml" and "ivy.xml" files we’ve seen previously, copies it to "download/maven" directory and adds to "compile" scope so our Java sources can compile.

GitHub Project

All examples demonstrated are available in my GitHub project. Feel free to clone and run it:

git clone git://github.com/evgeny-goldin/teamcity-download-examples.git
cd teamcity-download-examples
chmod +x run.sh dist/ant/bin/ant gradlew dist/maven/bin/mvn
./run.sh

Resources

The links below can provide you with more details:

• TeamCity – Patterns For Accessing Build Artifacts
• TeamCity – Accessing Server by HTTP
• TeamCity – Configuring Artifact Dependencies Using Ant Build Script
• Gradle – Ivy repositories
• "ivy-maven-plugin"

That’s it, you’ve seen it – TeamCity artifacts are perfectly accessible using either of 4 ways: direct HTTP access, Ant + Ivy, Gradle or Maven. Which one do you use? Let me know!

Here it goes:

• My Artifactory repository has moved from evgeny-goldin.org/artifactory to Artifactory Online at evgenyg.artifactoryonline.com. What this means for you is that now the repository is not expected to ever go down, unlike the previous situation where I couldn’t provide any uptime guarantee due to upgrade downtimes.
• But it doesn’t end there! Starting from this version, all releases are now available at Maven Central as well. I’ve been approached by quite a few people asking for this and even though I still believe it’s perfectly Ok for each project to run its own repository, I also understand those who are unwilling or unable to connect to any repository except Maven Central. So we have both now. I’m not going to publish snapshot builds to Maven Central, neither am I going to publish my other projects there yet, something that may change in the future.
• Starting from this version all plugins run only with Maven 3. Supporting Maven 2 has prevented me from using Aether library and has contributed to a messy code so I’ve decided to cut it this time. For those still running on Maven 2, you can use version “0.2.3.8″ but I strongly suggest that you upgrade.
• Plugins coordinates have changed to provide a Maven Central compatibility. They are now "com.github.goldin:something-maven-plugin".
• Java 7 is now fully supported. I run my tests with Sun’s Linux JDK but please, let me know if you experience any problems with OpenJDK.

As for the actual features implemented in this version, here’s a complete list of the issues addressed. Allow me to highlight just some of them:

• "ivy-maven-plugin" is a new plugin, allowing you to plug Ivy resolvers into Maven and behave similarly to traditional Maven <dependencies> by adding artifacts resolved into Maven scopes such as "compile" or "test". It expands the Maven boundaries and makes it possible to resolve artifacts from anywhere Ivy resolvers can get to: TeamCity build artifacts, local file system, remote URLs, etc. In short, any storage that is not compatible with Maven can be used with "ivy-maven-plugin" as long as you can configure a corresponding Ivy resolver. Now, how cool is that?!
• "jenkins-maven-plugin" now supports my favorite 3G technologies stack: Gradle, Groovy and GitHub together with bringing in Mercurial to the party. And there are many more “extension points” now, check this out: it is possible to “patch” XML configs with Groovy snippets and provide your own <task>, <prebuilders> and <postbuilders>.
• "copy-maven-plugin" has taken full advantage of the Aether library, as I’ve mentioned previously. Transitive dependencies resolution was re-implemented and, as a result, it has become significantly more capable. I’ll address this issue in a separate blog post. Build performance was one of the goals in this version and it brought a new option: skip copying and packing operations whenever possible which usually results in noticeable speedups.

Now, I wouldn’t be able to make this update available without your help.

I’m very thankful to Liya Katz from Tikal for bringing up lot’s of cool ideas to make the "copy-maven-plugin" better and being patient enough to go through several beta versions :) I would also like to say “Thank you” to all those people who have suggested new features, submitted bug reports and tried out snapshot builds. Your feedback is essential so, please, keep it coming!

I would like to thank JFrog for making the Artifactory Online service available and keeping it up 24×7. Thank you, guys! I’m glad to see major open-source projects such as Spring, Scala, Gradle, Grails and Jenkins switching over to Artifactory Online, and now with the JetBrains running its own Artifactory instance as well.

Speaking of JetBrains, I would like to thank the YouTrack team for granting me an open-source license to run evgeny-goldin.org/youtrack without having to worry about the number of registered users. And for making such a damn good product, of course!

Stay tuned, more releases are coming.

Luckily, TeamCity allows your build script to interact with the server using Service Messages. When you build script prints out the following command, TeamCity reads it and behaves accordingly.

This way it is possible to report a compilation or testing results, publish artifacts (you can find "##teamcity[publishArtifacts" messages in IntelliJ IDEA build log), and even update build parameters on the fly. However, what I was interested in is build progress report:

Adding a single println to a Maven build can be done with a bit of Groovy scripting:

<plugin>
  <groupId>org.codehaus.gmaven</groupId>
  <artifactId>gmaven-plugin</artifactId>
  <version>1.4</version>
  <executions>
    <execution>
      <id>display-progress-report</id>
      <phase>initialize</phase>
      <goals>
        <goal>execute</goal>
      </goals>
      <configuration>
        <providerSelection>1.8</providerSelection>
        <source>
        if ( System.getProperty( 'teamcity.version' ))
        {
          println "##teamcity[progressMessage '${project.groupId}:${project.artifactId}']"
        }
        </source>
      </configuration>
    </execution>
  </executions>
</plugin>

After adding this snippet to project's parent POM, the progress indicator becomes (see it live!)

And I'm now working on making this functionality built-in for TeamCity 7.1.

Same trick can be used in Gradle, but it is less required since TeamCity already reports Gradle tasks progress. Anyway, if you're interested, grab this code:

allprojects {
    ...
    if ( project.properties.teamcity )
    {
        tasks.all { it.doFirst { 
            logger.lifecycle( "##teamcity[progressMessage ':${project.name}:${it.name}']" )
        }}
    }
    ...
}

Part 1 – Maven Build Dependencies.
Part 2 – TeamCity Build Dependencies.
Part 3 – Artifactory Build Dependencies in TeamCity.

Introduction

The subject of build dependencies is neither a trivial nor a minor one. Various build tools approach this subject from different perspectives contributing various solutions, each with its own strengths and weaknesses.

Maven and Gradle users who are familiar with release and snapshot dependencies may not know about TeamCity snapshot dependencies or assume they’re somehow related to Maven (which isn’t true). TeamCity users who are familiar with artifact and snapshot dependencies may not know that adding an Artifactory plugin allows them to use artifact and build dependencies as well, on top of those provided by TeamCity.

Some of the names mentioned above seem not to be established enough while others may require a discussion about their usage patterns. Having this in mind I’ve decided to explore each solution in its own blog post, setting a goal of providing enough information so that people can choose what works best.

The first post explored Maven snapshot and release dependencies. This is the second post, which covers artifact and snapshot dependencies provided by TeamCity and the third and final part will cover the artifact and build dependencies provided by TeamCity Artifactory plugin.

Non-Maven Dependencies

While Maven-based dependencies management and artifact repositories are very common and widespread in Java, there are cases where you may still find them insufficient or inadequate for your needs. For starters, you may not be developing in Java or perhaps your build tool is not providing built-in integration with Maven repositories, as is the case with Ant (or its Gant and NAnt spin-offs), SCons, Rake or MSBuild. Secondly, snapshot Maven dependencies provide their own set of challenges covered in the previous blog post, making it harder to ensure correct snapshot dependency is used in a chain of builds.

In order to address these scenarios, TeamCity provides two ways to connect dependent build configurations and their outcomes: artifact and snapshot dependencies.

TeamCity Artifact Dependencies

The idea of artifact dependencies in TeamCity is very simple: download the artifacts produced by an other build before the current one begins. After the artifacts are downloaded to the folder specified (checkout directory by default), your build script can use them to achieve its goals. You can find configuration details in TeamCity documentation.

Naturally, this scheme is not suitable for build tools with automatic dependencies management, but it works well with build or shell scripts accepting and expecting local paths, relative to the checkout directory. Note that the copying works not only for the produced build binaries, but for any kind of binary or text files, like the TeamCity coverage report as demonstrated on the screenshot above.

There is one important detail about specifying artifact dependencies and that is “Get artifacts from” configuration where you specify what type of build should files be taken from. Possible values of this field are “last successful”, “finished”, “pinned”, or “tagged build”, as well as the build number or “Build from the same chain”. While most values should be trivial to understand with “Last successful build” being the default and generally suitable option, the definition of “same chain” build is directly related to TeamCity snapshot dependencies.

TeamCity Snapshot Dependencies

Imagine a monolithic multi-step build process (build, test, package, deploy) which you decide to split into multiple smaller builds, invoked sequentially, forming a chain of executions. Doing so allows one to configure or trigger every chain step separately and run certain steps in parallel in order to speedup the process (like executing tests or building independent components). Most of all, it makes the overall maintenance significantly easier. However, while doing so you need to ensure every chain step uses the same consistent set of sources pulled from VCS even if newer commits are made all the while chain steps are running. That’s what TeamCity snapshot dependencies are for: they connect several build configurations into a single chain of execution, called build chain, with every step using the same set of sources, regardless of VCS updates. Note that the TeamCity use of the term “snapshot dependencies” may confuse people familiar with Maven snapshot dependencies which are two unrelated concepts.

Snapshot dependencies are configured similarly to artifact dependencies. You can find configuration details in TeamCity documentation.

Using Artifact and Snapshot Dependencies Together

When applicable, it is recommended to define both kinds of dependencies between build configurations, as this ensures not only a consistent set of sources used throughout a chain steps but also a consistent flow of artifacts produced. Now the definition of “Build from the same chain” in artifact dependency mentioned above becomes clear, as this is the only meaningful option in this scenario.

In a way, you can think of build chain steps running in isolation from VCS updates after the first sources’ “snapshot” is taken. Chain artifacts are either re-created from the same sources or passed through chain steps with artifact dependencies. This makes chain steps consistent, reproducible and always up-to-date (when applied to using chain artifacts), something that can’t be easily achieved with Maven snapshot dependencies.

Build Chains Visibility in TeamCity 7.0

TeamCity 7.0 took the notion of build chains to a whole new level by providing build chains a new UI, making chain steps visible and re-runnable. Once you have snapshot dependencies defined, a new “Build Chains” tab appears in project reports, providing a visual representation of all related build chains and a way to re-run any chain step manually, using the same set of sources pulled originally.

Build Chain Triggering

Having build configurations connected with snapshot dependencies and, therefore, their builds grouped into build chains not only makes them more consistent regarding the sources used, it also impacts the way builds are added to the build queue: after a certain chain step is triggered, the default behavior is to add all preceding chain steps as well, keeping their respective order, in addition to the one that was triggered initially. Let me repeat it for more clarity: triggering certain chain configuration adds preceding (those to the left of it) and not subsequent (to the right of it) configurations to the build queue, although it may seem counterintuitive at first. The idea is to mark the location where chain execution stops, which is exactly the configuration that was triggered initially; it becomes the last execution step.

To trigger subsequent chain steps upon VCS changes found in a chain configuration, you can add a VCS trigger with the “Trigger on changes in snapshot dependencies” option to the configuration that would be the last execution step. This configuration is then triggered whenever any of the preceding chain steps is updated, which schedules the whole chain for execution.

Having this behavior in mind, you therefore need to decide which configurations are triggered automatically and which should be run manually. Usually, earlier chain steps having no impact on external environment can be triggered automatically by VCS trigger but final chain steps, potentially modifying external systems, are invoked manually after a human verification of the previous chain results. The process of running the final chain steps manually is usually referred to as “promoting” previously finished builds.

Sample Build Chain: Compile, Test, Deploy

Imagine three sample build configurations, "Compile", "Test" and "Deploy" connected into a build chain: "Deploy" is snapshot dependent on "Test" which is snapshot dependent on "Compile".

In this sample scenario the "Compile" and "Test" configurations are triggered automatically while "Deploy" is triggered manually, following the recommendations given above. VCS changes in "Compile" configuration only trigger an execution of this chain step, while VCS changes in "Test" configuration trigger "Compile" and "Test" execution (in that order).

Once a "Compile" configuration is added to the builds queue, its sources’ timestamp is recorded on the server to be used in all subsequent chain steps. If any of the chain steps is connected to a different VCS root, its sources are also pulled according to the same timestamp.

Promoting Finished Builds

As soon as the automatic chain execution stops (after running "Test"), you can continue it by clicking the corresponding “Run” button on the "Deploy" configuration that was not triggered (see the build chain screenshot above). Alternatively, it is possible to promote a finished "Test" build through its “Build Actions” and invoke configurations which are snapshot dependent on it – "Deploy" configuration in this case.

Summary

This article has provided an overview of TeamCity artifact and snapshot dependencies, build chains, how their steps are triggered and how finished builds are promoted. I hope you now have a good understanding of how it works and of when it is appropriate (or not) to use TeamCity build dependencies in addition to those provided by build tools such as Maven.

Please, refer to the TeamCity documentation for more information about this subject:

• Dependent Build
• Build Chain

The final blog post in the series will uncover how you can use the TeamCity Artifactory plugin in order to achieve a behavior which is similar to build chains for projects with Maven-based dependency management. Stay tuned!

Part 1 – Maven Build Dependencies.
Part 2 – TeamCity Build Dependencies.
Part 3 – Artifactory Build Dependencies in TeamCity.

Introduction

The subject of build dependencies is neither a trivial nor a minor one. Various build tools approach this subject from different perspectives contributing various solutions, each with its own strengths and weaknesses.

Maven and Gradle users who are familiar with release and snapshot dependencies may not know about TeamCity snapshot dependencies or assume they’re somehow related to Maven (which isn’t true). TeamCity users who are familiar with artifact and snapshot dependencies may not know that adding an Artifactory plugin allows them to use artifact and build dependencies as well, on top of those provided by TeamCity.

Some of the names mentioned above seem not to be established enough while others may require a discussion about their usage patterns. Having this in mind I’ve decided to explore each solution in its own blog post, setting a goal of providing enough information so that people can choose what works best.

This first post explores Maven snapshot and release dependencies. The second post covers artifact and snapshot dependencies provided by TeamCity, and the third and final part will cover artifact and build dependencies provided by the TeamCity Artifactory plugin.

Internal and External Dependencies

Build processes may run in total isolation by checking out the entire code base and building an application from scratch. This is the case for projects where relevant binary dependencies (if there are any) are kept in VCS together with the project sources. However, in many other cases build scripts rely on internal or external dependencies of some sort.

Internal dependencies are satisfied by our own code where we have full control over the project which can be split into multiple modules or sub-projects. External dependencies are satisfied by someone else’s code (on which we have no control) and we consume it or use it as clients. This can be a third-party library such as Spring or a component developed by another team.

This distinction is important since internal and external dependencies are usually accompanied by different release and upgrade cycles: internal dependencies may be modified, rebuilt and updated on an hourly basis while external dependencies’ release cycle is significantly slower with users applying the updates even less frequently, if at all. This is largely driven by the fact that internal dependencies are under our own control and have a narrow-scoped impact, limited by a specific project or module while external dependencies can only be used as-is, their impact is potentially company or world-wide, they are not scoped by any project and can be used anywhere. Naturally, this requires significantly higher standards of release stability, compatibility and maturity, hence slower release and update cycles.

Another aspect of “internal vs. external” dependency characteristics is expressed in how their versions are specified in a build script. Internal dependencies are usually defined using snapshot versions while external dependencies use release versions. The definition of “snapshot” and “release” versions was coined by Maven, which pioneered the idea of managing dependencies by a build tool. If you’re familiar with automatic dependencies management feel free to skip the following section which provides a quick overview of how it works.

Automatic Dependencies Management

In Maven dependencies are specified declaratively in a build script, an approach later followed by newer build tools such as Gradle, Buildr and sbt.

Maven:

<dependency>
    <groupId>org.codehaus.groovy</groupId>
    <artifactId>groovy-all</artifactId>
    <version>1.8.6</version>
    <scope>compile</scope>
</dependency>

Gradle:

compile "org.codehaus.groovy:groovy-all:1.8.6"

Buildr:

compile.with "org.apache.axis2:axis2:jar:1.6.1"

sbt:

libraryDependencies += "org.twitter4j"  % "twitter4j-core"  % "2.2.5"

Every dependency is identified by its coordinates and scope. Coordinates unambiguously specify the library and version used, while scope defines its visibility and availability in build tasks such as compilation or tests invocation.

For instance, "compile org.codehaus.groovy:groovy-all:1.8.6" would designate a Groovy "org.codehaus.groovy:groovy-all" distribution for version "1.8.6", used for source compilation and test invocation. Switching the scope to “test” or “runtime” would then narrow down the library visibility to tests-only or runtime-only, respectively.

When a build starts, dependencies are either located in a local artifacts repository managed by a build tool (similar to a browser cache) or downloaded from remote repositories, either public or private, such as Maven Central, Artifactory or Nexus. The build tool then adds the artifacts resolved to the corresponding classpaths according to their scopes. When assembling build artifacts, such as "*.war" or "*.ear" archives, all required dependencies are correctly handled and packaged as well.

Though dependencies management seems to be an essential part of almost any build, not all build tools provide a built-in support for it: Ant and MSBuild lack this capability, a gap later addressed by Ivy and NuGet to some extent. However, Ivy’s adoption was slower compared to Maven, while NuGet is a .NET-only tool. Over time, Maven artifact repositories and Maven Central have become a de facto mechanism for distributing and sharing Java artifacts. Being able to resolve and deploy these using Maven repositories has become a “must have” ability for all newer Java build tools.

Release and Snapshot Dependencies

As I mentioned previously, internal dependencies are normally defined using snapshot versions while external dependencies use release versions. Let’s look into release versions first as they are easier to reason about.

Release dependencies are those which have a fixed version number, such as the "1.8.6" version of the Groovy distribution. Whatever artifact repository is used by the build and whenever it attempts to locate this dependency, it is always expected to resolve the exact same artifact. This is the main principle of release dependencies: “Same version = same artifact”. Due to this fact, build tools do not check for a release dependency update once it is found and will only re-download the artifact if the local cache was emptied. And this all makes sense, of course, since we never expect to find divergent artifacts of the same library carrying an identical version number!

Snapshot dependencies are different and, as a result, way trickier to deal with. Snapshot dependency versions end with a special "-SNAPSHOT" keyword, like "3.2.0-SNAPSHOT". This keyword signals the build tools to periodically check an artifact with a remote repository for updates; by default, Maven performs this check on a daily basis. The function of snapshot dependencies, then, is to depend on someone else’s work-in-progress (think “nightly builds”): when product development moves from version "X" to version "X+1" its modules are versioned "X+1-SNAPSHOT".

Snapshot Dependencies Uncertainty

If the main principle of release dependencies was “Same version = same artifact” (after version ‘X’ of library is released, its artifacts are identical all around the world, forever), snapshot dependencies’ principle is “Same version = ever-updating artifact”. The benefit of this approach is that it enables retrieving frequent updates without the need to produce daily releases which would be highly impractical. The downside of it, however, is uncertainty – using snapshot dependencies in a build script makes it harder to know which version was used in a specific build execution. My "maven-about-plugin" stores a textual “about” file in every snapshot artifact in order to better identify its origins such as VCS revision and build number; this can be helpful but it only solves half of the problem.

Being a moving target by their definition, snapshot dependencies do not allow us to pin down versions on which we depend and therefore build reproducibility becomes harder to achieve. Also, in a series or pipeline of builds (when a finished build triggers an invocation of subsequent ones), an artifact produced by initial pipeline steps is not necessarily consumed by the closing ones as it may be long overridden since then by other build processes running at the same time.

One possible approach in this situation is to lock-down a dependency version in a build script using timestamp so it becomes "3.2.0-20120119.134529-1" rather than "3.2.0-SNAPSHOT". This effectively makes snapshot dependencies identical to release dependencies and disables an automatic update mechanism, making it impossible to use an up-to-date version even when one is available unless the timestamp is updated.

As you see, snapshot dependencies can be used where it makes sense but it should be done with caution and in small doses. If possible, it is best to manage a separate release lifecycle for every reusable component and let its clients use periodically updated release dependencies.

Summary

This article has provided an overview of automatic dependencies management by Java build tools together with an introduction to Maven release and snapshot dependencies. It also explained how snapshot dependencies’ advantages become debatable in the context of build reproducibility and build pipelines.

The following blog posts will explore the TeamCity build chains and Artifactory build isolation which allow to use consistent, reproducible and up-to-date snapshot versions throughout a chain of builds without locking down their timestamps in a build script. More to come!

See the presentation and buy the book. One of the most influencing books I have read recently.

Over time I couldn’t but notice that the overall behavior was getting painfully slower. Surely, this blog uses “WordPress Super Cache” plugin but apparently caching wasn’t enough in order to improve performance so I decided it’s time for something better.

Let me start with the final results. Initially, the main page load time was more than 5 terrible seconds. And now? We’re down to 1 second! Here you have it, one evening spent wisely made this blog more pleasant to browse and read.

Anyone who has ever dealt with web performance analysis knows that today’s browsers and extensions provide you with enough tools to make a good analysis of web performance and page load times. Yes, I’m talking about Firebug Net monitor, “Chrome Developer Tools” Network Panel, PageSpeed, YSlow, Speed Tracer and dynaTrace AJAX. Not only that, there are a number of online resources providing a similar kind of web profiling and monitoring, some of which I have listed below. In this specific case I have used GTmetrix, its “History” chart is shown above.

So what were those changes? Since I don’t develop WordPress plugins or themes I couldn’t do much about internal details, but there was still large enough room for improvements.

• First of all, I cleaned up WordPress from unused plugins and themes. That was an easy thing to do.
• I checked that “WordPress Super Cache” and “WP Smush.it” (which I had installed already in the past) are working properly and that the response is coming gzip-compressed.
• “WordPress Super Cache” cache timeout was set to zero to disable its garbage collection – cache is now emptied when new pages are added, this is necessary for “Recent Posts” sidebar update. I also switched the plugin to “mod_rewrite” from PHP to serve cache files.

  

• When Firebug Net monitor showed an extensive amount of external resources loaded, I aggressively cleaned up the sidebar from all additional widgets. Consequently, all my social, Twitter and Pinboard links are now gone (they can be found on evgeny-goldin.com, as before). Two “Twitter” buttons accompanying each post for easier retweeting were also removed for the same reason. The only external widgets I decided to keep were DZone buttons on some posts as they are an important part of the blog’s history.
• The Gravatar plugin that used to display a picture of me on the right was replaced with a locally stored image and an HTML markup, see “Custom Dashboard Widget”.
• I later attempted to combine, minify and compress the JS/CSS resources loaded. “JS & CSS Script Optimizer” plugin did just that. Two other plugins I have tried, “Better WordPress Minify” and “CSS-JS-Booster” didn’t perform equally well.
• Trying to compress HTML by eliminating unnecessary white spaces and line breaks resulted in an negligible 5% improvement so I didn’t bother to keep this optimization. Anyway, “WP-HTML-Compression” and “WP-Compress-HTML” are at your service.
• And then came the hardest part – theme update. I decided to switch to summary-only main page so that posts’ bodies are kept hidden: some of them contain embedded Flash videos which hurts the loading time severely. The new “Blue Modern” theme is simple, lightweight and, most importantly, it keeps the main page free from any embedded content.
• At some point the performance went up to 2 seconds, according to GTmetrix reports. It happened right after I added the Gr8Conf banner linked to “http://gr8conf.org/uploads/eu2012/Image/banners/gr8conf%20125×125.png”. After uploading the image and serving the local copy of it the performance went back to normal.

So that was the process. I believe that most of the time was spent on choosing the new WordPress theme (though there are so many of them, there are still not enough really good ones) but it was a very nice journey into some of the details of web performance. To reflect on it I’d suggest the following 4-step process to boost your WordPress blog’s performance:

1. Cleanup caches.

Assuming you’re using the “WordPress Super Cache” plugin, there are two caches you’re dealing with: the WordPress cache and the browser cache. The WordPress cache saves PHP and database processing results on the server side so when a cached page is requested it serves it immediately without involving the PHP preprocessor. The browser cache saves all the retrieved resources locally so when an image or a script are requested it serves a local copy after verifying with a server that it hasn’t been updated recently. Now, depending on what you’re going to measure, clean and repopulate the corresponding cache. Generally, I perform my measurements with WordPress’ cache pre-populated and with an empty browser cache. Measuring performance when both caches are full also makes sense, as it allows you to see how quickly the page performs in an ideal situation and how long does it take for a browser to run an up-to-date check with a server.

2. Analyze the timeline (waterfall) report.

Load the page in a browser and analyze its ‘timeline’ report, also referred to as ‘waterfall’, produced by either Firebug Net monitor, “Chrome Developer Tools” Network Panel or GTmetrix whose ‘timeline’ chart is shown below.

Note that report numbers don’t carry much meaning as they will most definitely vary wildly depending on client-server geographic locations and connection speeds. What matters more is the overall loading behavior and the way your changes impact it: how many resources are loaded, what resources are loaded from external (potentially slower) domains, are they compressed, combined together or minified, which resources load most sluggishly and so on. In addition, note that while this report is useful for spotting the most obvious bottlenecks, it provides a network-only view of the page loading process. For detailed reports on page rendering and processing behavior consider using the Speed Tracer (Chrome) or dynaTrace AJAX (IE, Firefox) browser extensions.

3. Analyze PageSpeed and YSlow reports.

If the ‘timeline’ report hasn’t provided you with enough data to work on, run PageSpeed or YSlow to get a detailed list of suggested optimizations. GTmetrix conveniently provides both reports online:

4. Implement the changes and repeat.

Every time WordPress plugins are added or configured, clean up the caches and measure again. Do not take any optimization for granted. GTmetrix makes it easier with its “History” chart where you can easily spot performance anomalies as your blog evolves.

Can it be better? Yes, absolutely. Looking on the main page source I still see references to external resources that simply shouldn’t be there, probably leftovers from plugins removed. So better results can be achieved. Naturally, not being able to control and fine-tune the WordPress application, HTTP headers and the content added by the plugins makes the task less trivial but as you’ve just witnessed, it is more or less doable nevertheless.

What were the lessons learned? I think minimization of external content is the major takeout here. Taking out absolutely everything except the content proved to be the most high impact performance optimization. Content compression, resources unification and minification surely help but not as much as having all social buttons removed and external images served from the local domain. In addition, I figured out that adding more links not only makes the blog slower, it doesn’t make anyone’s life easier as I’m sure most of you have enough links to follow.

What’s next? I believe that the next step can be more dramatic: stepping away from WordPress and switching to Jekyll-generated static content. On top of being significantly faster, it brings control over everything back to your hands which is absolutely a must if you’re seeking decent Web performance. Two beautiful examples of static content blogs would be igvita.com by Ilya Grigorik and stevelosh.com by Steve Losh. Try them out and see what fast really means!

To sum up this topic, I listed performance-related WordPress plugins and browser extensions mentioned in this post plus a number of on-line services you can use to analyze and monitor your blog performance:

WordPress plugins:

• WP Super Cache
• W3 Total Cache
• WP Smush.it
• WP-Optimize
• Custom Dashboard Widget
• JS & CSS Script Optimizer
• Better WordPress Minify
• CSS-JS-Booster
• WP-HTML-Compression
• WP-Compress-HTML

Browser extensions:

• Firebug (Firefox)
• Chrome Developer Tools (Chrome, built-in)
• PageSpeed (Firefox, Chrome)
• YSlow (Firefox, Chrome, Opera, Safari)
• Speed Tracer (Chrome)
• dynaTrace AJAX (IE, Firefox)

On-line services:

• Google Analytics Site Speed reports
• GTmetrix (“WordPress Optimization Guide”)
• Pingdom Tools
• ShowSlow
• Yottaa
• Page Speed Online
• WebPageTest
• Speed of the Web

You can also review my WordPress and web performance links. In particular, I’ve found the “Making the Web Faster at LinkedIn” video very useful.

What was your experience in optimizing WordPress? Which tools did you use? I’d love to hear!

Note that starting from Gradle version "1.0-milestone-9" you’re supposed to prepend dynamic properties with "ext.".

"build.gradle":

..
ext.codenarcRuleSetFiles = [ 'codenarc.groovy' ]
apply from: 'https://raw.github.com/evgeny-goldin/gradle-plugins/dev/src/main/groovy/CodeNarc.gradle'
..

"codenarc.groovy":

ruleset {

    description 'CodeNarc RuleSet'

    ruleset( "http://codenarc.sourceforge.net/StarterRuleSet-AllRulesByCategory.groovy.txt" ) {

        DuplicateNumberLiteral   ( enabled : false )
        DuplicateStringLiteral   ( enabled : false )
        BracesForClass           ( enabled : false )
        BracesForMethod          ( enabled : false )
        BracesForIfElse          ( enabled : false )
        BracesForForLoop         ( enabled : false )
        BracesForTryCatchFinally ( enabled : false )
        JavaIoPackageAccess      ( enabled : false )
        ConfusingMethodName      ( enabled : false )
        UnnecessarySubstring     ( enabled : false )
        FactoryMethodName        ( enabled : false )

        VariableName             ( finalRegex : /[a-zA-Z0-9_]+/ )
        LineLength               ( length     : 160   )
        MethodName               ( regex      : /[a-z][\w\s'\(\)]*/ ) // Spock method names
    }
}

Twitter never made it easy to follow people, lists or searches over RSS feed. In every case I had to search and try out various approaches to find those that work.

Well, I’m finally able to follow all things Twitter in Google Reader and it just makes one less network to visit. It is sad to witness social networks obscuring their feeds in attempt to drag you in. I can only hope it doesn’t become too ridiculous …

Anyway, Twitter feeds for users, lists and searches are shown below. Hope you find them useful.


User feed

• http://twitter.com/statuses/user_timeline/[user-name].rss
• http://twitter.com/statuses/user_timeline/[user-id].rss
• twitter.com/statuses/user_timeline/hackernewsbot.rss (preview)
• API Reference

"user-name" is how user normally appears on twitter and "user-id" is its numerical ID which can be found using this application. You can choose either of those.


List feed

• http://api.twitter.com/1/[user-name]/lists/[list-name]/statuses.atom
• api.twitter.com/1/evgeny_goldin/lists/devops/statuses.atom (preview)
• API Reference

Search feed

• http://search.twitter.com/search.atom?q=[query]
• http://search.twitter.com/search.atom?q=%23[query] (#query)
• http://search.twitter.com/search.atom?q=%40[query] (@query)
• search.twitter.com/search.atom?q=%40TeamCity (preview)
• API Reference

JetBrains feeds

• Atom feed of my “jetbrains” list – all JetBrains tweets from projects and people (preview)
• Projects blogs and updates feed (preview)
• Employees personal blogs feed (preview)

Chrome keyword searches

Twitter-related keyword searches go a long way to make life easier! Using searches shown above I can type "tw youtrack", "tl devops", and "ts anything" to read @YouTrack tweets, my “devops” list or search Twitter for “anything”. And "gr url" loads the feed url into Google Reader.


References


It wouldn’t be possible without the following posts whose authors made the effort of digging out this useful information. Thank you!

• Cynthia Ng – “Creating a Twitter Search, Hashtag, User, Favorites or List RSS Feed”
• Piers Dillon Scott – “As Twitter protects its ecosystem, here’s how to create an RSS feed of a Twitter List”

Also, I’ve started summarizing various steps to do something with makandracards. Try them out, they are really lovely. It is faster to work on a makandracard than to start a new Wiki page (if you have Wiki).

Here are mine so far:

• Getting Intellij IDEA jars using Ivy and TeamCity.
• Calling Ivy from Groovy or Java.
• Converting SVN to Mercurial using HgSubversion.

Audio file (mp3) of the session.

Audio file (mp3) of the session.

My second issue was converting MOD files recorded by the camera before uploading them to Vimeo. Initially, HandBrake looked like an excellent solution as it compresses mods to mkv, my preferable format. On top of that it was free, which doesn’t happen a lot in the Mac world. However, I found out my camera starts recording new mod file every 60 minutes. To verify that and see if any recording is lost in the middle I recorded my digital clock for about 3 hours and found out no time above one second is lost. To merge the files I switched to Video Converter Ultimate for Mac which is more capable than HandBrake although it is not free. This application also allows conversion of mkv files resulting from the merge to mp3 audio so that people would be able to download the smaller audio files to their players. As a long-time iPod listener, I believe having a downloadable mp3 file is definitely a must for any technical session being recorded.

Another application I found for conversion of video files (to audio and video formats, including mp4 and mp3) was Miro Video Converter, also a free one but with a limited functionality. For instance, it doesn’t allow to control the compression quality.

At the end of the day, video converter from Wondershare is the one I use the most although I have all 3 of them installed.

One thing I wouldn’t suggest, though, is clicking the “Remember it as my default option” checkbox in the converter welcome screen as I found no way to switch the conversion mode later and had to reinstall the application to undo the default “Video” mode.

Well, now I have created my own “Technical Sessions” Vimeo channel (reposted here in “Video” category) where all sessions I ever visit will be uploaded to. The link to corresponding mp3 files is available in every video “Description”. Right now most sessions are in Hebrew but this will change next year.

This time the subject was Groovy and Ronen Narkis (@narkisr) presented Groovy as a language, its powerful GDK, AST and MOP capabilities together with GPars examples. Ronen presentation is available in his GitHub repository.

I was invited to speak about Gradle and since my session was targeted to the audience where not everyone was familiar with either Groovy or Gradle I decided not to cover it feature-by-feature or go into any technical details (and we were short on time, as always). Instead, I picked 10 facts about Gradle which, coming from the Maven background, made me say “Hmm, that’s cool!” in the past. Hope you’ll share my experience so here it is.

My other presentations are now available in GitHub as well.