The Build Doctor is now running a new version of Wordpress, Thesis and some other plugins, on a new  VPS.  I have plans to make the blog more useful now that it’s not running an ancient version of Wordpress.  On the positive side of things I was very impressed that none of the vulnerabilities hit the blog.  Having your entire Wordpress install owned by the root user helps, certainly.

Anyway, I’ll be pretty busy for the first part of November.  Until I manage to get some of my devopsdays notes up, here’s some links to keep you going:

• The Ship’s Blog fires a shot across Maven’s bows.  The Sonatype guys respond with all cannons blazing.
• There’s yet another Ruby CI server.  Looks better than some of the alternatives.
• Erik visualizes build pain
• Damon Edwards is working on a Creative Commons licensed whitepaper on automated provisioning.

Related posts:

1. links for 2009-01-18 Sonatype
2. links for 2009-01-31 A Univer
3. links for 2008-12-24 collect

Related posts brought to you by Yet Another Related Posts Plugin.

I presented this talk at QCon London in March this year. InfoQ just released the video here.

How did it go? Fine. I did pitch it at too low a detail. This was partly because I’d stunned a group of managers into submission with the same talk. The attendees were really technically accomplished in this session, partly because Gus Power and Kris Lander did a barnstorming talk before me; they were mobbed afterwards. Dan North suggested in the bar afterwards that I could have raced through the slides and thrown the talk open. Oh well, there’s always a next time.

Thanks to Steve Freeman for including me in the track. If you have any questions, do tweet or email medic (at) build (hyphen) doctor (dot) com.

Link

Update: Ikenna wanted the slides. Get them here.

Related posts:

1. Continuous Integration Cage Fight: Team City Second to
2. Hosted Continuous Integration: Run Code Run Hosted
3. the industries first and only open-source continuous integration solution from http

Related posts brought to you by Yet Another Related Posts Plugin.

Thanks to Graeme Gillies for this funny anecdote from the Atlassian Giveaway. I love the fact that he persisted in nailing this.

When I started at a company (which shall remain nameless) being the
new (junior) guy I was made the “build guy” of one of their biggest
projects to date. I quickly learnt that the code had cyclic
dependencies amongst other things, which mean that if you did a fresh
checkout into a fresh workspace (no builds before) the code would not
compile because it relied on “cruft” and left over artifacts of
previous builds to build correctly.

They were using Apache Ant at the time (this was a J2EE project) and
then the decision was eventually made to migrate to Maven 1. This
would seem like a great thing to all, and finally a way of reviewing
and getting the build process back under order. Unfortunately what
happened instead was that when you tried to compile the build with
Maven it would break half way through complaining about dependencies,
after which I was instructed to then run the ant build which would run
and break half way through as well, when that happened, run maven
again, it should get further this time, when it breaks, run ant, then
maven, then ant, etc. Eventually what would happen is each build
script would break, but get further and further each time until
eventually one of them somehow gave you a completed build.

Took me the better half of 6 months but eventually I pulled it apart
and got it sorted.

Related posts:

1. Story: Wolf++ fixes the Deployment Process The last o
2. Story: Rolling your own Thanks to
3. Story: I call it “The Wheel” This anony

Related posts brought to you by Yet Another Related Posts Plugin.

I thought I was pretty good at guitar once. You only realise how much you have left to learn when you see someone who’s at the next level – and then you realise that they can kick your ass without breaking a sweat. Realising that you’re not as good as you thought you were can have two effects:you double your efforts, or give up in disgust.

Kohsuke Kawaguchi’s webinar on Hudson was an ass-kicking moment. In guitarist terms, he’s a monster. He’s Eddie Van Halen.

Kohsuke Kawaguchi

I started live tweeting his webinar once the Java client ground into life on my Macbook, but I switched to an editor when I realised how much depth there was to this. The rest of this post is a fairly linear set of notes on his talk. I’ll be adding links (and maybe a correction or two). Please do comment or tweet if you have anything to add or fix.

18:07 – Webinar client has sprung into life. Kohsuke is already talking about patterns for hudson. Lesson: don’t use hostnames in DNS names. Also, use port 80 for the service. Apache makes a good frontend with mod_proxy.

Sun Hudson server has 600GB of data. You need to back up at least some #Hudson data files. There’s a Hudson backup plugin. @kohsukekawa uses cpio. That’s kicking it old-school. They also recommend ZFS snapshots.

Hudson slave nodes need 1 170KB jar file to run. Single link, with no assumptions about transport. VM’s or recycled hardware can be slave. Do be careful about how you organise Hudson clusters. Be prepared to make more clusters if other people have different requirements.

@kohsukekawa just suggested using the native OS packages that they supply. Fine idea. I do that. Also, use a filesystem that grows well. Again, they use ZFS.

Deploying your operating system is perhaps the best way that you can treat your Hudson slave machines. Kohsuke recommends Windows Deployment Services for Windows slave machines. For pretty much any other OS he’s a big fan of the Hudson PXE plugin. PXE is a standard for booting a computer over Ethernet – Sun hardware always supported this out the box. I used to love booting and installing an operating system from the prompt.

Your other option is to clone Virtual Machines but this is far more painful – how do you version a few gigabytes of operating system? You may be able to get away with building systems by hand, but I’m not fond of the error rate.

Hudson can install lots of dependencies on the slave machine: with a Java Runtime environment and SSHD on your unix systems you can install the Hudson slave on the server you just need the hostname. For Windows, you can pass the administrator username and password via DCOM. It even works from a Hudson master running on Unix.

What if the Hudson master server can’t see the slave machines (ever had an over-zealous firewall administrator)? You can log into the slave machines and use Java Web Start to have the slave talk back to the master. You’re screwed if the slaves can’t see the master.

After you’ve launched a Hudson slave via JNLP you can install it as a Windows Service. The Unix option is to run a headless Java process.

Hudson will also maintain the JDK, Ant, and Maven for you. You need to declare the versions that you need. I hope the sheer enormity of this isn’t lost on some of the readers of this post. That saves so much monkeying around. You can also tell it how to run the installer to get other dependencies. I’d probably tell Hudson how to kick off a Puppet run. Kohsuhke also suggested:

• Rsync
• apt-cyg
• cfengine (the tool that inspired puppet)
• Ruby devs, you might like to use chef. I still can’t recommend it in production however
• Windows peeps, you can use Active Directory and the open source WPKG (which needs a reboot)

The downside to puppet/cfengine is Windows. We all need to talk to Windows systems. Hopefully Mr. Nasrat’s work on Puppet for Windows will pay dividends in that respect. Kohsuke recommends Cygwin for deploying apps on Windows.

What should the goals of your Hudson cluster be?

• Make the slaves a pawn: he used the term interchangeable and nameless. I like pawn. It conjures disposable build slaves for me. The benefit is load balancing within the cluster, reduced false positives, add easier lifecycle management.
• Depend on labels, not slaves – this improves utilisation. Use a group of slaves in a label so you depend on many hosts, not one.

Hudson is going to put me out of a job. There was a fascinating section on reliability: Hudson will monitor slaves and take them offline if they start running out for disk space, or the clocks aren’t in synch, etc. I believe that he said all these checks came from the field.

Hudson Housekeeping:

• Use NTP to keep the slaves in synch
• Keep /tmp clean. i use Puppet for this.
• Keep adequate records of changes to your cluster (goodness – old school systems admin again!)

Hudson also cleans up the mess after builds. As I understand it, Hudson will attempt to diff the process table before and after the build process, and kill off any processes that are left running. Again, that’s a simple idea that adds a lot to your build process. Of course you want to control the dependencies from your build. If you must have some process running (slow old enterprise software?), the wiki documents how you can switch this off.

Hudson will report load on your cluster. Load tends to even out if you have enough capacity.

Upgrading Hudson is just a matter of upgrading the war file. Do keep the old one. It’s possible to downgrade again. Hudson can also update itself, which I wouldn’t recommend if you use OS packages. You need to update your plugins yourself these days, which wasn’t always the case.

If a release has few subsequent bugfixes, it’s probably a good one.

The easiest way to make builds dependent is to trigger one job after another. You can trigger multiple parallel test jobs after a build, to get faster feedback.

The build promotion plugin allows you to pick up really good builds to promote downstream. It can do this automatically, or from test results. Once promoted, a build can be used further down the development process: to commence QA, deploy, integrate elsewhere, or push to Maven. An example of this: Sun use the promotion plugin for JAXB-RI to push to CVS for delivery to the JAX-WS RI. So good to see these guys dining on dogfood.

Concurrent builds allow you to get faster and faster feedback to the developers, and hence isolation of changes. Good for flickering builds. Use a timeout to stop a bad build exhausting all capacity.

Matrix projects allow you to concurrently build your project on different Operating Systems and middleware. Do use a ‘touchstone’ or ‘canary’ build first. Build them sequentially if you need the same resources (e.g. databases). What if your matrix gives you combinations that are the same, what do you do? You can filter out some combinations or suggest a coverage ratio and let Hudson work it out. Nicely declarative.

18:53 – just lost the audio. Kosuhke’s Conclusion:

• Planning helps on an enterprise scale Hudson deployment.
• Throwing lots of resource helps a LOT.

18:55 – audio is back and we’re in Q and A. I chose this moment to go dash for my train.

My conclusions: Hudson offers a huge amount of value for a tiny price. But not just in terms of numbers of features. That gets boring after a while. It’s the thoughtfulness of the features, and the attempts to address lifecycle that have me excited over Hudson tonight. Most other Continuous Integration tools that I have used exhibit a clear bias towards the development end of things. I haven’t seen a tool that reached so far towards the operational side of things. Putting the stack together on the build machines has typically left to the likes of me; now there’s a tool that does that

Don’t tell him, but I was pretty impressed with Kohsuke’s knowledge of systems, as well. I’m sure a lot of these features came from the community, but he has a pretty rounded view. Maybe it’s the Sun Microsystems influence – say what you like about them, but they know good engineering. In any case: we owe this man, the open source team, and Sun Microsystems a lot for pushing the envelope on Continuous Integration.

If I were a Continuous Integration tools vendor, I’d be feeling nervous about getting my guitar out.

Update: did you miss the webinar? Have a look at it here.

Image via Andre

Related posts:

1. Clover plugin for Hudson The Atlas
2. CI Cage Fight – Hudson The next
3. Sun enters the Continuous Integration business Sun sell

Related posts brought to you by Yet Another Related Posts Plugin.

Last week we interviewed PJ. This week it’s JTF, or Jeffrey Fredrick. He’s one of the guys who maintained CruiseControl and co-founded the OIF (the organisation that brings you CITCON). He’s also the Technical Evangelist for Urbancode, who bring you Ant Hill Pro.

My Flip cam called time on the interview by filling up at 9 minutes and 59 seconds. Jeffrey, thanks for the interview!

Related posts:

1. Video: Paul Julius Interview The main
2. Video Interview: Code Coverage with Atlassian Clover How do yo
3. Continuous Integration Cage Fight – the entire first series CITCON Eur

Related posts brought to you by Yet Another Related Posts Plugin.

Outsourcing Continuous Integration isn’t a new idea, but we’re seeing more and more traction in the space. The headline news is:

• It’s certainly not for everyone.
• The space is going to get more and more interesting as cloud services increase.

Not for everyone

If you’re at all paranoid about security, you’re unlikely to want outsourced CI.

You might fall at the first hurdle: will you be able to justify outsourcing the build of your most valuable asset to an auditor? If you can’t address the (low) risk of your code being subverted, it might be game over. Perhaps you can prove that no code built at your outsourced service is used. It also raises the question of your version control system. Is it inside your firewall? Do you need to provide secured access to your outsourcing partner? Perhaps you outsource version control elsewhere. Can those parties talk? Could there be a man-in-the-middle attack?

What about some more practical reasons for keeping it in house? You might depend on internal services for your build. What’s your internet connection like? Do you mind if you lose your connection to the Internet, and therefore the outsourced continuous integration server?

Cooler tools

Can your IT department supply you with Linux, XP, Vista, and now Windows 7 with a host of different browsers? Of course not. They aren’t there to deliver a glittering array of choice in operating systems. Your friendly local IT department is there to drive down the cost of computing by stamping a uniform operating system onto all your computers. Your helpful IT vendor is there to help said IT department, being the guys who pay Bill’s bill.

It’s a good thing Amazon branched out from selling dead trees. The Amazon Web Services tool-set is amazing. Want somewhere to keep all those built artifacts? Then how about S3? Need a few dozen build agents? EC2 is your friend. We’re really just getting started here. One of the most obvious uses for the cloud is in allowing you to test all those pesky client configurations: those permutations of Windows, IE and Firefox, for example. I predict that Continuous Integration vendors will quickly reach feature parity on this, because it’s so darn useful.

Such services will become more specialised as more service models evolve. Need to test with your enterprise stack? I imagine you’ll be able to piece together some of those components as well. Will there be an API for submitting builds to any build farm? I certainly hope so.

In the medium term, I’m not convinced that many CI servers will end up fully hosted on the cloud. What’s more likely is that many enterprises will end up with:

• One big, hand-rolled build machine, hosted at the firm.
• Lots of nodes in the cloud.
• A really freaking big Amazon EC2 bill.

This works, because you get to assume that you’re protecting the your assets, and just giving your built code a workout out there in cloud-land (I also predict the rise of compromised cloud servers, FWIW). You still need to deploy the app somewhere and fire up nodes to test against it, but you are limiting the opportunities to inject malicious code at build time. This allows you to keep built artifacts (be they in a Maven-style repo, or just spat out from an Ant build) on the inside of your network (ironically where you probably face the most realistic risks of attack – by disgruntled or financially compromised employees).

Perhaps some of the cloud vendors will acquire enough security certifications to convince auditors that it’s safe to use. And maybe, enough organisations will start thinking of operating systems and middleware as bigger code objects to play with via an API or toolset, rather than infrastructure to manage with a meatcloud.

Some vendors

So who actually provides outsourced Continuous Integration? This is by no means an exhaustive list. Tweet me if you have suggestions for the list. Thanks.

• Collabnet offer Team Forge, which looks like it used to be SourceForge Enterprise Edition. Remember that? I worked at a bank that used it. Happy times. [mainly due to NPR and Peet's Coffee. Though SFEE did work reasonably well for a large programme of work]
• Run Code Run – have built off the back of GitHub with a sweet little model – they consume hooks from GitHub, and trigger from those to build your Java apps. They are branching out from Ruby projects to include Java as well, and will rent you a private CI system by the month.
• CI in a Box is an Amazon EC2-based solution. I’m not sure who’s making money off of this one apart from Amazon – the house always wins. Looks like low cost and scalable Hudson implementation, anyway.
• Mike CI contacted me the other day – they have a new service – operated out of the UK, but available everywhere, of course. They are pre-launch, but they seem to be in a similar space to Run Code Run – allowing developers to easily adopt CI. They support Java but might also offer .NET. I’ll try and get something more in-depth, and pounce on them for an interview if they come to London. They seem really nice.
• CI Foundry is also new, and in beta. This is a bespoke service, so aimed at companies who want things done for them, or in situations where the standardised offerings don’t fit. DISCLAIMER: I’m behind this one. I’m going to be open about this. Compromising my editorial integrity would feel dirty. I’ll even try and get someone else to do reviews is there’s a problem.

Are you using outsourced Continuous Integration? Do you want to share your experiences? Tweet me!

Image thanks to Orin Zebest

Related posts:

1. Continuous Integration Cage Fight – the entire first series CITCON Eur
2. The quest for a decent Ruby Continuous Integration tool Git has be
3. Continuous Integration and Release wisdom Can

Related posts brought to you by Yet Another Related Posts Plugin.

The main reason I cannot miss a single CITCON Europe: catching up with all my friends, old and new. Paul Julius and I decided to have a formal catch up on camera. Less formal conversation was had, too.

Related posts:

1. Video: Jeffrey Fredrick Interview Last week
2. Continuous Integration Cage Fight – the entire first series CITCON Eur
3. Citcon Paris is filling up fast Citcon Eur

Related posts brought to you by Yet Another Related Posts Plugin.

Urbancode just released AntHillPro 3.7. What’s new?

• They just added a boatload of static analysis tools, including my favourites, Checkstyle and FindBugs. There’s also some serious security analysis tools in there. This stuff does catch errors.
• You can now write plug-ins via a handy API. They’re also going to eat their own dogfood. Mmm, tasty dogfood. They’ll be implementing new features using their own API.
• Git support, UI bling, security and command line features.

I shot the breeze with Urbancode’s Jeffrey Frederick at CITCON Paris: he’s really keen to help people write plugins for AntHillPro. It makes good sense. Psychologically it seems to be easier to write a plugin for an open source project, instead of a feature. Maybe because it’s your creation and not something you gave to the project? Bah, we’re egotistical sometimes.

Link

Related posts:

1. Citcon Paris is filling up fast Citcon Eur
2. links for 2009-01-20 AnthillP

Related posts brought to you by Yet Another Related Posts Plugin.

Related video from CITCON coming soon

Related posts:

1. Video: Jeffrey Fredrick Interview Last week
2. Story: Wolf++ fixes the Deployment Process The last o
3. Continuous Integration Cage Fight – the entire first series CITCON Eur

Related posts brought to you by Yet Another Related Posts Plugin.

The Atlassian guys emailed me a couple of days ago to tell me about their new Clover plugin for Hudson. What’s useful about it? You don’t need to write any Ant code to use Clover in your build – you install the plugin, enable it in your pointy-click Hudson configuration, and you’re off. You still need an Ant build, mind.

The plugin uses an Ant listener to spy on the build process and thereby work out what it needs to do. Also, it’ll trend your coverage over time.

It’ll work with Maven as well.

Link

Photo courtesy of Jessica

Related posts:

1. Hudson on Ubuntu with Big Visible results in six steps We’v
2. CI Cage Fight – Hudson The next
3. Kohsuke Kawaguchi Hudson Webinar I thought

Related posts brought to you by Yet Another Related Posts Plugin.