Join us Tuesday, February 21 at 11:00AM (GMT-0500) for a 30 minute demonstration of Nexus 2.0, with Jason van Zyl, Sonatype Founder & CTO. You’ll see all the new features and learn how Nexus 2.0 will help you:

• Avoid downtime by using a highly available architecture
• Improve repository management with enhanced component information
• Standardize on a single repository for .NET, Java, and OSGi

Reserve Your Seat

If you register, you’ll also receive access to the recording after the event. So if something comes up and you can’t make it, you won’t miss out.

Two weeks ago, all Scala projects required a little bit of extra configuration to point to a custom repository for Scala artifacts hosted at scala-tools.org. Today, Scala artifacts are now available directly from Central. The contents of scala-tools.org are now integrated into the Sonatype OSS repository hosting service, and other projects have started to publish artifacts Central.

The Scala community will see immediate benefits from this move. There are no more extra repositories to configure. It just got incrementally easier to use Scala. If you are new to Scala, you don’t need to reconfigure your repository manager to proxy another remote repository. The community will benefit from Sonatype’s continued investment in the infrastructure that runs Central: a cluster of machines in both the US and the EU continuously monitored by a dynamic DNS server that can reroute traffic instantly in the event of downtime.

How did this happen? Joshua Suereth, David Bernard, and Derek Chen-Becker provided the bulk of the administrative work, and they recently decided to decommision this server and transition repository hosting to the free Sonatype OSS service. Here’s the announcement by Joshua Suereth to the user forums on scala-lang.org on January 17th:

Scala-tools.org is going down and not accepting any new OSS projects. For those of us who wish to continue release software, I recommend migrating over to Sonatype. They put a few (good practice) limitations on contributions, but scala-tools.org would have done the same before long anyway. The benefit of Sonatype hosting is that your projects will make it onto the maven-central repository and benefit from the myriads of mirrors. Here’s the link for how to get started contacting Sonatype: http://nexus.sonatype.org/oss-repository-hosting.html

Publishing Your Scala Project to Central via Sonatype OSS

If you maintain a project that previously published to the scala-tools.org repository, here are three resources that provide guidance for Scala developers looking to publish Scala artifacts to Central:

• Publishing artifacts to Sonatype instruction written by Joshua Suereth on publishing to Sonatype OSS
• Publishing SBT Projects to Sonatype OSS from Cake Solution’s Specs2 Spring Project
• PublishToSonatype.scala some Scala code from Paul Phillips to automate the process of publishing artifacts to Sonatype’s OSS

“Wait, that’s how Central works?”

There’s something so automatic about dependency management in Maven that it often takes people a few months to understand exactly where those JAR files are coming from. In an 8 hour Maven class, I get to dependencies in the third hour, and after describing Central, what it is was like before Central, how metadata is stored in a repository alongside binaries, transitive dependencies, etc…. it all falls into place, and people realize that this simple thing they’ve grown accustomed to is only easy because of a ten year effort to refine the model, the creation of a support structure for source forges at places like Oracle and Google, and a constant investment in infrastructure.

On the one hand, it’s a great success that Central is, for the most part, an invisible utility that supports developers.  On the other hand, it’s the kind of thing that people can start to take for granted very easily.

For example, a few months ago I spoke to someone who worked in an environment disconnected from the internet for security reasons.   This individual was talking about how limiting it was to have to download JARs from open source projects manually and assemble them in a project.   His words were: “It’s like programming Java in 2001 again.”

How can you help?

Imagine millions of developer spread all over the world: different time zones, different applications, but they all hit the same service: Central.    Some regions have more developers than others so we certainly see peaks in usage throughout the day, but in general, Central’s serving thousands of files throughout the world at any given time during the day.

Maybe someone just installed Maven for the first time, or maybe they blew away a local repository, with numbers like these we see a world that has a constant appetite for artifacts.   It isn’t a problem for Central, and I’m not writing this because Central is falling down on the job. Central can handle it, but it certainly isn’t the most efficient way to support millions developers.  It isn’t a good use of network bandwidth, and it isn’t a good use of energy to constantly cart around the same static JARs over and over and over again when the solution is so easy.

If everyone who used a build tool that interacted with Central adopted a repository manager such as Nexus we’d have a faster, more responsive system.   Central’s maintainers would be focused less on addressing the occasional runaway build and could spend more time and resource on increasing availability and functionality of this essential service.

Broken Builds

The other factor playing into this is that Maven builds only download releases once.   It isn’t like these build tools are repeatedly returning to Central to download release artifacts over and over again.

Well… actually… that isn’t true, we’ve seen some installations of Hudson configured to delete a local repository before every build placing a high load on Central.   Imagine a build that downloads 50 MB of dependencies running once every 5 minutes.  That’s one build consuming ~14 GB a day never mind the time wasted downloading static artifacts. While these broken builds are the exception, they do still show up from time to time.  Central can handle the load, but imagine 1000 of these broken builds running continuously and you can see the challenge.

A Simple Reminder: Please Proxy Central

We’re constantly watching the performance of the system and making sure it stays up and running for an entire world of developers.  If you use a build tool that hits Central whether it is Buildr or Maven or Gradle or Ivy, you can help us by running a Nexus instance.

Even if all of your builds work perfect, running a local Nexus instance helps preserve Central as a public, free resource and it will lead to faster, more responsive builds.

In this post, I focus on storage requirements for Nexus. I discuss general recommendations and point you at resources we’ve developed to help you come up with accurate estimates for how much disk space you’ll need.

Disk Space

Disk space is going to be the critical parameter for a Nexus installation. At its core, Nexus is simply a collection of files and a set of services to index and serve these files. If you integrate Nexus into your development process and come to depend on it as a collaboration mechanism, you can easily consume hundreds of gigabytes (or even terabytes) of space.

Coming up with a simple guideline for storage requirements is difficult as it depends on a number of factors: How many projects do you have? How large are the artifacts being deployed to Nexus? How frequently are these artifacts deployed? and How long do you keep your releases? How much open source are you consuming from Central? How often do you update external dependencies? and How many 3rd party artifacts do you need to upload?

If you deploy artifacts to Nexus, your internal, hosted repositories are what will end up consuming the most space over time. At a large organization with hundreds of projects and frequent releases, it is very easy to create systems that consume a surprising amount of space. If you are interested in diving into the details and coming up with an estimate for your organization, watch “Getting Scientific about Sizing Nexus”.

An Initial Starting Point

While some of our engineers like to aim high with an initial recommendation of 250-500 GB, I like to aim a little lower. Sure, if you are rolling Nexus out to a 5,000 developer installation with thousands of projects, you may very well want to start with 1 TB. On the other hand, if you are gradually rolling Nexus out to a department or two, you should start with a more reasonable number: 50 or 100 GB.

I recommend starting with 50 or 100 GB, and I also recommend being prepared to expand that number as needed. Starting with this smaller number avoids the problem of procuring a huge chunk of disk space only to watch it sit idle for the months (or years) it will take you to consume all this space. Aim low, plan to expand.

Conclusion

Your initial estimate for disk space consumption is going to be just that, an estimate. Having set up scores of Nexus instances for organizations of all sizes, my experience has been that you’ll want to do some ballpark estimates and then multiply that estimate by a factor of two or three. When you connect systems like Hudson to Nexus and deploy snapshots from every integration build, you’ll appreciate the extra space.

As you start to use Nexus, you’ll have to tweak your scheduled jobs to make sure that you are periodically removing old snapshots and regularly keeping an eye on storage. If you expand the number of projects or developers using a Nexus instance, you’ll want to revisit some of these initial estimates and make sure that your system has enough storage to keep track of all the artifacts it is caching and storing.

Sometimes Releases are Disposable

For example, consider a company that creates a large web-based system. They may deploy new versions of components to production multiple times a day. If this seems unrealistic, know that system administrators for popular services like Facebook, Last.fm, and Flickr have talked openly about how frequently code is pushed to production systems. A few times a day isn’t odd in some of these environments, and with a large system, you’d consume terabytes of space to keep all of those old releases around.

Last time I worked at a large consumer-focused web site, pushing something to production once a day wasn’t uncommon, and even the idea of rolling back to anything other than yesterday’s build was laughably impractical. If you identified a bug in a CMS or a production database, you’d just craft the fix it and move on. This is especially true of larger, web-facing systems in which the only reality is the code that is running in production today. A site like Flickr gains nothing from being able to roll back to a release from last September.

The Other Side of the Coin: Releases are Forever

Compare this to the production release schedule of a critical, supported product and it’s like night and day. If you are coding some serious banking system you might be lucky to have a release once a month. In all likelihood you might be looking at a quarterly release.  When you are working on critical applications, ship software, or have infrequent release cycles then the ability to roll back to previously released binaries is very important. When you work somewhere fast-paced with a very short release cycle, there’s not much value in retaining older releases.

For this reason, we’re clarifying this point: if it makes sense for you to delete release artifacts from a hosted repository, go for it. Just remember to re-index the server once you’ve manipulated the storage folder.

We put together the following video to share more of our thoughts on this topic.

• Can you describe the differences between permissive, weakly protective and copyleft licenses?
• Do you understand the ramifications of including copyleft licensed components in your commercial applications?
• Do you know how component dependencies affect your application’s licensing?

If you want to brush up on your knowledge, please check out our short paper on open source licensing available here.

10.     Establish mechanisms to monitor the effectiveness of your open source governance program

Your OSS governance program is in place. You’ve educated everyone on the policy, standardized components, built open source management into your development process and are continuously monitoring production applications.  You’re done, right?  Almost, but not quite.   You’ll want to set up check points to monitor its effectiveness to learn what’s working and what’s not.

A good place to start is by monitoring open source downloads from outside sources.  Just because a component was downloaded doesn’t necessarily mean it’s being used in an application, but if you see problematic components coming in,  it’s probably worth investigating.  If you’re using an enterprise repository manager (and we suggest you do), you’ll want to know if it’s being used or being circumvented.

You may also want to audit your key applications. It would be surprising to find problematic components, especially if the apps were developed after your policy was in place. But, if you did, it would be an indication that something went wrong. Maybe a development group was unaware of the new policy, or perhaps they don’t yet have the tools needed to effectively follow it.

A final place to monitor is code delivered by outside contractors, consultants, or ISVs. You’ll want to be sure that they are following your governance policies and haven’t inadvertently included components with security or license issues.

Now that you know what you want to monitor, the next question is how? That’s why we created Insight. Insight reports all of your organizations open source downloads.  You’ll also have tools for analyzing and reporting details of all the components used in your Java projects, whether they were developed internally or delivered by a 3^rd party.

You can learn more about Insight here.

We want your opinion

• Integrated security and licensing reports to discover problematic components
• Improved availability with our new Nexus Availability Architecture
• Ability to manage both Java and .NET components from a single repository

The webinar recording is now available here.

Special pre-release pricing is available for Nexus Pro 2.0. Check out the webinar or contact a representative for more information.