The general experience is great, most of the time I don’t miss Ubuntu. The only thing that works great with Ubuntu where I didn’t yet find a good alternative on Arch is vmbuilder to bootstrap virtual machines in a KVM/libvirt environment.

I really like the up to date packages and the rolling releases. The ArchWiki is very helpful and the community is great.

Of course this is marketing material but nevertheless it contains a lot of truth. Many organizations lack a process to ensure the libraries they are using in their applications are up to date. The larger an organization is the higher the probability that they prefer not to update their dependencies because they fear to break something. Never touch a running system – even if it is insecure.

You can argue that the metrics they use are inaccurate as a vulnerability in a library that is used in an application does not imply that the application itself is vulnerable. However if the application is not affected by the vulnerability of a dependant library this is more often by coincidence than by analysis and informed decision.

For applications that we are building for our customers we have a few rules in place that lower the risks involved:

• We prefer proven frameworks and libraries with a good security track
• We check the general code quality of frameworks and libraries we use before we include them
• Each iteration starts with updating the dependencies of our applications to their latest stable version

While this works well for applications while they are built it does not help for the phase where no active development takes place. It also doesn’t help with security issues that are discovered and need an immediate fix for the release currently deployed to production.

Therefore we offer support contracts for our applications that cover the latest production release in supported environments. To minimize cost we do not support older versions or milestone, beta and candidate releases.

For those versions we provide our customers with security fixes for vulnerabilities found in one of the supported products or the libraries used in one of these products. This of course includes monitoring the libraries and frameworks we use for reported vulnerabilities and security issues.

We also encourage our customers to plan for maintenence releases at least every six months to keep the dependencies up to date even if there are no new features to be included.

The server supports a number of password hash functions including MD5, SHA, SMD5 and SSHA so that the clear text password used for authentication is not stored on the server and an attacker who gains access to the data can not use it for authentication unless he breaks the hash.

Password checks are implemented in the class SimpleAuthenticator that includes the following code:

// Get the stored password, either from cache or from backend
byte[] storedPassword = principal.getUserPassword();

// Short circuit for PLAIN TEXT passwords : we compare the byte array directly
// Are the passwords equal ?
if ( Arrays.equals( credentials, storedPassword ) )
{
    if ( IS_DEBUG )
    {
        LOG.debug( "{} Authenticated", opContext.getDn() );
    }

    return principal;
}

The provided credentials are compared to the stored password which can either be a plain password or the hash of a password. This causes ApacheDS to allow users to authenticate either with the password or the corresponding hash. So authentication of a user with the password abc which is stored as the salted SHA1 hash {SSHA}lIifvzM278asTV8NtjfO3EV3z4caaC5uJPouWw== will succeed if either the original password or the hash is provided.

Both calls will succeed equally:

ldapsearch -h localhost -p 10389 -D uid=admin,ou=system -x -w 'abc'
ldapsearch -h localhost -p 10389 -D uid=admin,ou=system -x \
  -w '{SSHA}lIifvzM278asTV8NtjfO3EV3z4caaC5uJPouWw=='

An attacker who gains access to the stored hash will thus be able to successfully authenticate as any user without having to know the password.

It seems all versions of ApacheDS 1.5.x including 1.5.7 are vulnerable. The new 2.0 branch does not seem vulnerable.

I’ve notified the Apache Security Team on 2012-03-12 and informed them on 2012-03-15 that I will publish this blog entry on 2012-03-19 after they remained silent for three days.

Emmanuel Lécharny finally replied that he does not consider 1.5.7 stable and that

People using the server *must* use 2.0.0-Mx versions, even if this version is not stabilized yet.

The reason they still link to the vulnerable 1.5.7 version in their “Latest Downloads” section without a word on the security issue is

Pure laziness… Sadly, we are knees deep into coding, and we have neglected the web site and the doco :/

Seems priorities are more on publishing good news.

Update 2012-03-27: Now more than two weeks after the notification they had plenty of time writing emails explaining why this isn’t a problem but apparently no time to remove the link to the vulnerable version from the Latest Downloads section.

The screenshot included an additional detail that caught our attention. A few toolbars were installed in Internet Explorer including the Pdfforge Toolbar:

This toolbar comes with PDFCreater and Wikipedia has some details on it:

The installation package includes a closed-source browser toolbar that is considered by many users to be malicious software (see below). Although technically an optional component, the opt-out procedure is a multi-step process which is considered by many to be intentionally confusing.

The toolbar injects additional JavaScript code into the browser to track its users, report visited sites and show ads.

It turned out that the toolbar had added an indexOf() function to Array.prototype:

Array.prototype.indexOf=function(a){if(this===void 0||this===null)
throw new TypeError;var b=Object(this),c=b.length>>>0;
if(c===0)return-1;var d=0,e=d;arguments.length>0&&(
d=Number(arguments[1]),d!==d?d=0:d!==0&&d!==1/e&&d!==-(1/e)&&
(d=(d>0||-1)*Math.floor(Math.abs(d))));if(d>=c)return-1;
var f=d>=0?d:Math.max(c-Math.abs(d),0);for(;f<c;f++)
if(f in b&&b[f]===a)return f;return-1}

This seems to be a workaround for older versions of Internet Explorer that don’t support the indexOf() function natively. To understand how the code ended up in the info messages field here is how the application iterated over the array of messages to display:

for (var i in messages) {
  // adds an element for each key in the messages object
  // including messages["indexOf"]
}

The correct way to iterate over an array is however

for (var i = 0; i < messages.length; i++) {
  // adds elements only for the values in the array
}

Yet another case where the two loops show different results is the following:

var a = [];
a[5] = 5;

for (var i=0; i<a.length; i++) {
    // iterates over numeric indexes from 0 to 5
}

compared to

for (var key in a) {
    // shows only the index of "5" and ignores 0 to 4
}

What can we learn from this?

• Do not use for..in with arrays
• Do not modify built-in datatypes as others may depend on them
• Be careful what you install on your computer

Recent versions of Tomcat include some code that makes you aware of problems when you undeploy the application:

SEVERE: The web application [] registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to unregister it when the web application was stopped.
 To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
SEVERE: The web application [] appears to have started a thread named [Thread-14] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-15] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-16] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-17] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [] appears to have started a thread named [Thread-18] but has failed to stop it. This is very likely to create a memory leak.

As you can see Tomcat managed to unregister the JDBC driver that the application had failed to unregister but could do nothing regarding the threads that had been started but not stopped.

I ran the application with YourKit attached to check that the WebappClassLoader had actually leaked and to see what those threads were that prevented it from being garbage collected. The “Paths from GC Roots” view in YourKit is well suited for this:

As you can see there are four threads from ONS that prevent the ClassLoader from being garbage collected: oracle.ons.SenderThreads and oracle.ons.ReceiverThreads.

I wrote a small ServletContextListener that shuts down ONS to get rid of them. After that I noticed that Oracle registered a OracleDiagnosabilityMBean that I had to unregister. Finally I made sure the JDBC drivers that the application had registered were properly unregistered from DriverManager.

With those changes in place the application undeployed well and was fully garbage collected.

Here is the code:

import oracle.ons.ONS;
import oracle.ons.SenderThread;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.util.ReflectionUtils;

import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;
import java.lang.management.ManagementFactory;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.sql.Driver;
import java.sql.DriverManager;
import java.util.ArrayList;
import java.util.Enumeration;
import java.util.Hashtable;
import java.util.List;

public class CleanUpListener implements ServletContextListener {
    private Logger logger = LoggerFactory.getLogger(getClass());

    @Override
    public void contextInitialized(ServletContextEvent sce) {
        // do nothing
    }

    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        shutdownOns();
        deregisterJdbcDrivers();
    }

    @SuppressWarnings("unchecked")
    private void shutdownOns() {
        logger.info("Shutting down ONS");
        final Method getRunningONS = ReflectionUtils.findMethod(ONS.class, "getRunningONS");
        final Method shutdown = ReflectionUtils.findMethod(ONS.class, "shutdown");
        ReflectionUtils.makeAccessible(getRunningONS);
        ReflectionUtils.makeAccessible(shutdown);
        final ONS ons = (ONS) ReflectionUtils.invokeMethod(getRunningONS, null);
        if (ons == null) {
            return;
        }
        
        ReflectionUtils.invokeMethod(shutdown, ons);

        final Field senders = ReflectionUtils.findField(ONS.class, "senders");
        ReflectionUtils.makeAccessible(senders);
        final List<SenderThread> senderThreads = (List<SenderThread>) ReflectionUtils.getField(senders, ons);
        if (senderThreads == null) {
            return;
        }
        
        final Method stopThread = ReflectionUtils.findMethod(SenderThread.class, "stopThread");
        ReflectionUtils.makeAccessible(stopThread);
        for (SenderThread senderThread : senderThreads) {
            ReflectionUtils.invokeMethod(stopThread, senderThread);
        }
    }
    
    private void deregisterJdbcDrivers() {
        logger.info("Deregistering JDBC Drivers");
        final Enumeration<Driver> driverEnumeration = DriverManager.getDrivers();
        final List<Driver> drivers = new ArrayList<Driver>();
        while (driverEnumeration.hasMoreElements()) {
            drivers.add(driverEnumeration.nextElement());
        }

        for (Driver driver : drivers) {
            if (driver.getClass().getClassLoader() != getClass().getClassLoader()) {
                logger.debug("Not deregistering {} as it does not originate from this webapp", driver.getClass().getName());
                continue;
            }
            try {
                DriverManager.deregisterDriver(driver);
                logger.debug("Deregistered JDBC driver '{}'", driver.getClass().getName());
                if ("oracle.jdbc.OracleDriver".equals(driver.getClass().getName())) {
                    deregisterOracleDiagnosabilityMBean();
                }
            } catch (Throwable e) {
                logger.error("Deregistration error", e);
            }
        }
    }

    private void deregisterOracleDiagnosabilityMBean() {
        final ClassLoader cl = Thread.currentThread().getContextClassLoader();
        try {
            final MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
            final Hashtable<String, String> keys = new Hashtable<String, String>();
            keys.put("type", "diagnosability");
            keys.put("name", cl.getClass().getName() + "@" + Integer.toHexString(cl.hashCode()).toLowerCase());
            mbs.unregisterMBean(new ObjectName("com.oracle.jdbc", keys));
            logger.info("Deregistered OracleDiagnosabilityMBean");
        } catch (javax.management.InstanceNotFoundException e) {
            logger.debug("Oracle OracleDiagnosabilityMBean not found", e);
        } catch (Throwable e) {
            logger.error("Oracle JMX unregistration error", e);
        }
    }
}

For my new T420s I decided to give Arch Linux a try after I had a short look at Mint, Debian, Fedora and openSUSE.

The Arch Way makes Arch different as it values simplicity and takes an elegant, minimalist approach. The core installation installs just what you absolutely need so you are greeted by a friendly command line when it’s done. After that you can easily install a graphical environment including Gnome 3 with or without shell, KDE and Xfce along with whatever packages you need. Software patches are kept to a minimum so most Arch packages are identical or at least very close to their upstream counterparts. The difference is similar to the “Google experience” of a Nexus One or Nexus S and the modified Android versions sold by the manufacturers. New versions of upstream software end up in Arch within days.

Arch is a rolling release distro that allows for a one-time installation and perpetual software upgrades. There is no need to reinstall or upgrade the system from one version to the next. Everything in Arch is bleeding edge.

Installation on my T420s did not work as smooth as Ubuntu. The latest official installation ISO is from May 2010 and does not support the network card of the T420s. There are newer testing versions but they have issues setting up full disk encryption. So I went with the official ISO and installed the updates via WLAN which worked well.

If you come from the Debian world you are used to apt for package management. Arch comes with pacman which is similar but a lot faster. Bringing your system up to date just means you have to run pacman -Syu instead of apt-get update && apt-get upgrade. After that you have the latest kernel (2.6.39 as of now). I Installed a few additional packages including XF86, Gnome, Chromium, Thunderbird and Libre Office and had a nice working system. Sound and the Touchpad including scrolling worked out of the box. For the network I decided to use Gnome’s NetworkManager so I installed network-manager-applet and modemmanager and got an easy to manage network setup with support for wired lan, WLAN and wireless broadband. My Ericsson F5521gw was recognized automatically and UMTS worked right away.

The Arch Wiki is a great resource and the home of the excellent documentation of Arch. For each part of the system you have multiple options to choose from and they are well explained.

Up to now I am really happy with Arch.

A few notes:

To setup focus follows mouse in Gnome run

gconf-editor /apps/metacity/general/focus_mode

and change the value from click to sloppy.

To make the massive Gnome 3 title bars a bit less high run

sed -i \
 "/title_vertical_pad/s/value=\"[0-9]\{1,2\}\"/value=\"0\"/g" \
 /usr/share/themes/Adwaita/metacity-1/metacity-theme-3.xml

sudo apt-get install git-svn
git svn clone file:///var/lib/svn/repos/asterisk-java \
  --no-metadata -A authors.txt \
  -t tags -b branches -T trunk asterisk-java

The authors file maps the users in Subversion to the git users. While Subversion usually uses a short username, git uses the full name with email address as a globally unique identifier for users. My authors file looked like this:

srt = Stefan Reuter <stefan.reuter@example.com>
root = Stefan Reuter <stefan.reuter@example.com>

As I had direct filesystem access to the Subversion repositories I chose to use the file protocol instead of HTTP so it was much faster. The --no-metadata option tells git-svn not to include the orignal Subversion revision number in every commit. This removes clutter from the history.

Finally git-svn creates branches for all tags in Subversion which is a bit nasty. So I converted them to git tags:

cd asterisk-java
for tag in `git branch -r | grep '^  tags' | sed 's,  tags/,,'`
do
  echo Converting Tag $tag
  git tag $tag tags/$tag
  git branch -r -d tags/$tag
done

There it is a nice git repository containing the full history. I only had to push it to github to make it available to everybody who is interested:

git remote add origin git@github.com:srt/asterisk-java.git
git push origin master
git push --tags

What can developers learn from CVE-2010-1622?

The exploit requires manipulating the class loader property so that it will download code from an external site. So you can prevent the attack by disallowing modifications of the class loader and by disallowing your application to download and run code from external sites.

Be explicit! Explicitly allow binding of certain properties. This prevents the exploit from working as there is no valid use case that requires access to the class property. Explicitly whitelisting properties also makes sure users cannot change the id of the object bound to a form or altering data that is managed internally like “date of creation” or “last modified by” properties.

To prevent code from external sites being downloaded and executed you can make sure your applications behaves well when run with a security manager. While this is a common concept used for client side code like applets it is far less common for server side applications. Tomcat usually works well with a security manager though it is not enabled by default. Making sure you appplications works with a security manager is also a variant of being explicit: You explicitly grant certain privileges to your code bases and disallow everything else that might be abused by attackers.

What can system administrators learn from CVE-2010-1622?

Your applications should run in a demilitarized zone where they are unable to access the internet or your intranet. If you really need access to external resources use a proxy server and white list the URLs your application needs to contact. Doing so prevents attackers from making your application download external code.

If your applications are built in a way that they work with a security manager use it! For Tomcat there is a short Howto available.

What could SpringSource have done better?

When Spring Source announced the release of 3.0.3 they reported to have fixed “more than a hundred minor issues” — no indication of the security fix. This could be understandable as they have released the fix 2 days prior to publishing the advisory. I do not understand why they did not announce it later however. The advisory was published as silently as possible although the vulnerability is rated critical, can be exploited remotely and probably affects a large number of applications.
I would have preferred receiving the security advisory through the usual channels used for announcements in addition to the security team page.

Having a look at cve.mitre.org reveals another interesting fact. The CVE id was assigned on April, 29th. That is almost 2 months before the advisory was published. The bug was fixed on May, 27th.
Why does it take more than 4 weeks for a 3 line fix? Why does it take almost 3 additional weeks after the fix to announce the vulnerability?
I would have preferred a priority fix as soon as possible after discovery and a release following short time after that.

Finally SpringSource dicided not to provide a fixed release for dm Server, a product based on Spring Framework, which is also vulnerable. Users are advised to manually patch it instead. SpringSource also continues to provide the vulnerable dm Server 2.0.2 for download without any warning.
I would have preferred to receive a fixed release of dm Server instead of seeing SpringSource continue to ship products containing known security issues.

What can you learn from CVE-2010-1622?

I will follow up with the lessons learned for application developers and system administrators in the next days. Stay tuned.

There is also an interesting analysis of the issue at blog.o0o.nu.

Our Code Style differs mainly in two aspects:

• We place curly braces on new lines
• We use a maximum line length of 130 characters

I am well aware that there are religious opinions regarding those questions but after reconsidering our style we came to the conclusion that both differences actually make sense.

Lining up curly braces vertically (called Allman or ANSI style) makes it easy to check that the braces match. The indented code is clearly set apart from the containing statement by lines that are almost completely whitespace which makes the code easier to read. In contrast to K&R style it consumes more space but today’s screens are well capable to show more than the ancient 24 lines.

Good:

if (condition)
{
    body;
}

Bad:

if (condition) {
    body;
}

The reason to limit the line length to 80 characters comes from the limitation of old printers that usually printed 80 characters per line. We seldomly print code on paper and modern printers are well capable to print at other resolutions. Limiting us to to 80 characters would waste a lot of screen space as with a decent screen size showing 130 characters still leaves enough place for the IDE to show up additional frames including outlines and project files left and right to the code.

What we learned: Use the mainstream conventions by default. Challenge them and check if the underlying assumptions hold true in your environment. Only derive if there is enough benefit to justify the change.