/usr/bin/ld: cannot find crt1.o: No such file or directory
/usr/bin/ld: cannot find crti.o: No such file or directory
collect2: ld returned 1 exit status

It turns out that recent Debian releases and their derivatives have transitioned to multiarch support. The idea is to allow seamless support of binaries built for different architectures, related or not, to live concurrently on the same system. In theory it should be more portable than the lib32/lib64 conventions of the past. Unfortunately, the side effect is that some applications depending on the old paths are now woefully broken. The good news is a simple workaround will allow older GCC versions to build without issue. Building an older version of GCC is still straightforward when following the instructions in my previous post, applying the necessary patches and running the following command to fix-up the compiler paths:

sed -i 's|\.\./lib64|x86_64-linux-gnu|' gcc-<version>/gcc/config/i386/t-linux64

For example, the following will build and install GCC 4.7.0 from source to a custom prefix:

tar xzf gcc-4.7.0.tar.gz
sed -i 's|\.\./lib64|x86_64-linux-gnu|' gcc-4.7.0/gcc/config/i386/t-linux64
mkdir ../gcc-4.7.0-objdir
cd ../gcc-4.7.0-objdir
../gcc-4.7.0/configure --prefix=/opt/x86_64/gcc/gcc-4.7.0 --enable-languages=c,c++
make
make install

Voila! Older GCC versions should now build and compile applications without issue.

// iteration sadness :(
final NodeList nodes = document.getChildNodes();
for(int i = 0; i < nodes.getLength(); i++) {
  final Node node = nodes.item(i);
  // use node
}

Ideally, the code could have been written utilizing Java 5′s foreach loop.

// iteration happiness which is currently impossible
for(final Node node : document.getChildNodes()) {
  // use node
}

As a workaround, I’ve written a simple wrapper class for org.w3c.dom.NodeList that makes it iterable. Hopefully, it’s useful for others.

Download: IterableNodeList.java

package com.trevorpounds.dom.util;
 
import java.util.Iterator;
import java.util.NoSuchElementException;
 
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
 
/**
 * Simple org.w3c.dom.NodeList iterable wrapper
 * that can be used with Java's foreach loop.
 *
 * @author Trevor Pounds
 */
public class IterableNodeList implements Iterable<Node>, Iterator<Node> {
   /** List of nodes */
   private final NodeList nodeList;
 
   /** List of nodes current index. */
   private int index = 0;
 
   /** Constructs iterable node list. */
   public IterableNodeList(final NodeList nodeList) {
      this.nodeList = nodeList;
   }
 
   /** {@inheritDoc} */
   public boolean hasNext() {
      return index < nodeList.getLength();
   }
 
   /** {@inheritDoc} */
   public Iterator<Node> iterator() {
      return this;
   }
 
   /** {@inheritDoc} */
   public Node next() throws NoSuchElementException {
      if(!hasNext()) {
         throw new NoSuchElementException();
      }
      return nodeList.item(index++);
   }
 
   /** {@inheritDoc} */
   public void remove() throws IllegalStateException, UnsupportedOperationException {
      throw new UnsupportedOperationException();
   }
}

Utilizing the wrapper class we can rewrite the sad code above to the following.

for(final Node node : new IterableNodeList(document.getChildNodes())) {
  // use node
}

For example suppose I wanted to model the following C function that returns a simple pass/fail result and outputs a newly allocated integer using a pointer-to-pointer.

int alloc_opaque_type(T** ppT);

The JNI equivalent would instead substitute the method’s first argument with a long primitive array.

public static native int alloc_opaque_type(long[] ppT);

The native JNI implementation then requires a slight pointer fixup between the Java primitive array type and the native pointer type which is accomplished using a simple union cast below.

JNIEXPORT jint JNICALL Java_Wrapper_alloc_1opaque_type
   (JNIEnv* env, jclass klass, jlongArray ppT)
{
   union
   {
      T* p;
      jlong    l;
   } __u;
   int retval = alloc_opaque_type(&__u.p);
   env->SetLongArrayRegion(ppT, 0, 1, &__u.l);
   return retval;
}

After the native code is compiled and loaded, you can then use the method with an opaque pointer as follows:

public long allocAndGetPointer()
{
   long[] opaque_ptr_ptr = new long[] { 0 };
   if(alloc_opaque_type(opaque_ptr_ptr))
      { return long[0]; }
   throw new RuntimeException("Could not allocate opaque pointer!");
}

class VMObject
{
   klass_type* klass;
   lock_type*  lock;
};

class VMArray : VMObject
{
   size_t length;
};

I won’t get into too much detail about the klass_type pointer since this one speaks for itself. It’s primary purpose is to provide all of the runtime type and method dispatch information to the VM. The lock_type is also straight forward providing the internal mechanism for Java’s synchronized keyword. However, it also happens to double every object’s overhead whether or not synchronization is used at all. A good resource that I found touching on the subject is available on the c2.com wiki which outlines the pitfalls of the Java object model and parallels my own independent discoveries. Unfortunately, Java was never designed for data-only types (i.e. structs in C/C++). As a result, all objects in Java require both fields regardless of synchronization or virtual dispatch. I came to the same conclusion as the authors at c2.com wiki; the memory overhead can only be reduced in two ways: (1) provide a method to mark objects as unsynchronizable and (2) incorporate a data-only type. The changes are not impossible; in fact both could be retrofitted into future Java releases with a little bit of effort. The addition of an java.lang.concurrent.Unsynchronizable interface would provide a backward compatible way for the Java compiler and VM’s object model to support a lockless subtype. Supporting a data-only type is a bit trickier but could be achieved again using an updated Java object model in addition to some compile-time and run-time optimizations.

@Metadata(name="size (KB)", value="1024")
@Metadata(name="size (MB)", value="1")
@Test public void myPerfTest()
{
   // do stuff
}

Unfortunately, multiple annotations are not allowed; this is apparent when you try to compile via javac.

./org/junitbench/examples/MetadataTest.java:[23, 18] duplicate annotation

Additionally, the java reflection API does not provide a mechanism for retrieving multiple annotations by name (see: java.lang.reflect.AnnotatedElement.getAnnotation(…)). If it was explicitly allowed by the language I could probably scrape by using one of the existing reflection API’s get*Annotations(…) methods with a custom annotation filter. However, since it is not allowed in any capacity another workaround was required; I did this by defining an enclosing annotation type that accepted an unlimited number of nested annotation types.

@Metadata({
   @Data(name="size (KB)", value="1024"),
   @Data(name="size (MB)", value="1")
})
@Test public void myPerfTest()
{
   // do stuff
}

The above annotation types are defined as follows:

@Inherited
@Retention(RetentionPolicy.RUNTIME)
@Target({})
public @interface Data
{
   String name();
   String value();
}

@Inherited
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD})
public @interface Metadata
{
   Data[] value();
}

The important thing to be aware of is the use of the nested @Data annotation as the array value type within the enclosing @Metadata annotation definition.

env = Environment()
GCC_VERSION = commands.getoutput(env['CXX'] + ' -dumpversion')

Note, in the example I opted to use the -dumpversion flag instead of the -v or –version flags because from my experience I have found the results can be inconsistent, from a parsing aspect, depending on the GCC build being used (i.e. differences in vanilla and GNU/Linux distribution patched builds). After detecting the version, the results are used to configure the appropriate compile time and link flags (e.g. -O1 vs. -O2, -m32 vs. -m64, etc.). The version is also used to determine which run-time dependencies (i.e. libstdc++.so, libgcc_s.so) need to be linked and distributed with the application. Fortunately, a convenient mechanism exists, the -print-file-name option, which can be used to determine GCC’s library awareness. For example, the option can be used to resolve the libstdc++.so and libgcc+_s.so dependencies as follows.

g++-4.1 -print-file-name=libgcc_s.so.1
/usr/lib/gcc/x86_64-linux-gnu/4.1.3/../../../../lib/libgcc_s.so.1

g++-4.1 -m32 -print-file-name=libgcc_s.so.1
/usr/lib/gcc/x86_64-linux-gnu/4.1.3/../../../../lib32/libgcc_s.so.1

g++-4.1 -print-file-name=libstdc++.so.6
/usr/lib/gcc/x86_64-linux-gnu/4.1.3/../../../../lib/libstdc++.so.6

g++-4.1 -m32 -print-file-name=libstdc++.so.6
/usr/lib/gcc/x86_64-linux-gnu/4.1.3/../../../../lib32/libstdc++.so.6

Additionally, the -print-file-name option works in conjunction with the -m32 or -m64 flags. This allows the build system, depending on the bit-ness of the target system, to resolve the absolute path of the respective i386 or x86-64 compatible libraries. However, I did find that the -print-file-name option in GCC versions <= 3.2.1 seemed to resolve to invalid paths in some circumstances. For those particular compilers another mechanism would need to be used.

As an addendum, I've compiled a quick cross-reference of various GCC features which I acquired from some of my previous work as well as utilizing some of the compiler options above.

* Technically i386/x86-64 multilib support is available in these versions but does not seem to be as widley supported. See the notes in my previous post for more info.

** Does not work in all cases. In my limited amount of testing I found that GCC 3.2.1 and earlier had trouble resolving libstdc++.so shipped with the compiler. I would test this option thoroughly if it needs to be used with older compilers.

/lib – 32-bit libraries
/lib64 – 64-bit libraries

However, on Debian based systems the following convention is used:

/lib – 64-bit libraries
/lib64 – symlink to /lib
/lib32 – 32-bit libraries

Unfortunately, this convention confuses the older GCC configuration scripts. However, this limitation has been resolved in later releases.

GCC >= 4.2.4
To start off, I’ll talk about building modern GCC versions. From my limited testing I found that any version starting with 4.2.4 and later can be built by following the standard GCC build procedure outlined on the web. However, for documentation purposes, I’ve listed the commands I used to build and install GCC 4.4.2 to a custom installation prefix (e.g. /opt).

tar xzf gcc-4.4.2.tar.gz
mkdir gcc-4.4.2-objdir
cd gcc-4.4.2-objdir
../gcc-4.4.2/configure --prefix=/opt/x86_64/gcc/gcc-4.4.2 --enable-languages=c,c++
make
make install

multilib support is auto-enabled using the above configure options but it can also be explicitly enabled by adding –enable-multilib to the argument list.

GCC 3.3.x to 4.2.3
Building versions 3.3.x to 4.2.3 is a little more involved but essentially boils down to applying a slightly modified version of the lib path awareness patch mentioned above. For simplicity’s sake I’ve attached the patch (gcc-multilib-fix-v3.3.x-to-v4.2.3.debian.x86_64.diff) and used the following commands to build the compiler.

tar xzf gcc-3.4.6.tar.gz
cd gcc-3.4.6
patch -p0 < gcc-multilib-fix-v3.3.x-to-v4.2.3.debian.x86_64.diff
mkdir ../gcc-3.4.6-objdir
cd ../gcc-3.4.6-objdir
../gcc-3.4.6/configure --prefix=/opt/x86_64/gcc/gcc-3.4.6 --enable-languages=c,c++
make
make install

GCC 3.2.x
Building GCC 3.2.x requires following the same basic steps as above but requires an additional fix that prevents emitting an .eh_frame for the crt*S.o objects. The original bug fix can be found on the GCC mailing list. To build this version of GCC, download the attached patch (gcc-multilib-fix-v3.2.x.debian.x86_64.diff) and perform the following commands.

tar xzf gcc-3.2.3.tar.gz
cd gcc-3.2.3
patch -p0 < gcc-multilib-fix-v3.2.x.debian.x86_64.diff
mkdir ../gcc-3.2.3-objdir
cd ../gcc-3.2.3-objdir
../gcc-3.2.3/configure --prefix=/opt/x86_64/gcc/gcc-3.2.3 --enable-languages=c,c++
make
make install

multilib i386/x86-64 support is available in this version, however, I seem to recall reading on the web that proper x86-64 support did not become production ready until the 3.3.x releases. At the time of this post I could not find any resource to back this claim. However, I did find the quote, "The x86-64 port has been significantly improved," in the 3.3.x change log which potentially confirms this. In any case, I compiled a few simple programs for an x86-64 target and none of them had any run-time problems as far as invalid calculations or heinously crashing but if anyone knows for sure I would be interested to know.

GCC 3.1.x
Up until now most of the builds were fairly easy to "frontport" to run on x86-64 Debian. With GCC 3.1.x, I began to run into more difficult problems. Along with the .eh_frame patch used above, this version needed work to fix the lib32/lib64 path lookup mechanisms for multilib support; FWIW, this is the first GCC version that supported x86-64 host/targets. Also, an additional workaround was needed to forward some older libc symbols (e.g. __ctype_*) to their corresponding available equivalents (e.g. __ctype_*_loc). I've compiled the corresponding fixes into a patch (gcc-multilib-fix-v3.1.x.debian.x86_64.diff) which can be used to build this version.

tar xzf gcc-3.1.1.tar.gz
cd gcc-3.1.1
patch -p0 < gcc-multilib-fix-v3.1.x.debian.x86_64.diff
mkdir ../gcc-3.1.1-objdir
cd ../gcc-3.1.1-objdir
../gcc-3.1.1/configure --prefix=/opt/x86_64/gcc/gcc-3.1.1 --enable-languages=c,c++ --enable-threads=posix
make
make install

Without the --enable-threads=posix flag, the configure script will build using the default thread model (e.g single). If you do not wish to support posix threads the option can be omitted.

GCC 3.0.x
As I mentioned above, the GCC 3.1.x series of compilers were the first to natively support x86-64 targets. In order to build GCC 3.0.x (for an i386 target) on a x86-64 host the source required a change to the GCC standard library prefix directories (e.g. /lib to /lib32) and a few tool-chain workarounds (e.g. -m32 flags). Probably, the most interesting part of the changeset is the addition of --32 to the GCC assembler spec options. Without the flag, the assembler chokes with an invalid operand error (e.g. Error: suffix or operands invalid for `push'). Alternatively, I could have specified -Wa,--32 on the GCC command line to workaround the tool-chain differences but doing this is more burdensome and error prone than having the default behavior compiled into the spec file. Additionally, the same __ctype_* symbol lookup change in the GCC 3.1.x patch is also required. Again, I've provided the compatibility changes in a patch (gcc-v3.0.x.debian.x86_64.diff) which can be built using the following commands.

tar xzf gcc-3.0.4.tar.gz
cd gcc-3.0.4
patch -p0 < gcc-v3.0.x.debian.x86_64.diff
mkdir ../gcc-3.0.4-objdir
cd ../gcc-3.0.4-objdir
../gcc-3.0.4/configure --prefix=/opt/i386/gcc/gcc-3.0.4 --enable-languages=c,c++ --enable-threads=posix --host i386-pc-linux-gnu
make
make install

GCC 2.95.x
Finally! We come to the GCC 2.95.x series which, rightfully so, required quite a bit of effort to configure and build for an x86_64 host. About half of the changes needed for GCC 3.0.x are also applicable here (e.g. Makefile fixes, lib32 fix-ups, spec flags, etc). Unfortunately, the other half are workarounds for compiler and glibc nuances that are fixed in later releases. One of the big changes needed is a workaround for the internal use of an _IO_MTSAFE_IO define. This define causes an error due to a missing header due to difference in glibc headers included with modern Linux distributions (see here and here for more info). Luckily, a straightforward workaround is to use glibc's generic stdio-lock.h header, which in my case needed to be pulled from the glibc 2.10.1 source to match my installed version. Additionally, another glibc fix is required due to an unnamed union declaration (see this bug) inside a pthread header which is not supported by GCC 2.95.x. To workaround the issue I pulled in the glibc 2.10.1 system dependent pthreadtypes.h header from the source and appended a variable name (e.g. __gcc_295_workaround__) to the unnamed union declaration. Build this version using the provided patch (gcc-v2.95.x.debian.x86_64.diff) and the following instructions.

tar xzf glibc-2.10.1.tar.gz
mkdir -p gcc-2.95.3/glibc-workaround/include/bits
cp glibc-2.10.1/bits/stdio-lock.h gcc-2.95.3/glibc-workaround/include/bits
cp glibc-2.10.1/nptl/sysdeps/unix/sysv/linux/x86_64/bits/pthreadtypes.h gcc-2.95.3/glibc-workaround/include/bits
sed -i -n '1h;1!H;${;g;s/\(__pthread_slist_t __list;\n[ \t]*}\)/\1 __gcc_295_workaround__/g;p;}' gcc-2.95.3/glibc-workaround/include/bits/pthreadtypes.h
tar xzf gcc-2.95.3.tar.gz
cd gcc-2.95.3
patch -p0 < gcc-v2.95.x.debian.x86_64.diff
mkdir ../gcc-2.95.3-objdir
cd ../gcc-2.95.3-objdir
../gcc-2.95.3/configure --prefix=/opt/i386/gcc/gcc-2.95.3 --enable-languages=c,c++ --enable-threads=posix --enable-shared --host i386-pc-linux-gnu
make
make install

Be sure to use the same version of the glibc source for the patches as the one of the target system. Since my target/host system uses glibc 2.10.1 I made sure to pull the corresponding source.

Additional Notes
I used GCC 3.4.6 in all cases when building the GCC 2.95.x and 3.x.x versions. The host compiler can be configured by setting the CC environment variable to the full path of your compiler (e.g. CC=/opt/x86_64/gcc/gcc-3.4.6/bin/gcc). All of the GCC 4.x.x versions can pretty much be built with any of the older GCC compilers. However, your mileage may vary in some circumstances. In those cases you can always try to build with a different host compiler when the current one is not working out. What seemed to work best for me was building each subsequent version in descending order and bootstrapping a failed build using an older GCC host if necessary.

To force linking against a particular symbol you need to use the same .symver pseudo-op that is used for defining versioned symbols in the first place. In the following example I make use of glibc’s realpath, but want to make sure it is linked against an older 2.2.5 version.

#include <limits.h>
#include <stdlib.h>
#include <stdio.h>
 
__asm__(".symver realpath,realpath@GLIBC_2.2.5");
int main()
{
   char* unresolved = "/lib64";
   char  resolved[PATH_MAX+1];
 
   if(!realpath(unresolved, resolved))
      { return 1; }
 
   printf("%s\n", resolved);
 
   return 0;
}

If you were to use objdump on the resulting binary you would see that it is indeed using realpath@GLIBC_2.2.5! Also note that other symbols have been resolved to their defaults so you need to make sure you add a .symver pseudo-op for each symbol you want to force to an older version.

0000000000000000      F *UND*  0000000000000000         realpath@GLIBC_2.2.5
...
0000000000000000      F *UND*  0000000000000000         __stack_chk_fail@@GLIBC_2.4

The .symver pseudo-op can be used this way to force any symbol to be linked against an older one so long as it is valid. To ease linking against older glibc versions I’ve provided a simple header which can be used to force linking against the minimum glibc version for a give x86 architecture. The minimum versions I am using were taken from shlib-versions file of the glibc git tree.

#ifndef __GLIBC_COMPAT_SYMBOL_H__
#define __GLIBC_COMPAT_SYMBOL_H__ 1
 
/**
 * add other architecures below
 */
#ifdef __amd64__
   #define GLIBC_COMPAT_SYMBOL(FFF) __asm__(".symver " #FFF "," #FFF "@GLIBC_2.2.5");
#else
   #define GLIBC_COMPAT_SYMBOL(FFF) __asm__(".symver " #FFF "," #FFF "@GLIBC_2.0");
#endif /*__amd64__*/
 
#endif /*__GLIBC_COMPAT_SYMBOL_H__*/

To use the glibc compatible header with the realpath example above, we merely use the GLIBC_COMPAT_SYMBOL macro with the appropriate symbol:

GLIBC_COMPAT_SYMBOL(realpath)

To get a basic idea of what shared libraries and symbols were listed in the wrapper binary I ran the following command:

objdump -x wrapper-linux-x86-32

Which gives quite a bit of useful info about the executable:

...
Dynamic Section:
NEEDED               libm.so.6
NEEDED               libpthread.so.0
NEEDED               libc.so.6
...
Version References:
required from libpthread.so.0:
0x0d696912 0x00 05 GLIBC_2.2
0x0d696911 0x00 04 GLIBC_2.1
0x0d696910 0x00 03 GLIBC_2.0
required from libc.so.6:
0x0d696913 0x00 08 GLIBC_2.3
0x0d696911 0x00 07 GLIBC_2.1
0x0d696912 0x00 06 GLIBC_2.2
0x0d696910 0x00 02 GLIBC_2.0
...
SYMBOL TABLE:
...
00000000    F *UND*  00000167      strchr@@GLIBC_2.0
00000000    F *UND*  00000078      nanosleep@@GLIBC_2.0
...
00000000    F *UND*  00000496      realpath@@GLIBC_2.3
...

In the output above I’ve omitted the uninteresting parts and left the important tidbits for the purposes of this article. (1) We can see that this binary requires the shared libraries: libm.so.6, libpthread.so.0, libc.so.6, (2) it has various glibc version references: 2.0, 2.1, 2.2, 2.3, and (3) has some interesting symbols appended with a unique version suffix.

So what? Well, after grepping through the large output from objdump I found that the only requirement on glibc 2.3 was the realpath function. Hmm…that’s interesting, if it wasn’t for this one function the executable would be binary compatible with the glibc installed on this old 2.4 linux distribution. Also, after browsing through the git tree I found that realpath has been available since at least glibc 2.1 (i.e. stdlib/canonicalize.c@202). So why is the dynamic linker requiring a newer version of glibc even though the function has been around for so long? The answer has to do with the versioned symbol scheme introduced in glibc 2.1 (http://people.redhat.com/drepper/symbol-versioning) which is an “extension” of Sun’s own symbol versioning scheme (http://docs.sun.com/app/docs/doc/817-1984/appendixb-45356?a=view).

As I mentioned above realpath has been around since glibc 2.1, yet this executable requires 2.3. The mechanism behind this is that the linker can be used to create global versioned symbol aliases to local symbols generated at compiled time. The internal linker interface provides two mechanisms for defining the aliases and requires the source to be inlined with a simple assembly pseduo-op .symver.

.symver actual, alias@version

.symver actual, alias@@version

The single @ op can be used to define any number of versioned symbols with the same base name. The double @@ can only be defined once for a given symbol since it denotes the default version to use. The linker also requires the use of a map file for defining if a symbol is global or local. A global symbol can be exported, while a local symbol is kept private. Each entry within the map file should correspond to a given version within the source file’s .symver definition.

VER_1.0 {
   global: alias
   local: *
};

To illustrate the full mechanics of the versioned symbol linking mechanism I’ve provided a better example. Say for example we release a library version 1.0 with a function foo. We could version it by adding the simple assembly to the function definition in the source foo.c:

__asm__(".symver foo, foo@@FOO_1.0");
int foo() {
   return 0;
}

Then we would need to create a map file (foo.map):

FOO_1.0 {
   foo;
};

After which we would compile our simple shared library.

gcc -shared -fPIC -Wl,--version-script foo.map foo.c -o libfoo.so.1

At this point we could distribute the library with the foo.h and let others compile against it to their hearts content. But what happens when we want to release another version of the library with potential behavioral changes that may affect the use of foo in other programs? We could introduce a new foo with with the updated behavior and still keep the old fucntion around for legacy programs. This can be done by renaming the original definition of foo to foo_1_0 and adding a new definition of foo called foo_1_1, like so:

/* old foo */
__asm__(".symver foo_1_0, foo@FOO_1.0");
int foo_1_0() {
   return 0;
}
 
/* new foo */
__asm__(".symver foo_1_1, foo@@FOO_1.1");
int foo_1_1() {
   return -1;
}

Then we update our existing map file (foo.map) to the following:

FOO_1.0 {
   foo;
};
FOO_1.1 {
   foo;
} FOO_1.0;

After which we would compile our simple shared library again and redistribute to our customers noting the change in behavior. Programs compiled against the old version and new version can now operate concurrently utilizing the updated library.