Testing LLVM 3.0 on Ubuntu 11.10 ARM

LLVM 3.0 was released some time ago and I’ve thought it’ll be good to give it a try on stock Ubuntu 11.10 ARM. That means I’ve tested LLVM 3.0 with Ubuntu provided GNU C++ 4.6.1 and Clang 2.9. GNU C++ configuration looks:

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabi/4.6.1/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.1-9ubuntu3' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-objc-gc --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) 

I’ve compiled LLVM 3.0 by those two compilers with default configuration and then with optimization flags set to -O0, -O1, -O2 and -O3. The table below lists sum of unexpected failures and unexpected passes with appropriate links to the tests output files. What’s surprising to me is that Clang on ARM even in version 2.9 performs so well. I know, Clang depends on LLVM and LLVM by default checks for Clang as a preferred compiler and both projects are mainly developed by Apple’s engineers, but still this is on native ARM/Linux system, so nothing like cross-compilation from MacOSX/x64 to iOS/ARM!

	-O0	-O1	-O2	-O3	default
GCC 4.6.1	6	6	51	51	51
Clang 2.9	1147	8	12	12	12

So as you can see GCC still wins on the lowest number of failures while using -O0/-O1, but Clang performs very well on -O2/-O3/default optimization levels. Please note that the excess number of failures on -O0 with Clang is probably caused by the fact that LLVM code requires some optimization to be performed on it to behaves correctly and it looks like Clang does not perform such optimization while GCC does when compiling with -O0.
Also what’s kind of surprise to me is to see Clang compilation performance. I’ve not marked hard numbers since this was not the task for this testing, but I’ve been surprised to see what GCC took around 700 minutes, Clang did in about 400 minutes. I’m talking about default compilation on i.MX53 Quick Start Board here.

Now the questions are: how Clang compiled LLVM affects GHC tests and GHC compilation speed? (i.e. I may use -O3 compiled LLVM for this). Also how would the numbers look like while testing latest greatest Linaro GCC and Clang 3.0? Perhaps material for another post or two…

LLVM on ARM testing

In my LLVM on ARM post I’ve claimed that the best GCC to compile LLVM on ARM natively (no cross compiling from x86/amd64 to ARM!) is GCC 4.4.1. Last week I kept all my available hardware running to test LLVM 2.9 and LLVM HEAD more thoroughly so I’ll have some proof that this claim is actually true. Well, it’s indeed true at least so far, but there are also other GCC versions able to get to the same level of quality as 4.4.1 while compiling LLVM source base on ARM.
Anyway, I’ve tested 6 GCC versions on both LLVM 2.9 and LLVM HEAD as of following commit

commit ca42299619cf47371a42c2bda87d067e003657ea
Author: Eric Christopher <echristo@apple.com>
Date:   Wed Jun 29 17:53:29 2011 +0000

    Move XCore from getRegClassForInlineAsmConstraint to
    getRegForInlineAsmConstraint.
    
    Part of rdar://9643582
    
    
    git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@134080 91177308-0d34-0410-b5e6-96231b3b80

I’ll list results in two tables, one for LLVM 2.9 and one for LLVM HEAD. Compilation parameters were -O0, -O1, -O2, and default which is -O3. I’ve run only LLVM tests which are distributed with LLVM itself (in the same package, not separate huge LLVM testsuite!) and which are invoked by make check. The tables lists tested GCC versions, compile parameters and a number of unexpected failures which I got from this combination. Interested reader might click on the number to download output of the make check command to see which tests exactly failed and why. I hope someone from LLVM community and also from Linaro community will actually do, I’ve done this testing as a service to both communities… Below the tables you can also find gcc -v output for all GCC version involved for the idea how compilers where configured and built.

LLVM 2.9 results:

	-O0	-O1	-O2	default
GCC 4.3.4 (1)	1	1	45	39
GCC 4.4.1 (2)	1	1	45	39
GCC 4.4.5 (3)	54	54	98	92
GCC 4.5.2 (4)	54	54	112	112
GCC 4.6.1/2011.05 (5)	1	1	59	59
GCC 4.6.1/2011.06 (6)	1	1	59	59

LLVM HEAD results:

	-O0	-O1	-O2	default
GCC 4.3.4 (1)	29	29	82	82
GCC 4.4.1 (2)	29	29	82	82
GCC 4.4.5 (3)	29	29	82	82
GCC 4.5.2 (4)	29	29	100	100
GCC 4.6.1/2011.05 (5)	29	29	100	100
GCC 4.6.1/2011.06 (6)	29	29	100	100

Interesting is number of regression presented in LLVM HEAD in comparison with LLVM 2.9. Compilers were configured as:

(1) GCC 4.3.4: run on Ubuntu 10.04.2 LTS on i.MX53 Quick Start Board together with binutils 2.20.1-system.20100303

Using built-in specs.
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.3.4-10ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --disable-sjlj-exceptions --with-arch=armv6 --with-tune=cortex-a8 --with-float=softfp --with-fpu=vfp --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.3.4 (Ubuntu 4.3.4-10ubuntu1) 

(2) GCC 4.4.1: run on Ubuntu 11.04 on Pandaboard together with binutils 2.21.0.20110327

Using built-in specs.
Target: arm-linux-gnueabi
Configured with: /export/home/karel/src/gcc-4.4.1-t2/gcc-4.4-4.4.1/src/configure -v --with-pkgversion='Ubuntu 4.4.1-4ubuntu9' --enable-languages=c,c++ --prefix=/export/home/karel/arm-sfw/gcc-4.4.1-4ubuntu9 --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --without-included-gettext --enable-threads=posix --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --disable-sjlj-exceptions --with-arch=armv6 --with-tune=cortex-a8 --with-float=softfp --with-fpu=vfp --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9) 

(3) GCC 4.4.5: run on Ubuntu 11.04 on Pandaboard together with binutils 2.21.0.20110327

Using built-in specs.
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.4.5-15ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.4 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.4.5 (Ubuntu/Linaro 4.4.5-15ubuntu1) 

(4) GCC 4.5.2: run on Ubuntu 11.04 on Pandaboard together with binutils 2.21.0.20110327

Using built-in specs.
COLLECT_GCC=gcc-4.5
COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4) 

(5) GCC 4.6.1 Linaro 2011.05: compiler compiled by me from Linaro 2011.05 source distribution. Run on Ubuntu 11.04 on Pandaboard together with binutils 2.21.0.20110327

Using built-in specs.
COLLECT_GCC=/export/home/karel/arm-sfw/gcc-4.6-linaro-2011.05-0/bin/gcc
COLLECT_LTO_WRAPPER=/export/home/karel/arm-sfw/gcc-4.6-linaro-2011.05-0/libexec/gcc/arm-linux-gnueabi/4.6.1/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../gcc-linaro-4.6-2011.05-0/configure -v --enable-languages=c,c++ --prefix=/export/home/karel/arm-sfw/gcc-4.6-linaro-2011.05-0 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --without-included-gettext --enable-threads=posix --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3 --with-mode=arm --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.6.1 20110506 (prerelease) (Linaro GCC 4.6-2011.05-0) 

(6) GCC 4.6.1 Linaro 2011.06: compiler compiled by me from Linaro 2011.06 source distribution. Run on Ubuntu 11.04 on Pandaboard together with binutils 2.21.0.20110327

Using built-in specs.
COLLECT_GCC=/export/home/karel/arm-sfw/gcc-4.6-linaro-2011.06-0/bin/gcc
COLLECT_LTO_WRAPPER=/export/home/karel/arm-sfw/gcc-4.6-linaro-2011.06-0/libexec/gcc/arm-linux-gnueabi/4.6.1/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../gcc-linaro-4.6-2011.06-0/configure -v --enable-languages=c,c++ --prefix=/export/home/karel/arm-sfw/gcc-4.6-linaro-2011.06-0 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --without-included-gettext --enable-threads=posix --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3 --with-mode=arm --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.6.1 20110526 (prerelease) (Linaro GCC 4.6-2011.06-0) 

I hope someone will find those results useful. If anything here is wrong or require correction or if someone needs to perform some additional test, please ask in comments.