Testing LLVM 3.1 on Ubuntu 11.10/12.04 ARM

Few weeks ago LLVM project released LLVM 3.1 release and I decided to give it a try on my ARM boards. I’ve tested on Ubuntu 11.10 on Freescale donated i.MX53 Quick Start Board and on Ubuntu 12.04 on Pandaboard. The results are pretty interesting as is shown in the table below. The table lists number of unexpected failures of basic LLVM testsuite when LLVM is compiled with specific optimization option and with specific GNU C compiler (on appropriate Ubuntu). Please note that Ubuntu 11.10 is last soft-float ABI Ubuntu and it provides GNU C 4.6.1:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabi/4.6.1/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.1-9ubuntu3' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-objc-gc --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) 

On the other hand, Ubuntu 12.04 is the first which provides hard-float ABI and it comes with GNU C 4.6.3:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/4.6/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.3-1ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-float=hard --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) 

And now, finally those interesting results. The numbers are clickable and link to the testsuite log file which you can see for your reference.

-O0 -O1 -O2 default
GCC 4.6.1 (Ubuntu 11.10) 6 6 54 54
GCC 4.6.3 (Ubuntu 12.04) 6 6 6 6

So it looks like GCC 4.6.3 did a very nice job here. Honestly speaking I’m not sure if this is GCC or ABI switch from soft-float to hard-float and I’m not able to verify it since Ubuntu 12.04 is only hard-float ABI but my bet is on GCC here.

Advertisements

Testing LLVM 3.0 on Ubuntu 11.10 ARM

LLVM 3.0 was released some time ago and I’ve thought it’ll be good to give it a try on stock Ubuntu 11.10 ARM. That means I’ve tested LLVM 3.0 with Ubuntu provided GNU C++ 4.6.1 and Clang 2.9. GNU C++ configuration looks:

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabi/4.6.1/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.1-9ubuntu3' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-objc-gc --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) 

I’ve compiled LLVM 3.0 by those two compilers with default configuration and then with optimization flags set to -O0, -O1, -O2 and -O3. The table below lists sum of unexpected failures and unexpected passes with appropriate links to the tests output files. What’s surprising to me is that Clang on ARM even in version 2.9 performs so well. I know, Clang depends on LLVM and LLVM by default checks for Clang as a preferred compiler and both projects are mainly developed by Apple’s engineers, but still this is on native ARM/Linux system, so nothing like cross-compilation from MacOSX/x64 to iOS/ARM!

-O0 -O1 -O2 -O3 default
GCC 4.6.1 6 6 51 51 51
Clang 2.9 1147 8 12 12 12

So as you can see GCC still wins on the lowest number of failures while using -O0/-O1, but Clang performs very well on -O2/-O3/default optimization levels. Please note that the excess number of failures on -O0 with Clang is probably caused by the fact that LLVM code requires some optimization to be performed on it to behaves correctly and it looks like Clang does not perform such optimization while GCC does when compiling with -O0.
Also what’s kind of surprise to me is to see Clang compilation performance. I’ve not marked hard numbers since this was not the task for this testing, but I’ve been surprised to see what GCC took around 700 minutes, Clang did in about 400 minutes. I’m talking about default compilation on i.MX53 Quick Start Board here.

Now the questions are: how Clang compiled LLVM affects GHC tests and GHC compilation speed? (i.e. I may use -O3 compiled LLVM for this). Also how would the numbers look like while testing latest greatest Linaro GCC and Clang 3.0? Perhaps material for another post or two…

LLVM on ARM

Since I’m going to try to use LLVM for GHC registerised build I need to make sure it’s kind of the best build I can get on the target hardware. To explain more: LLVM is really a picky project from the C++ compiler point of view. If you read some part of LLVM Getting Started manual (here) you will see a lot of references for “miscompilation” and such. Fortunately LLVM even in standard distribution comes with nice testsuite which you can use to verify quality of the build. So when I installed freescale’s recommended image it provides ubuntu’s gcc 4.4.3-1 compiler (I don’t know exactly now, since I’ve changed the compiler later, see below). When I tried to compile LLVM 2.9 by this I’ve got quite a lot of tests failing. To be precise 58 tests failing, this is not so nice. I’ve looked around and found that this gcc/ubuntu version defaults to thumb code generation, so next step was to enforce ARM code generation (-marm mode option) and viola I’ve been on 18 tests failures! Much better, but still not best since I remember in the past I’ve been able to get into just 1 failure, but this was probably on the older Ubuntu! Anyway, this Ubuntu compiler was tough so I started to downgrade the compiler and to make long story short I’ve ended on:

$ gcc -v
Using built-in specs.
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.4.1-4ubuntu9' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --program-suffix=-4.4 --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv6 --with-tune=cortex-a8 --with-float=softfp --with-fpu=vfp --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.4.1 (Ubuntu 4.4.1-4ubuntu9)

Mark this version since it’s able to compile LLVM 2.9 into a build where only one test is failing!

$ make check
llvm[0]: Running test suite
make[1]: Entering directory `/export/home/karel/src/obj-llvm/test'
Making a new site.exp file...
Making LLVM 'lit.site.cfg' file...
Making LLVM unittest 'lit.site.cfg' file...
( ulimit -t 600 ; ulimit -d 512000 ; ulimit -m 512000 ; ulimit -v 1024000 ; \
/export/home/karel/src/llvm-2.9/utils/lit/lit.py -s -v . )
FAIL: LLVM :: CodeGen/Thumb/select.ll (1593 of 5840)
******************** TEST 'LLVM :: CodeGen/Thumb/select.ll' FAILED ********************
Script:
--
/export/home/karel/src/obj-llvm/Release/bin/llc < /export/home/karel/src/llvm-2.9/test/CodeGen/Thumb/select.ll -march=thumb | grep beq | /export/home/karel/src/obj-llvm/Release/bin/count 1
/export/home/karel/src/obj-llvm/Release/bin/llc < /export/home/karel/src/llvm-2.9/test/CodeGen/Thumb/select.ll -march=thumb | grep bgt | /export/home/karel/src/obj-llvm/Release/bin/count 1
/export/home/karel/src/obj-llvm/Release/bin/llc < /export/home/karel/src/llvm-2.9/test/CodeGen/Thumb/select.ll -march=thumb | grep blt | /export/home/karel/src/obj-llvm/Release/bin/count 3
/export/home/karel/src/obj-llvm/Release/bin/llc < /export/home/karel/src/llvm-2.9/test/CodeGen/Thumb/select.ll -march=thumb | grep ble | /export/home/karel/src/obj-llvm/Release/bin/count 1
/export/home/karel/src/obj-llvm/Release/bin/llc < /export/home/karel/src/llvm-2.9/test/CodeGen/Thumb/select.ll -march=thumb | grep bls | /export/home/karel/src/obj-llvm/Release/bin/count 1
/export/home/karel/src/obj-llvm/Release/bin/llc < /export/home/karel/src/llvm-2.9/test/CodeGen/Thumb/select.ll -march=thumb | grep bhi | /export/home/karel/src/obj-llvm/Release/bin/count 1
/export/home/karel/src/obj-llvm/Release/bin/llc < /export/home/karel/src/llvm-2.9/test/CodeGen/Thumb/select.ll -mtriple=thumb-apple-darwin | grep __ltdf2
--
Exit Code: 1
Command Output (stderr):
--
Expected 3 lines, got 1.
--

********************
Testing Time: 928.90s
********************
Failing Tests (1):
LLVM :: CodeGen/Thumb/select.ll

Expected Passes : 5249
Expected Failures : 48
Unsupported Tests : 542
Unexpected Failures: 1
make[1]: *** [check-local-lit] Error 1
make[1]: Leaving directory `/export/home/karel/src/obj-llvm/test'
make: *** [check] Error 2

and in addition this is just a Thumb code generator error and I’m certainly not going to target Thumb! i.e. rather ARM or Thumb2 of course…