Saturday, April 28, 2012

Illegal Instruction in Code Compiled with Clang

I was inspired by the effort to rebuild the Debian archive with clang, to try compiling UnDBX with clang:
CC=clang ./configure
I was a bit surprised when clang complained about an (legitimate) operator precedence issue, which gcc never pointed out to me. I fixed this by judicious use of parenthesis, and recompiled.

But the generated binary failed to run - I got an "Illegal instruction" error. This error persisted with optimizations turned off (i.e. -O0). I tried running the binary under valgrind and got the following:
vex x86->IR: unhandled instruction bytes: 0x66 0xF 0xEF 0xC0
==28891== valgrind: Unrecognised instruction at address 0x8048fc3.
followed by a verbose error message explaining that this may be caused either by trying to execute code in a non-code area of memory, or by a bug in valgrind.

I used objdump to find the offending instruction/address:
$ objdump -d -S undbx | grep -i 8048fc3
 8048fc3:       66 0f ef c0             pxor   %xmm0,%xmm0

I don't grok x86 assembly much, so it took me some googling to find that the combination of pxor and xmm type registers is in fact an SSE instruction. The CPU on this box is an aging 32-bit Mobile AMD Athlon XP, and the CPU flags in /proc/cpuinfo include SSE, so I'm not sure what to make of this.

The same code compiles and runs nicely on my newly upgraded laptop which sports a slightly less lame 64-bit Mobile AMD Sempron processor, running a 64-bit kernel.

In any case, I tried compiling like this:
CC='clang -mno-sse' ./configure
and I got a working executable.

My guess is that this is a bug in clang (maybe Debian bug #632472?), but as I said, I'm not sure.


  1. PXOR, when on XMM registers (there is an MMX version) is SSE2. Athlon XP only has SSE1 (I had one).

    1. Thanks! This means that it is a compiler bug (or at least a compiler configuration bug). I should really ditch that box and be done with it...