illumos gate

Here's a reproducer:

#!/bin/ksh

tf=test$$.c

cat << EOM > $tf
int main(){return(0);}
EOM

for gcc in 4.4.4 7 9 10; do
       for a in 32 64; do
                [ $a = 32 ] && lib=/lib || lib=/lib/amd64
                echo
                echo "+ gcc $gcc, $a" 
                [ $gcc = 4.4.4 ] && b=/opt/gcc-4.4.4/bin/gcc || b=gcc-$gcc
                $b -m$a -o /dev/null -L$lib \
                    -Wl,-zassert-deflib -Wl,-zfatal-warnings $tf -lmtmalloc
        done
done

rm -f $tf

which outputs, for example with gcc 10 on OmniOS:

+ gcc 10, 32
ld: warning: dynamic library found on default search path (/usr/lib): libmtmalloc.so
collect2: error: ld returned 1 exit status

+ gcc 10, 64

The root cause of this is gcc invoking the linker with specific -L options for the standard library directory for a 64-bit link, whereas it elides them for a 32-bit one.
The linker sees that the library was found in a user provided path and so the assertion is not raised.

There are several possibilities on how to fix this. I have a test patch for gcc but will continue to explore options.

This has been integrated into the illumos gcc patch set in https://github.com/illumos/gcc/tree/il-10_3_0