illumos gate

Interesting. I've started using gcc51 in OmniOS's bloody for building non-ON bits. Does libc's lack of support here cause a problem?

Downloading any recent (as of this comment) OmniOS bloody should have gcc51-built omnios-build userland bits in it. I'd appreciate knowing if there are any interaction problems.

It kind of does cause problems. If you pass -fstack-protector when compiling libssp gets linked in automatically, but if you are building objects separately and then do the final link with ld libssp does not get linked in, and you get unresolved symbol errors. The solution is to make sure that -lssp is passed when linking with ld. Linux (and BSD, afaik) include SSP support directly in libc so there's no need to futz with the separate libssp library.

The "on by default in newer gcc" from a similar bug had me concerned. I see no libssp linkages in omnios-build built tools, so perhaps 5.1.0 doesn't add -fstack-protector by default.

I haven't seen it either. It must be disabled on Solaris/illumos, probably due to the issues I mentioned above.

I suppose adding some filter symbols to libc should be good enough.

EDIT: Never mind. That won't help.

To support the stack protector there are a few things we need:

1. libc needs the __stack_chk_fail function
2. libc needs the __stack_chk_guard pointer that is initialized with random data early on in program lifetime (more on how we do that later)
3. i386 needs the ability to link in the __stack_chk_fail_local function which is what was traditionally delivered by libssp_nonshared.a
4. The compiler needs to be updated to know about number 3.

What we have put together implements both 1 and 2 in libc. We use getrandom(2) to get random data for guard early on in the program. If that fails we fallback to a variant of the GCC stackguard paper's recommendation of the canary suggested by Alex Wilson. We use the '\0' and '\n' bytes, but also use the low two bytes of gethrtime() as a way to have some poor-quality randomness in there that the attacker can't readily guess, but still have the terminator canaries that are more likely to stop a wayward string function.

One difference from gcc, though more like musl, is that we do not attempt to try and print out anything to stderr when this occurs. We consider the process to be in a compromised state (and honestly it's hard to know that we'll be able to honestly call into libc) and so just call upanic(2).

For issue number three we are implementing an archive library that we're calling libssp_ns.a. This purposefully has a different name so that folks who have the existing gcc-based stack protector libraries installed do not have a conflict in any way. This will allow the illumos/gcc spec files to be updated to point to this to make this a first class citizen in those environments, though it also allows one the ability to link with -lssp_ns manually as we are doing in #13274 to avoid the compiler flag day.

To test the stack protector we did a few things:

• We wrote small test programs that would trigger it on different heuristics and verified that we triggered the stack protector and the upanic(2) debugging support.
• I went back and undid the ix for #13242 and verified that upon replicating the issue, that the stack protector would have caught it and that we properly aborted the process.
• When running with #13274 we were able to catch a small issue in libvarpd/kernel interfaces and verified that this fired and worked in a multi-threaded daemon.
• Andy wrote up patches for gcc and verified that everything worked end to end there.