illumos gate

When doing builds of SmartOS we put objects into the proto area that will be used in the new system and cannot run on the current system. This includes things like a new libc which could have arbitrary changes relative to the currently running libc and kernel. As part of this, often tools are made as part of the build that run against the host system libraries. These tools should never link or use items in the proto area. While illumos is fairly clean in this respect, OS-2724 and OS-2215, show some of the many problems that we've had to deal with here. These generally hang the build or have spectacular failure modes.

To deal with this, we'd like to have the runtime link-editor (rtld) have a way to say that certain library locations are toxic. This does not change how the programs are loaded, but rather if they do try to load a toxic library, they should exit or potentially abort. We probably should leave suid/sgid binaries alone and we should probably also ignore the interpreter that they have set.

What this could look like is LD_TOXIC_PATH_32=<path>:<path>, LD_TOXIC_PATH_64=<path>:<path> and LD_TOXIC_PATH=<path>:<path>...

LD_TOXIC_PATH applies to everything, LD_TOXIC_PATH just to 32 bit dependencies, LD_TOXIC_PATH_64 to 64 bit dependencies. This can be specified similar to LD_LIBRARY_PATH; however, I believe that anything that matches <path> should fail. eg. it'd be great to be able to do LD_TOXIC_PATH=$WS_ROOT/proto and know that anything under there whether in $WS_ROOT/proto/lib $WS_ROOT/proto/usr/lib, or any of the 64-bit directories are all set.