signed math leads getelfshdr astray
This is an upstream of https://smartos.org/bugview/OS-6158 from illumos-joyent.
As several crashdumps have demonstrated, there are several parts of the in-kernel ELF handling which could benefit from having their unnecessary and incorrect use of signed types be cleaned up.
As part of the integration into illumos-joyent, the code was tested using https://github.com/IOActive/Melkor_ELF_Fuzzer
The same approach should be taken with the merge into gate.
Updated by Andy Fiddaman 6 months ago
I've tested this with a suite of corrupt and crafted ELF files that previously caused the system to variously panic, hang (usually due to all kernel memory being consumed) or end up with unkillable processes. In all cases, the patched system dealt with them correctly.
I also ran 50,000 fuzzed ELF files generated by https://github.com/omniosorg/Melkor_ELF_Fuzzer (the illumos branch there) with no problems. Previously, a run would stop within the first 100 or so files.
Using systems onud to these DEBUG and non-DEBUG bits, I've done full illumos-gate and omnios-userland builds with no problems.