Feature #14090
closed
ld(1) could use a normal allocator
Added by Rich Lowe 8 months ago.
Updated about 1 month ago.
Category:
tools - gate/build tools
Description
At the minute, ld(1) uses a custom, mmap-based allocator and in the usual case frees no memory (apparently for performance).
This has proven to be a major disadvantage in the face of large-scale metaprogramming in the manner now common to C++ and Rust which may create vast amounts of input data, the vast majority of which we will discard.
The first step of the journey to make this better is to use a regular memory allocator, so that we can see what's going on and begin shaking out the low-hanging fruit, and hunting for the biggest wins.
We should make ld(1) use libumem, and libld no longer contain a custom allocator, falling back on the allocators of its consumers (such that ld(1) now uses umem, but ld.so.1 still uses its own custom libc-free malloc).
testing:
Built illumos, installed the result and built illumos again and used the system because of the possible impact on ld.so
Checked we get the ld.so internal malloc
Checked a running binary on the new system doesn't contain umem in ld.so's linkmap.
Checked, in combination with jclulow, that we can extract useful umem debugging information from the link editor now.
Checked, in combination with jclulow, that allocator metadata doesn't (temporarily) make our problems worse.
testing:
Negligible effect on illumos build time
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit fa4a3e77edc40df6d92e8da6fc4961d275e9896d
commit fa4a3e77edc40df6d92e8da6fc4961d275e9896d
Author: Richard Lowe <richlowe@richlowe.net>
Date: 2021-09-29T18:57:11.000Z
14090 ld(1) could use a normal allocator
Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
Approved by: Robert Mustacchi <rm@fingolfin.org>
- Related to Bug #14127: ld(1) can double free when cleaning up added
This was backed out as it introduced #14127, since fixed.
The same testing as then applies now, in addition jperkin did a full pkgsrc build to flush out anything similar to #14127
git commit fb12490ab4d1e87e7a63e457dd4fba1ea34c402a
commit fb12490ab4d1e87e7a63e457dd4fba1ea34c402a
Author: Richard Lowe <richlowe@richlowe.net>
Date: 2022-04-21T20:37:47.000Z
14090 ld(1) could use a normal allocator
Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
Reviewed by: Michael van der Westhuizen <r1mikey@gmail.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by