Project

General

Profile

Actions

Feature #14090

closed

ld(1) could use a normal allocator

Added by Rich Lowe 8 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
tools - gate/build tools
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

At the minute, ld(1) uses a custom, mmap-based allocator and in the usual case frees no memory (apparently for performance).
This has proven to be a major disadvantage in the face of large-scale metaprogramming in the manner now common to C++ and Rust which may create vast amounts of input data, the vast majority of which we will discard.

The first step of the journey to make this better is to use a regular memory allocator, so that we can see what's going on and begin shaking out the low-hanging fruit, and hunting for the biggest wins.

We should make ld(1) use libumem, and libld no longer contain a custom allocator, falling back on the allocators of its consumers (such that ld(1) now uses umem, but ld.so.1 still uses its own custom libc-free malloc).


Related issues

Related to illumos gate - Bug #14127: ld(1) can double free when cleaning upClosedRich Lowe

Actions
Actions #1

Updated by Electric Monk 8 months ago

  • Gerrit CR set to 1715
Actions #2

Updated by Rich Lowe 8 months ago

testing:
Built illumos, installed the result and built illumos again and used the system because of the possible impact on ld.so
Checked we get the ld.so internal malloc
Checked a running binary on the new system doesn't contain umem in ld.so's linkmap.
Checked, in combination with jclulow, that we can extract useful umem debugging information from the link editor now.
Checked, in combination with jclulow, that allocator metadata doesn't (temporarily) make our problems worse.

Actions #3

Updated by Rich Lowe 8 months ago

testing:
Negligible effect on illumos build time

Actions #4

Updated by Electric Monk 8 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit fa4a3e77edc40df6d92e8da6fc4961d275e9896d

commit  fa4a3e77edc40df6d92e8da6fc4961d275e9896d
Author: Richard Lowe <richlowe@richlowe.net>
Date:   2021-09-29T18:57:11.000Z

    14090 ld(1) could use a normal allocator
    Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
    Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions #5

Updated by Marcel Telka 8 months ago

  • Related to Bug #14127: ld(1) can double free when cleaning up added
Actions #6

Updated by Rich Lowe 2 months ago

This was backed out as it introduced #14127, since fixed.
The same testing as then applies now, in addition jperkin did a full pkgsrc build to flush out anything similar to #14127

Actions #7

Updated by Electric Monk about 1 month ago

git commit fb12490ab4d1e87e7a63e457dd4fba1ea34c402a

commit  fb12490ab4d1e87e7a63e457dd4fba1ea34c402a
Author: Richard Lowe <richlowe@richlowe.net>
Date:   2022-04-21T20:37:47.000Z

    14090 ld(1) could use a normal allocator
    Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
    Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
    Reviewed by: Michael van der Westhuizen <r1mikey@gmail.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF