Bug #13247


CTF conversion fails with large files

Added by Andy Fiddaman 7 months ago. Updated 6 months ago.

tools - gate/build tools
Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


In continuing the rollout of CTF to more of OmniOS userland, I hit a memory scaling problem with a 30MiB shared library with 3,816 compilation units. ctfconvert cannot even initialise the die list without running out memory (ctfconvert is 32-bit).

Talking to Jonathan Perkin, he hit the same issue with the same shared library ( a while back, and wrote a patch for ctfconvert. This is open as and but CTF in gate has moved on a bit since.

Here's the write-up from his issue:

During the work to CTF convert pkgsrc, exposed memory scaling issues with ctfconvert. It is a 29MB shared library containing 3,449 DIEs.

The current conversion process allocates memory as follows:

  • A new ctf_die_t is created for each DIE during the initialisation process.
  • Each ctf_die_t holds a DWARF handle open on the input file.
  • Each ctf_die_t includes a new ctf_file_t allocation, each of which mmap()'s its own private copy of the CTF data, symtab, and strtab from the object.
  • During the merge process, for each DIE, an extra ctf_file_t is allocated, again with its own mmap'ed private copies, as the destination for the merge output.
  • These allocations are only freed at the end of the entire conversion process.
  • With all these allocations, a 32-bit ctfconvert process runs out of available memory while processing

With the proposed patch the conversion process is changed to be as follows:

  • Instead of allocating a full ctf_die_t for every DIE up-front, we instead defer full initialisation to the main conversion process in ctf_dwarf_convert_one().
  • DIEs are processed in batches, defaulting to a batch size of 256 (configurable via -b on the command line if necessary).
  • Each batch is converted and merged until a single merged ctf_file_t is returned as the result.
  • After processing a batch, the input DIEs for that batch are freed.
  • The merged ctf_file_t is added as an input to the next batch.
  • The process continues until we have processed all batches and end with a final merged ctf_file_t, or a failure.

The default batchsize of 256 is based on a few constraints:

  • Processing DIEs in multiple batches means that ctf_id_t's will be different compared to those generated by a previous ctfconvert. If we're able to choose a batchsize which is larger than the
    number of DIEs in most objects then we will avoid changing ctf_id_t's. Whilst mostly cosmetic, it's still nice to avoid differences if possible.
  • Performance goes up as the batchsize increases, at least when using the default of 4 threads.
  • The batchsize needs to be well below the number of DIEs that can be processed without hitting memory limits.

Related issues

Related to illumos gate - Bug #13251: CTF conversion fails if any CU is missing DWARF dataClosedAndy Fiddaman

Related to illumos gate - Bug #13252: ctf_update()/ctf_dwarf_convert_function() leak memoryClosedAndy Fiddaman

Related to illumos gate - Bug #13278: CTF assertion failed cmp->cm_tmap[id].cmt_map == suidClosedAndy Fiddaman


Also available in: Atom PDF