Project

General

Profile

Actions

Bug #14418

closed

mman.h symbol visibility is a mess

Added by Robert Mustacchi 6 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Category:
headers
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Right now mman.,h has many different contoritions for getting access to symbols in its #ifdef's. However, critically, even if you ask for __EXTENSIONS__ and a bunch of XPG standards, which should really be expected these days, there is no way to get access to a number of the symbols such as getpagesizes or memcntl. In addition, there are old versions of several of the functions that take caddr_t arguments rather than the standardized void *.

Now, mman.h has a bit of history due to the fact that it contains both things that were visible in POSIX.1/POSIX.2 before the merge with the eventual single unix spec. The following symbols were present in SUS V4.2:

  • mmap
  • munmap
  • mprotect
  • msync

The following symbols were all added in SUSv5 and previously were part of the POSIX realtime specs:

  • mlock
  • munlock
  • mlockall
  • munlockall
  • shm_open
  • shm_unlink

SUSv6 / POSIX added a single function, posix_memalign.

Finally, we have a bunch of functions that are extensions that we've added over the decades:

  • mincore
  • memcntl
  • getpagesizes
  • getpagesizes2
  • mmapobj
  • meminfo

In this change, we opt to standardize each of these groups. In particular we now make every block in the vein of strict symbols, were appropriate. The following table summarizes what we've done:

Type .Pre-Processor
Our Extensions !defined(STRICT_SYMBOLS)
mmap group Nothing -- always visible
mlock group !defined(STRICT_SYMBOLS) OR defined(POSIX_C_SOURCE) OR defined(XPG_5)
posix_memalign !defined(_STRICT_SYMBOLS) OR defined(_XPG6)

It's worth a bit of history and explanation with these. The easy ones here for the moment are posix_memalign, because it's a simple XPG6 addition. Our own extensions have used various awkward symbol gates; however, the relevant things for us is basically no strict symbols being defined. We've opted for this as a simple way to get the no-posix or extensions defined. The mmap group was always defined in illumos headers, regardless of whether standards were present. The major difference there was whether or not the functions described took a caddr_t or a void *. As part of this clean up, we opt to only expose the void * to software, hoping this'll result in a more portable world. Finally the mlock group was in the realtime posix group which requires POSIX_C_SOURCE > 2 and weren't in XPG until v5, so hence we have that morass.

This is still a work in progress so the above is subject to change.


Related issues

Has duplicate illumos gate - Bug #14504: madvise() not declared if _XOPEN_SOURCE definedDuplicateRobert Mustacchi

Actions
Actions #1

Updated by Robert Mustacchi 6 months ago

  • Description updated (diff)
Actions #2

Updated by Electric Monk 6 months ago

  • Gerrit CR set to 1965
Actions #3

Updated by Robert Mustacchi 6 months ago

A first pass set of tests was a new suite of symbol visibility tests for this:

rm@beowulf:~$ pfexec /opt/libc-tests/bin/libctest 
Test: /opt/libc-tests/tests/aligned_alloc.32 (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/aligned_alloc.64 (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_threads.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_threads.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_tss.32 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_tss.64 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/call_once.32 (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/call_once.64 (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/catopen (run as root)                 [00:00] [PASS]
Test: /opt/libc-tests/tests/endian.32 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/endian.64 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/env-7076.32 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/env-7076.64 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/fnmatch.32 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/fnmatch.64 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/fpround_test (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/i18n/bindtextdomain_test (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/memchr.32 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/memchr.64 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/memset_s.32 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/memset_s.64 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/newlocale_test (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/nl_langinfo_test (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/posix_memalign.32 (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/posix_memalign.64 (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-6961.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-9511.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-9511.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/priv_gettext (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/psignal (run as root)                 [00:00] [PASS]
Test: /opt/libc-tests/tests/pthread_attr_get_np (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/qsort/qsort_test (run as root)        [00:02] [PASS]
Test: /opt/libc-tests/tests/quick_exit (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4key.ksh (run as root)      [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_fork (run as root)  [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_forkall (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_forksig (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_prefork (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_preforkall (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_preforksig (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/chacha (run as root)           [00:00] [PASS]
Test: /opt/libc-tests/tests/random/getentropy (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/getrandom (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_child (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_inval (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_mlock (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_region (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_split (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_split_vpp (run as root)    [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_vpp (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/regex/regex_test (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/select/select.sh (run as root)        [00:06] [PASS]
Test: /opt/libc-tests/tests/set_constraint_handler_s.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/set_constraint_handler_s.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fileno.32 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fileno.64 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fmemopentest.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fmemopentest.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftell_ungetc.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftell_ungetc.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftello_12768.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftello_12768.lfs (run as root)  [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream.32 (run as root)      [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream.64 (run as root)      [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream_reopen.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream_reopen.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/open_memstreamtest.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/open_memstreamtest.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/orientation_test.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/orientation_test.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/test_mbrtowc.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/test_mbrtowc.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.32 (run as root) [00:01] [PASS]
Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.64 (run as root) [00:01] [PASS]
Test: /opt/libc-tests/tests/strerror (run as root)                [00:00] [PASS]
Test: /opt/libc-tests/tests/thread_name (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/timespec_get.32 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/timespec_get.64 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/uchar.32 (run as root)                [00:00] [PASS]
Test: /opt/libc-tests/tests/uchar.64 (run as root)                [00:00] [PASS]
Test: /opt/libc-tests/tests/utimes.32 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/utimes.64 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7344.32 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7344.64 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7350.32 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7350.64 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsrtombs_test (run as root)          [00:01] [PASS]
Test: /opt/libc-tests/tests/wctype_test (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/setup (run as root)           [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/assert_h (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/ctype_h (run as root)         [00:26] [PASS]
Test: /opt/libc-tests/tests/symbols/dirent_h (run as root)        [00:06] [PASS]
Test: /opt/libc-tests/tests/symbols/fcntl_h (run as root)         [00:16] [PASS]
Test: /opt/libc-tests/tests/symbols/locale_h (run as root)        [00:12] [PASS]
Test: /opt/libc-tests/tests/symbols/math_h (run as root)          [00:04] [PASS]
Test: /opt/libc-tests/tests/symbols/netdb_h (run as root)         [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/pthread_h (run as root)       [00:03] [PASS]
Test: /opt/libc-tests/tests/symbols/signal_h (run as root)        [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/stdalign_h (run as root)      [00:02] [PASS]
Test: /opt/libc-tests/tests/symbols/stddef_h (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/stdio_h (run as root)         [00:15] [PASS]
Test: /opt/libc-tests/tests/symbols/stdlib_h (run as root)        [00:14] [PASS]
Test: /opt/libc-tests/tests/symbols/stdnoreturn_h (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/string_h (run as root)        [00:02] [PASS]
Test: /opt/libc-tests/tests/symbols/strings_h (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_mman_h (run as root)      [00:26] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_stat_h (run as root)      [00:08] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_time_h (run as root)      [00:03] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_timeb_h (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/time_h (run as root)          [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/threads_h (run as root)       [00:27] [PASS]
Test: /opt/libc-tests/tests/symbols/ucontext_h (run as root)      [00:03] [PASS]
Test: /opt/libc-tests/tests/symbols/unistd_h (run as root)        [00:21] [PASS]
Test: /opt/libc-tests/tests/symbols/wchar_h (run as root)         [00:11] [PASS]
Test: /opt/libc-tests/tests/symbols/wctype_h (run as root)        [00:27] [PASS]

Results Summary
PASS     118

Running Time:   00:04:18
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20220118T174543

In parallel I'll be working with folks to get this reviewed and tested against larger package builds.

Actions #4

Updated by Robert Mustacchi 5 months ago

  • Has duplicate Bug #14504: madvise() not declared if _XOPEN_SOURCE defined added
Actions #5

Updated by Robert Mustacchi 3 months ago

In addition to the final versions of the symbol tests, with the help of Andy, Jonathan, and Rich I was able to verify:

  • That we didn't regress any of the OmniOS core and extra packages
  • That all of pkgsrc seemed fine in the end. The original pkgsrc bulk builds caught some of the issues around symbol visibility.
  • Rich went through and verified that this fixed issues with a recent LLVM build
Actions #6

Updated by Electric Monk 3 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit df5cd018c34371890eeeb8c930245b9323e8aa25

commit  df5cd018c34371890eeeb8c930245b9323e8aa25
Author: Robert Mustacchi <rm@fingolfin.org>
Date:   2022-04-20T17:46:45.000Z

    14418 mman.h symbol visibility is a mess
    Reviewed by: Gergő Mihály Doma <domag02@gmail.com>
    Reviewed by: Rich Lowe <richlowe@richlowe.net>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF