Project

General

Profile

Feature #7092

Want support for stdio memory streams

Added by Adam Števko over 4 years ago. Updated 8 months ago.

Status:
Closed
Priority:
Normal
Category:
lib - userland libraries
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

When trying to compile goacess-1.0 on illumos I got following error:

Undefined                       first referenced
 symbol                             in file
open_memstream                      src/json.o

Code: https://github.com/allinurl/goaccess/blob/master/src/json.c#L1036

open_memstream is not supported on illumos and it should be added. This will help with compatibility.

Interesting URLs:
- http://src.illumos.org/source/xref/freebsd-head/lib/libc/stdio/open_memstream.c


Related issues

Related to illumos gate - Bug #12357: getc/putc_unlocked need to set orientationClosedRobert Mustacchi

Actions
Related to illumos gate - Feature #12358: Need mbrtowc variant that indicates consumed zero bytesClosedRobert Mustacchi

Actions
Related to illumos gate - Bug #12392: ftello64 doesn't handle ungetc() correctly when unbufferedClosedRobert Mustacchi

Actions
Related to illumos gate - Feature #12359: Want a means to set the umem mtbf at runtineClosedRobert Mustacchi

Actions
Has duplicate illumos gate - Feature #10578: implement fmemopen(3C)DuplicateRobert Mustacchi2019-03-21

Actions
Blocks illumos gate - Bug #1609: Umbrella bug for POSIX.1-2008 (SUSv4) system/library callsNew2011-10-07

Actions
#1

Updated by Jorge Schrauwen about 2 years ago

Looks like Samba 4.9.0 also wants it now, I get link errors on this. I opened https://bugzilla.samba.org/show_bug.cgi?id=13629 for this.

#2

Updated by Robert Mustacchi 10 months ago

  • Assignee set to Robert Mustacchi
  • Start date deleted (2016-06-09)
#3

Updated by Robert Mustacchi 10 months ago

  • Category set to lib - userland libraries
  • Tags deleted (needs-triage)
#4

Updated by Robert Mustacchi 9 months ago

  • Subject changed from Add open_memstream and friends to Want support for stdio memory streams
  • % Done changed from 0 to 90
#5

Updated by Robert Mustacchi 9 months ago

#6

Updated by Robert Mustacchi 8 months ago

  • Related to Bug #12357: getc/putc_unlocked need to set orientation added
#7

Updated by Robert Mustacchi 8 months ago

  • Related to Feature #12358: Need mbrtowc variant that indicates consumed zero bytes added
#8

Updated by Robert Mustacchi 8 months ago

  • Related to Bug #12392: ftello64 doesn't handle ungetc() correctly when unbuffered added
#9

Updated by Robert Mustacchi 8 months ago

  • Related to Feature #12359: Want a means to set the umem mtbf at runtine added
#10

Updated by Robert Mustacchi 8 months ago

To test the new memory streams I did a combination of a couple of different things:

  • I imported several tests from OpenBSD to cover some basic behavior of open_memstream(3C) and fmemopen(3C). I also used several tests that they had put together for other parts of stdio.
  • I wrote out my own unit tests for open_memsrteam(3C), open_wmemstream(3C), and fmemopen(3C) that covered more of the corner case behaviors and other conditions like the failure to allocate memory.
  • I made sure that all of the tests employ umem debugging to catch for memory corruption issues.
  • I updated the symbols tests to cover the new functions.
  • I exercised the resulting system using a couple of different languages and tools to make sure that stdio wasn't totally broken.
  • Compared some of the behaviors that we have to glibc, musl, and the BSDs
  • I also run the following other test suites (mostly in case they exercised libc/stdio):
    • elf-tests
    • parts of crypto-tests (some tests time out because my laptop CPU isn't fast enough for the current time out)
    • os-tests
    • util-tests (spurious failures from lacking ctf tools, etc.)

Here's a run of the full libc test suite:

rm@nienor:/opt/libc-tests$ pfexec ./bin/libctest 
Test: /opt/libc-tests/tests/aligned_alloc.32 (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/aligned_alloc.64 (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_threads.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_threads.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_tss.32 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_tss.64 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/call_once.32 (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/call_once.64 (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/catopen (run as root)                 [00:00] [PASS]
Test: /opt/libc-tests/tests/endian.32 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/endian.64 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/env-7076.32 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/env-7076.64 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/fnmatch.32 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/fnmatch.64 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/fpround_test (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/memset_s.32 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/memset_s.64 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/newlocale_test (run as root)          [00:01] [PASS]
Test: /opt/libc-tests/tests/nl_langinfo_test (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/posix_memalign.32 (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/posix_memalign.64 (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-6961.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-9511.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-9511.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/priv_gettext (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/psignal (run as root)                 [00:00] [PASS]
Test: /opt/libc-tests/tests/pthread_attr_get_np (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/quick_exit (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4key.ksh (run as root)      [00:03] [PASS]
Test: /opt/libc-tests/tests/random/arc4random (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_fork (run as root)  [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_forkall (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_forksig (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_prefork (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_preforkall (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_preforksig (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/chacha (run as root)           [00:00] [PASS]
Test: /opt/libc-tests/tests/random/getentropy (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/getrandom (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_child (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_inval (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_mlock (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_region (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_split (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_split_vpp (run as root)    [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_vpp (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/regex/regex_test (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/select/select.sh (run as root)        [00:08] [PASS]
Test: /opt/libc-tests/tests/set_constraint_handler_s.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/set_constraint_handler_s.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fileno.32 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fileno.64 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fmemopentest.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fmemopentest.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftell_ungetc.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftell_ungetc.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream.32 (run as root)      [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream.64 (run as root)      [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream_reopen.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream_reopen.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/open_memstreamtest.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/open_memstreamtest.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/orientation_test.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/orientation_test.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/test_mbrtowc.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/test_mbrtowc.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.32 (run as root) [00:01] [PASS]
Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.64 (run as root) [00:01] [PASS]
Test: /opt/libc-tests/tests/strerror (run as root)                [00:00] [PASS]
Test: /opt/libc-tests/tests/thread_name (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/timespec_get.32 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/timespec_get.64 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7344.32 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7344.64 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7350.32 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7350.64 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsrtombs_test (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/wctype_test (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/setup (run as root)           [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/assert_h (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/ctype_h (run as root)         [00:24] [PASS]
Test: /opt/libc-tests/tests/symbols/dirent_h (run as root)        [00:05] [PASS]
Test: /opt/libc-tests/tests/symbols/fcntl_h (run as root)         [00:15] [PASS]
Test: /opt/libc-tests/tests/symbols/locale_h (run as root)        [00:11] [PASS]
Test: /opt/libc-tests/tests/symbols/math_h (run as root)          [00:04] [PASS]
Test: /opt/libc-tests/tests/symbols/netdb_h (run as root)         [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/pthread_h (run as root)       [00:02] [PASS]
Test: /opt/libc-tests/tests/symbols/signal_h (run as root)        [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/stdalign_h (run as root)      [00:02] [PASS]
Test: /opt/libc-tests/tests/symbols/stddef_h (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/stdio_h (run as root)         [00:14] [PASS]
Test: /opt/libc-tests/tests/symbols/stdlib_h (run as root)        [00:13] [PASS]
Test: /opt/libc-tests/tests/symbols/stdnoreturn_h (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/string_h (run as root)        [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/strings_h (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_stat_h (run as root)      [00:07] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_time_h (run as root)      [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_timeb_h (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/time_h (run as root)          [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/threads_h (run as root)       [00:25] [PASS]
Test: /opt/libc-tests/tests/symbols/ucontext_h (run as root)      [00:02] [PASS]
Test: /opt/libc-tests/tests/symbols/unistd_h (run as root)        [00:19] [PASS]
Test: /opt/libc-tests/tests/symbols/wchar_h (run as root)         [00:10] [PASS]
Test: /opt/libc-tests/tests/symbols/wctype_h (run as root)        [00:24] [PASS]

Results Summary
PASS     107

Running Time:   00:03:31
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20200326T085624

And here's what a run of the individual memstream tests that I wrote looks like:

rm@nienor:/opt/libc-tests$ ./tests/stdio/memstream.32 
TEST PASSED: fmemopen: bad mode argument
TEST PASSED: fmemopen: bad buffer size, valid buf
TEST PASSED: fmemopen: bad buffer size, NULL buf
TEST PASSED: fmemopen: invalid NULL buf, mode: r
TEST PASSED: fmemopen: invalid NULL buf, mode: w
TEST PASSED: fmemopen: invalid NULL buf, mode: a
TEST PASSED: fmemopen: invalid NULL buf, mode: ax
TEST PASSED: fmemopen: bad open ask for SIZE_MAX bytes
TEST PASSED: fmemopen: simulate malloc failure at open
TEST PASSED: open_memstream: bad buf
TEST PASSED: open_memstream: bad size
TEST PASSED: open_memstream: bad buf and size
TEST PASSED: open_memstream: simulate malloc failure at open
TEST PASSED: open_wmemstream: bad buf
TEST PASSED: open_wmemstream: bad size
TEST PASSED: open_wmemstream: bad buf and size
TEST PASSED: open_wmemstream: simulate malloc failure at open
TEST PASSED: fmemopen: write beyond end of buffer: putc (buf smaller than BUFSIZ)
TEST PASSED: fmemopen: write beyond end of buffer: putc (line buffering)
TEST PASSED: fmemopen: write beyond end of buffer: putc (no stdio buffering)
TEST PASSED: fmemopen: write beyond end of buffer: fwrite (buf smaller than BUFSIZ)
TEST PASSED: fmemopen: write beyond end of buffer: fwrite (line buffering)
TEST PASSED: fmemopen: write beyond end of buffer: fwrite (no stdio buffering)
TEST PASSED: fmemopen: write beyond end of buffer: fwrite 2 (buf smaller than BUFSIZ)
TEST PASSED: fmemopen: write beyond end of buffer: fwrite 2 (line buffering)
TEST PASSED: fmemopen: write beyond end of buffer: fwrite 2 (no stdio buffering)
TEST PASSED: fmemopen: write beyond end of buffer: fputs (buf smaller than BUFSIZ)
TEST PASSED: fmemopen: write beyond end of buffer: fputs (line buffering)
TEST PASSED: fmemopen: write beyond end of buffer: fputs (no stdio buffering)
TEST PASSED: fmemopen: default position and log. size, mode: r
TEST PASSED: fmemopen: default position and log. size, mode: r+
TEST PASSED: fmemopen: default position and log. size, mode: w
TEST PASSED: fmemopen: default position and log. size, mode: w+
TEST PASSED: fmemopen: default position and log. size, mode: a
TEST PASSED: fmemopen: default position and log. size, mode: a+
TEST PASSED: fmemopen: default position and log. size, mode: a, nul byte
TEST PASSED: fmemopen: default position and log. size, mode: a+, nul byte
TEST PASSED: fmemopen: default position and log. size, mode: a+, NULL buf
TEST PASSED: fmemopen: read until EOF with fgetc
TEST PASSED: fmemopen: read until EOF with fgets
TEST PASSED: fmemopen: read until EOF with fread
TEST PASSED: fmemopen: read until EOF with fread 2
TEST PASSED: fmemopen: invalid seeks
TEST PASSED: fmemopen: w+ mode truncates buffer
TEST PASSED: fmemopen: NULs properly inserted (w)
TEST PASSED: fmemopen: NULs properly inserted (a)
TEST PASSED: fmemopen: read NUL character normally
TEST PASSED: open_memstream: default position and logical size
TEST PASSED: wopen_memstream: default position and logical size
TEST PASSED: open_memstream: read doesn't work
TEST PASSED: open_wmemstream: read doesn't work
TEST PASSED: open_memstream: flush failure due to induced memory failure
TEST PASSED: open_wmemstream: flush failure due to induced memory failure
TEST PASSED: open_[w]memstream: bad seeks
TEST PASSED: open_memstream: appends NULs
TEST PASSED: open_wmemstream: appends NULs
TEST PASSED: open_wmemstream: handles embedded NULs
TEST PASSED: open_wmemstream: write wide chars
TEST PASSED: open_wmemstream: seeking doesn't grow
TEST PASSED: open_wmemstream: Write mb sequences
TEST PASSED: open_wmemstream: detect bad utf-8 sequence
TEST PASSED: open_wmemstream: detect bad utf-8 sequence 2 (fflush)
TEST PASSED: open_wmemstream: ftell buffering behavior
63/63 tests passed
#11

Updated by Electric Monk 8 months ago

  • Status changed from New to Closed
  • % Done changed from 90 to 100

git commit cd62a92d4a964bfe61d35ba2301b69e65e22a509

commit  cd62a92d4a964bfe61d35ba2301b69e65e22a509
Author: Robert Mustacchi <rm@fingolfin.org>
Date:   2020-03-26T07:42:53.000Z

    7092 Want support for stdio memory streams
    12360 fwrite can loop forever on zero byte write
    12392 ftello64 doesn't handle ungetc() correctly when unbuffered
    Reviewed by: John Levon <john.levon@joyent.com>
    Reviewed by: Yuri Pankov <ypankov@fastmail.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

#12

Updated by Yuri Pankov 5 months ago

  • Blocks Bug #1609: Umbrella bug for POSIX.1-2008 (SUSv4) system/library calls added

Also available in: Atom PDF