Project

General

Profile

Bug #6782

head can't handle embedded nul characters

Added by Richard PALO over 4 years ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Category:
cmd - userland programs
Start date:
2016-03-22
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

as indicated on illumos@
'head -c 1 /dev/zero |wc -c' just hangs while 'sh -c 'head -c 1 /dev/zero|wc -c' correctly returns '1'

truss reveals a nasty endless loop:

$ truss  -u libc head -c 1 /dev/zero
 ...
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1:    read(3, "\0\0\0\0\0\0\0\0\0\0\0\0".., 8192)    = 8192
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 /1@1:    -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0)
 /1@1:    <- libc:fgets() = 0x8047680
 /1@1:    -> libc:strlen()
 /1@1:    <- libc:strlen() = 0
 /1@1:    -> libc:printf()
 /1@1:    <- libc:printf() = 0
 /1@1:    -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0)
 /1@1:    <- libc:fflush() = 0
 ...

#1

Updated by Robert Mustacchi about 2 months ago

  • Subject changed from /bin/head experiences issues when accessing /dev/zero to head can't handle embedded nul characters
  • Category set to cmd - userland programs
  • Assignee set to Robert Mustacchi
  • % Done changed from 0 to 70
  • Tags deleted (needs-triage)

Taking a look at this, the problem is the way that head was written and somehow predates the work done in #3682. The crux of this is that it uses fgets(3C) and then calls strlen(3C) on the buffer. This breaks any time a line has an embedded nul in it. As an example, if you run the following:

$ printf "hello\0world\n" | ./head -n 1
hello$

Note, the trailing '$' is the start of the next prompt. We have no good way of realizing with fgets, strlen, and printf, that when we hit the first '\0' that we actually read substantially more. The fix to this, thankfully is simple. We just can do raw reads and go from there.

#2

Updated by Electric Monk about 2 months ago

  • Gerrit CR set to 922
#3

Updated by Robert Mustacchi about 1 month ago

To test this I wrote a new test suite that covered a bunch of different things we've seen with head(1) over the years and included a number of tests specific to this behavior. I ran the libc and util tests with this change, which included the new tests:

rm@beowulf:~$ pfexec /opt/util-tests/bin/utiltest 
Test: /opt/util-tests/tests/allowed-ips (run as root)             [00:01] [PASS]
Test: /opt/util-tests/tests/chown_test (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/date_test (run as root)               [00:00] [PASS]
Test: /opt/util-tests/tests/find/findtest (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/grep_test (run as root)               [00:05] [PASS]
Test: /opt/util-tests/tests/head/head_test (run as root)          [00:00] [PASS]
Test: /opt/util-tests/tests/libjedec_test (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/libsff/libsff (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/make_test (run as root)               [00:01] [PASS]
Test: /opt/util-tests/tests/mdb/mdbtest (run as root)             [00:00] [PASS]
Test: /opt/util-tests/tests/mergeq/mqt (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/mergeq/wqt (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/printf_test (run as root)             [00:00] [PASS]
Test: /opt/util-tests/tests/set-linkprop (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/sleep/sleeptest (run as root)         [00:30] [PASS]
Test: /opt/util-tests/tests/smbios (run as root)                  [00:00] [PASS]
Test: /opt/util-tests/tests/xargs_test (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/awk/runtests.sh (run as nobody)       [02:59] [PASS]
Test: /opt/util-tests/tests/ctf/precheck (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/ctf/ctftest (run as root)             [00:06] [PASS]
Test: /opt/util-tests/tests/demangle/afl-fast (run as root)       [00:01] [PASS]
Test: /opt/util-tests/tests/demangle/gcc-libstdc++ (run as root)  [00:00] [PASS]
Test: /opt/util-tests/tests/demangle/llvm-stdcxxabi (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libcustr/custr_remove (run as root)   [00:00] [PASS]
Test: /opt/util-tests/tests/libcustr/custr_trunc (run as root)    [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_00_blank (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_01_boolean (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_02_numbers (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_03_empty_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_04_number_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_05_strings (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_06_nested (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_07_nested_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/sed/sed_addr (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/sed/multi_test (run as root)          [00:00] [PASS]

Results Summary
PASS      35

Running Time:   00:03:51
Percent passed: 100.0%

rm@beowulf:~$ pfexec /opt/libc-tests/bin/libctest 
Test: /opt/libc-tests/tests/aligned_alloc.32 (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/aligned_alloc.64 (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_threads.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_threads.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_tss.32 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/c11_tss.64 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/call_once.32 (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/call_once.64 (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/catopen (run as root)                 [00:00] [PASS]
Test: /opt/libc-tests/tests/endian.32 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/endian.64 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/env-7076.32 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/env-7076.64 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/fnmatch.32 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/fnmatch.64 (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/fpround_test (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/i18n/bindtextdomain_test (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/memset_s.32 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/memset_s.64 (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/newlocale_test (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/nl_langinfo_test (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/posix_memalign.32 (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/posix_memalign.64 (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-6961.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-9511.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/printf-9511.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/priv_gettext (run as root)            [00:00] [PASS]
Test: /opt/libc-tests/tests/psignal (run as root)                 [00:00] [PASS]
Test: /opt/libc-tests/tests/pthread_attr_get_np (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/quick_exit (run as root)              [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4key.ksh (run as root)      [00:01] [PASS]
Test: /opt/libc-tests/tests/random/arc4random (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_fork (run as root)  [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_forkall (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_forksig (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_prefork (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_preforkall (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/arc4random_preforksig (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/random/chacha (run as root)           [00:00] [PASS]
Test: /opt/libc-tests/tests/random/getentropy (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/getrandom (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_child (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_inval (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_mlock (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_region (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_split (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_split_vpp (run as root)    [00:00] [PASS]
Test: /opt/libc-tests/tests/random/inz_vpp (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/regex/regex_test (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/select/select.sh (run as root)        [00:08] [PASS]
Test: /opt/libc-tests/tests/set_constraint_handler_s.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/set_constraint_handler_s.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fileno.32 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fileno.64 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fmemopentest.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/fmemopentest.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftell_ungetc.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftell_ungetc.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftello_12768.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/ftello_12768.lfs (run as root)  [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream.32 (run as root)      [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream.64 (run as root)      [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream_reopen.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/memstream_reopen.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/open_memstreamtest.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/open_memstreamtest.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/orientation_test.32 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/orientation_test.64 (run as root) [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/test_mbrtowc.32 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/stdio/test_mbrtowc.64 (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.32 (run as root) [00:01] [PASS]
Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.64 (run as root) [00:01] [PASS]
Test: /opt/libc-tests/tests/strerror (run as root)                [00:00] [PASS]
Test: /opt/libc-tests/tests/thread_name (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/timespec_get.32 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/timespec_get.64 (run as root)         [00:00] [PASS]
Test: /opt/libc-tests/tests/uchar.32 (run as root)                [00:00] [PASS]
Test: /opt/libc-tests/tests/uchar.64 (run as root)                [00:00] [PASS]
Test: /opt/libc-tests/tests/utimes.32 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/utimes.64 (run as root)               [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7344.32 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7344.64 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7350.32 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp-7350.64 (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp.32 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsncasecmp.64 (run as root)          [00:00] [PASS]
Test: /opt/libc-tests/tests/wcsrtombs_test (run as root)          [00:01] [PASS]
Test: /opt/libc-tests/tests/wctype_test (run as root)             [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/setup (run as root)           [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/assert_h (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/ctype_h (run as root)         [00:23] [PASS]
Test: /opt/libc-tests/tests/symbols/dirent_h (run as root)        [00:06] [PASS]
Test: /opt/libc-tests/tests/symbols/fcntl_h (run as root)         [00:15] [PASS]
Test: /opt/libc-tests/tests/symbols/locale_h (run as root)        [00:11] [PASS]
Test: /opt/libc-tests/tests/symbols/math_h (run as root)          [00:04] [PASS]
Test: /opt/libc-tests/tests/symbols/netdb_h (run as root)         [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/pthread_h (run as root)       [00:03] [PASS]
Test: /opt/libc-tests/tests/symbols/signal_h (run as root)        [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/stdalign_h (run as root)      [00:02] [PASS]
Test: /opt/libc-tests/tests/symbols/stddef_h (run as root)        [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/stdio_h (run as root)         [00:14] [PASS]
Test: /opt/libc-tests/tests/symbols/stdlib_h (run as root)        [00:12] [PASS]
Test: /opt/libc-tests/tests/symbols/stdnoreturn_h (run as root)   [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/string_h (run as root)        [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/strings_h (run as root)       [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_stat_h (run as root)      [00:07] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_time_h (run as root)      [00:03] [PASS]
Test: /opt/libc-tests/tests/symbols/sys_timeb_h (run as root)     [00:00] [PASS]
Test: /opt/libc-tests/tests/symbols/time_h (run as root)          [00:01] [PASS]
Test: /opt/libc-tests/tests/symbols/threads_h (run as root)       [00:23] [PASS]
Test: /opt/libc-tests/tests/symbols/ucontext_h (run as root)      [00:03] [PASS]
Test: /opt/libc-tests/tests/symbols/unistd_h (run as root)        [00:20] [PASS]
Test: /opt/libc-tests/tests/symbols/wchar_h (run as root)         [00:09] [PASS]
Test: /opt/libc-tests/tests/symbols/wctype_h (run as root)        [00:24] [PASS]

Results Summary
PASS     114

Running Time:   00:03:30
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20200921T115123

I also ran a wsdiff on bits built with the new head present and bits without it. The wsdiff had the following files which are mostly dof or related to having dates still embedded plus the hlista/b ones which folks have started to see more often but we haven't root caused:

usr/sbin/isns
usr/sbin/iasl
usr/sbin/acpixtract
usr/sbin/acpidump
usr/bin/dns-sd
usr/lib/libc/libc_hwcap1.so.1
usr/lib/libc/libc_hwcap3.so.1
usr/lib/libc/libc_hwcap2.so.1
usr/lib/nfs/mountd
usr/lib/amd64/libzpool.so.1
usr/lib/smbsrv/amd64/libfksmbsrv.so.1
usr/lib/smbsrv/libfksmbsrv.so.1
usr/lib/spell/hlista
usr/lib/spell/hlistb
usr/lib/libzpool.so.1
etc/motd
lib/amd64/libc.so.1
lib/libc.so.1
opt/SUNWdtrt/tst/i386/ustack/tst.helper.exe
opt/SUNWdtrt/tst/i386/ustack/tst.annotated.exe
#4

Updated by Electric Monk about 1 month ago

  • Status changed from New to Closed
  • % Done changed from 70 to 100

git commit a9e414682948591ec63d5ab2cd11ba55603b59fa

commit  a9e414682948591ec63d5ab2cd11ba55603b59fa
Author: Robert Mustacchi <rm@fingolfin.org>
Date:   2020-09-23T04:28:51.000Z

    6782 head can't handle embedded nul characters
    13150 head -v doesn't work with a single file
    Reviewed by: Andy Fiddaman <andy@omniosce.org>
    Approved by: Richard Lowe <richlowe@richlowe.net>

#5

Updated by Electric Monk about 1 month ago

git commit d996a5676844ee9f31d0caa59de3cfaf48e07e77

commit  d996a5676844ee9f31d0caa59de3cfaf48e07e77
Author: Robert Mustacchi <rm@fingolfin.org>
Date:   2020-09-23T14:51:03.000Z

    6782 head can't handle embedded nul characters (add missing file)
    Approved by: Dan McDonald <danmcd@joyent.com>

Also available in: Atom PDF