Bug #6782
head can't handle embedded nul characters
Start date:
2016-03-22
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
Description
as indicated on illumos@
'head -c 1 /dev/zero |wc -c' just hangs while 'sh -c 'head -c 1 /dev/zero|wc -c' correctly returns '1'
truss reveals a nasty endless loop:
$ truss -u libc head -c 1 /dev/zero ... /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1: read(3, "\0\0\0\0\0\0\0\0\0\0\0\0".., 8192) = 8192 /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 /1@1: -> libc:fgets(0x8047680, 0x400, 0x80622b0, 0x0) /1@1: <- libc:fgets() = 0x8047680 /1@1: -> libc:strlen() /1@1: <- libc:strlen() = 0 /1@1: -> libc:printf() /1@1: <- libc:printf() = 0 /1@1: -> libc:fflush(0x8062290, 0x8047680, 0x80622b0, 0x0) /1@1: <- libc:fflush() = 0 ...
Updated by Robert Mustacchi about 2 months ago
- Subject changed from /bin/head experiences issues when accessing /dev/zero to head can't handle embedded nul characters
- Category set to cmd - userland programs
- Assignee set to Robert Mustacchi
- % Done changed from 0 to 70
- Tags deleted (
needs-triage)
Taking a look at this, the problem is the way that head was written and somehow predates the work done in #3682. The crux of this is that it uses fgets(3C) and then calls strlen(3C) on the buffer. This breaks any time a line has an embedded nul in it. As an example, if you run the following:
$ printf "hello\0world\n" | ./head -n 1 hello$
Note, the trailing '$' is the start of the next prompt. We have no good way of realizing with fgets, strlen, and printf, that when we hit the first '\0' that we actually read substantially more. The fix to this, thankfully is simple. We just can do raw reads and go from there.
Updated by Robert Mustacchi about 1 month ago
To test this I wrote a new test suite that covered a bunch of different things we've seen with head(1) over the years and included a number of tests specific to this behavior. I ran the libc and util tests with this change, which included the new tests:
rm@beowulf:~$ pfexec /opt/util-tests/bin/utiltest Test: /opt/util-tests/tests/allowed-ips (run as root) [00:01] [PASS] Test: /opt/util-tests/tests/chown_test (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/date_test (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/find/findtest (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/grep_test (run as root) [00:05] [PASS] Test: /opt/util-tests/tests/head/head_test (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libjedec_test (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libsff/libsff (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/make_test (run as root) [00:01] [PASS] Test: /opt/util-tests/tests/mdb/mdbtest (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/mergeq/mqt (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/mergeq/wqt (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/printf_test (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/set-linkprop (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/sleep/sleeptest (run as root) [00:30] [PASS] Test: /opt/util-tests/tests/smbios (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/xargs_test (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/awk/runtests.sh (run as nobody) [02:59] [PASS] Test: /opt/util-tests/tests/ctf/precheck (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/ctf/ctftest (run as root) [00:06] [PASS] Test: /opt/util-tests/tests/demangle/afl-fast (run as root) [00:01] [PASS] Test: /opt/util-tests/tests/demangle/gcc-libstdc++ (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/demangle/llvm-stdcxxabi (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libcustr/custr_remove (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libcustr/custr_trunc (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_00_blank (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_01_boolean (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_02_numbers (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_03_empty_arrays (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_04_number_arrays (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_05_strings (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_06_nested (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/libnvpair_json/json_07_nested_arrays (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/sed/sed_addr (run as root) [00:00] [PASS] Test: /opt/util-tests/tests/sed/multi_test (run as root) [00:00] [PASS] Results Summary PASS 35 Running Time: 00:03:51 Percent passed: 100.0% rm@beowulf:~$ pfexec /opt/libc-tests/bin/libctest Test: /opt/libc-tests/tests/aligned_alloc.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/aligned_alloc.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/c11_threads.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/c11_threads.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/c11_tss.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/c11_tss.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/call_once.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/call_once.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/catopen (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/endian.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/endian.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/env-7076.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/env-7076.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/fnmatch.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/fnmatch.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/fpround_test (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/i18n/bindtextdomain_test (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/memset_s.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/memset_s.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/newlocale_test (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/nl_langinfo_test (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/posix_memalign.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/posix_memalign.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/printf-6961.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/printf-9511.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/printf-9511.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/priv_gettext (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/psignal (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/pthread_attr_get_np (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/quick_exit (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/arc4key.ksh (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/random/arc4random (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/arc4random_fork (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/arc4random_forkall (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/arc4random_forksig (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/arc4random_prefork (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/arc4random_preforkall (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/arc4random_preforksig (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/chacha (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/getentropy (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/getrandom (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/inz_child (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/inz_inval (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/inz_mlock (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/inz_region (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/inz_split (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/inz_split_vpp (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/random/inz_vpp (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/regex/regex_test (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/select/select.sh (run as root) [00:08] [PASS] Test: /opt/libc-tests/tests/set_constraint_handler_s.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/set_constraint_handler_s.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/fileno.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/fileno.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/fmemopentest.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/fmemopentest.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/ftell_ungetc.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/ftell_ungetc.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/ftello_12768.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/ftello_12768.lfs (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/memstream.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/memstream.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/memstream_reopen.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/memstream_reopen.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/open_memstreamtest.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/open_memstreamtest.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/orientation_test.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/orientation_test.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/test_mbrtowc.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/stdio/test_mbrtowc.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.32 (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/strcoll-strxfrm-6907.64 (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/strerror (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/thread_name (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/timespec_get.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/timespec_get.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/uchar.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/uchar.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/utimes.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/utimes.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/wcsncasecmp-7344.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/wcsncasecmp-7344.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/wcsncasecmp-7350.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/wcsncasecmp-7350.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/wcsncasecmp.32 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/wcsncasecmp.64 (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/wcsrtombs_test (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/wctype_test (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/symbols/setup (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/symbols/assert_h (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/symbols/ctype_h (run as root) [00:23] [PASS] Test: /opt/libc-tests/tests/symbols/dirent_h (run as root) [00:06] [PASS] Test: /opt/libc-tests/tests/symbols/fcntl_h (run as root) [00:15] [PASS] Test: /opt/libc-tests/tests/symbols/locale_h (run as root) [00:11] [PASS] Test: /opt/libc-tests/tests/symbols/math_h (run as root) [00:04] [PASS] Test: /opt/libc-tests/tests/symbols/netdb_h (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/symbols/pthread_h (run as root) [00:03] [PASS] Test: /opt/libc-tests/tests/symbols/signal_h (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/symbols/stdalign_h (run as root) [00:02] [PASS] Test: /opt/libc-tests/tests/symbols/stddef_h (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/symbols/stdio_h (run as root) [00:14] [PASS] Test: /opt/libc-tests/tests/symbols/stdlib_h (run as root) [00:12] [PASS] Test: /opt/libc-tests/tests/symbols/stdnoreturn_h (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/symbols/string_h (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/symbols/strings_h (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/symbols/sys_stat_h (run as root) [00:07] [PASS] Test: /opt/libc-tests/tests/symbols/sys_time_h (run as root) [00:03] [PASS] Test: /opt/libc-tests/tests/symbols/sys_timeb_h (run as root) [00:00] [PASS] Test: /opt/libc-tests/tests/symbols/time_h (run as root) [00:01] [PASS] Test: /opt/libc-tests/tests/symbols/threads_h (run as root) [00:23] [PASS] Test: /opt/libc-tests/tests/symbols/ucontext_h (run as root) [00:03] [PASS] Test: /opt/libc-tests/tests/symbols/unistd_h (run as root) [00:20] [PASS] Test: /opt/libc-tests/tests/symbols/wchar_h (run as root) [00:09] [PASS] Test: /opt/libc-tests/tests/symbols/wctype_h (run as root) [00:24] [PASS] Results Summary PASS 114 Running Time: 00:03:30 Percent passed: 100.0% Log directory: /var/tmp/test_results/20200921T115123
I also ran a wsdiff on bits built with the new head present and bits without it. The wsdiff had the following files which are mostly dof or related to having dates still embedded plus the hlista/b ones which folks have started to see more often but we haven't root caused:
usr/sbin/isns usr/sbin/iasl usr/sbin/acpixtract usr/sbin/acpidump usr/bin/dns-sd usr/lib/libc/libc_hwcap1.so.1 usr/lib/libc/libc_hwcap3.so.1 usr/lib/libc/libc_hwcap2.so.1 usr/lib/nfs/mountd usr/lib/amd64/libzpool.so.1 usr/lib/smbsrv/amd64/libfksmbsrv.so.1 usr/lib/smbsrv/libfksmbsrv.so.1 usr/lib/spell/hlista usr/lib/spell/hlistb usr/lib/libzpool.so.1 etc/motd lib/amd64/libc.so.1 lib/libc.so.1 opt/SUNWdtrt/tst/i386/ustack/tst.helper.exe opt/SUNWdtrt/tst/i386/ustack/tst.annotated.exe
Updated by Electric Monk about 1 month ago
- Status changed from New to Closed
- % Done changed from 70 to 100
git commit a9e414682948591ec63d5ab2cd11ba55603b59fa
commit a9e414682948591ec63d5ab2cd11ba55603b59fa
Author: Robert Mustacchi <rm@fingolfin.org>
Date: 2020-09-23T04:28:51.000Z
6782 head can't handle embedded nul characters
13150 head -v doesn't work with a single file
Reviewed by: Andy Fiddaman <andy@omniosce.org>
Approved by: Richard Lowe <richlowe@richlowe.net>
Updated by Electric Monk about 1 month ago
git commit d996a5676844ee9f31d0caa59de3cfaf48e07e77
commit d996a5676844ee9f31d0caa59de3cfaf48e07e77
Author: Robert Mustacchi <rm@fingolfin.org>
Date: 2020-09-23T14:51:03.000Z
6782 head can't handle embedded nul characters (add missing file)
Approved by: Dan McDonald <danmcd@joyent.com>