Project

General

Profile

Bug #11552

Want a more modern nawk(1)

Added by Cody Mello 6 months ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Cody Mello
Category:
cmd - userland programs
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

The current illumos nawk is effectively a fork from the state of Brian Kernighan's nawk in 1989. Sun then restyled the code, and made the occasional independent fix, but they never pulled in upstream changes again. Since the two source bases had diverged significantly, I tried to recreate the history as best as I could, and used that to inform several updates for SmartOS: a mostly non-functional change, a functional change, and one notable breaking change. I then followed up with two other changes (a fix to field splitting, and support for the RT variable). This update for illumos contains all of those changes.

Note that this change only catches nawk(1) up to about late 2018, since I haven't had a chance to evaluate other recent changes yet.

History

#1

Updated by Cody Mello 5 months ago

To test these changes, I made sure that all of the updates pulled in had appropriate tests in the test suite. I also verified that (with the changes in #11551) the gate will be able to build with the new binary. I rebooted OmniOS onto the build, and it came up fine (aside from some dbus issues due to illumos upstream libc not having the inotify_* functions that are in SmartOS and OmniOS).

Additionally, SmartOS has been running with an updated nawk since release-20190620, and we haven't yet seen or received reports of any issues.

Running the updated test suite against an updated nawk binary:

cpm@utu ~ % cd ws/illumos-gate/proto/root_i386/opt/util-tests/tests/awk 
cpm@utu ...util-tests/tests/awk : master % ./runtests.sh -o ~/ws/illumos-gate/proto/root_i386/usr/bin/nawk 
Running AWK tests ($AWK="/home/cpm/ws/illumos-gate/proto/root_i386/usr/bin/nawk")

# Examples from "The AWK Programming Environment" 

examples/awk/p.1... ok
examples/awk/p.10... ok
examples/awk/p.11... ok
examples/awk/p.12... ok
examples/awk/p.13... ok
examples/awk/p.14... ok
examples/awk/p.15... ok
examples/awk/p.16... ok
examples/awk/p.17... ok
examples/awk/p.18... ok
examples/awk/p.19... ok
examples/awk/p.2... ok
examples/awk/p.20... ok
examples/awk/p.21... ok
examples/awk/p.21a... ok
examples/awk/p.22... ok
examples/awk/p.23... ok
examples/awk/p.24... ok
examples/awk/p.25... ok
examples/awk/p.26... ok
examples/awk/p.26a... ok
examples/awk/p.27... ok
examples/awk/p.28... ok
examples/awk/p.29... ok
examples/awk/p.3... ok
examples/awk/p.30... ok
examples/awk/p.31... ok
examples/awk/p.32... ok
examples/awk/p.33... ok
examples/awk/p.34... ok
examples/awk/p.35... ok
examples/awk/p.36... ok
examples/awk/p.37... ok
examples/awk/p.38... ok
examples/awk/p.39... ok
examples/awk/p.4... ok
examples/awk/p.40... ok
examples/awk/p.41... ok
examples/awk/p.42... ok
examples/awk/p.43... ok
examples/awk/p.44... ok
examples/awk/p.45... ok
examples/awk/p.46... ok
examples/awk/p.47... ok
examples/awk/p.48... ok
examples/awk/p.48a... ok
examples/awk/p.48b... ok
examples/awk/p.49... ok
examples/awk/p.5... ok
examples/awk/p.50... ok
examples/awk/p.51... ok
examples/awk/p.52... ok
examples/awk/p.5a... ok
examples/awk/p.6... ok
examples/awk/p.7... ok
examples/awk/p.8... ok
examples/awk/p.9... ok
examples/awk/p.table... ok

# One True AWK Example Programs

examples/awk/t.0... ok
examples/awk/t.0a... ok
examples/awk/t.1... ok
examples/awk/t.1.x... ok
examples/awk/t.2... ok
examples/awk/t.2.x... ok
examples/awk/t.3... ok
examples/awk/t.3.x... ok
examples/awk/t.4... ok
examples/awk/t.4.x... ok
examples/awk/t.5.x... ok
examples/awk/t.6... ok
examples/awk/t.6.x... ok
examples/awk/t.6a... ok
examples/awk/t.6b... ok
examples/awk/t.8.x... ok
examples/awk/t.8.y... ok
examples/awk/t.addops... ok
examples/awk/t.aeiou... ok
examples/awk/t.aeiouy... ok
examples/awk/t.arith... ok
examples/awk/t.array... ok
examples/awk/t.array1... ok
examples/awk/t.array2... ok
examples/awk/t.assert... ok
examples/awk/t.avg... ok
examples/awk/t.b.x... ok
examples/awk/t.be... ok
examples/awk/t.beginexit... ok
examples/awk/t.beginnext... ok
examples/awk/t.break... ok
examples/awk/t.break1... ok
examples/awk/t.break2... ok
examples/awk/t.break3... ok
examples/awk/t.bug1... ok
examples/awk/t.builtins... ok
examples/awk/t.cat... ok
examples/awk/t.cat1... ok
examples/awk/t.cat2... ok
examples/awk/t.cmp... ok
examples/awk/t.coerce... ok
examples/awk/t.coerce2... ok
examples/awk/t.comment... ok
examples/awk/t.comment1... ok
examples/awk/t.concat... ok
examples/awk/t.cond... ok
examples/awk/t.contin... ok
examples/awk/t.count... ok
examples/awk/t.crlf... ok
examples/awk/t.cum... ok
examples/awk/t.d.x... ok
examples/awk/t.delete0... ok
examples/awk/t.delete1... ok
examples/awk/t.delete2... ok
examples/awk/t.delete3... ok
examples/awk/t.do... ok
examples/awk/t.e... ok
examples/awk/t.else... ok
examples/awk/t.exit... ok
examples/awk/t.exit1... ok
examples/awk/t.f... ok
examples/awk/t.f.x... ok
examples/awk/t.f0... ok
examples/awk/t.f1... ok
examples/awk/t.f2... ok
examples/awk/t.f3... ok
examples/awk/t.f4... ok
examples/awk/t.for... ok
examples/awk/t.for1... ok
examples/awk/t.for2... ok
examples/awk/t.for3... ok
examples/awk/t.format4... ok
examples/awk/t.fun... ok
examples/awk/t.fun0... ok
examples/awk/t.fun1... ok
examples/awk/t.fun2... ok
examples/awk/t.fun3... ok
examples/awk/t.fun4... ok
examples/awk/t.fun5... ok
examples/awk/t.getline1... ok
examples/awk/t.getval... ok
examples/awk/t.gsub... ok
examples/awk/t.gsub1... ok
examples/awk/t.gsub3... ok
examples/awk/t.gsub4... ok
examples/awk/t.i.x... ok
examples/awk/t.if... ok
examples/awk/t.in... ok
examples/awk/t.in1... ok
examples/awk/t.in2... ok
examples/awk/t.in3... ok
examples/awk/t.incr... ok
examples/awk/t.incr2... ok
examples/awk/t.incr3... ok
examples/awk/t.index... ok
examples/awk/t.intest... ok
examples/awk/t.intest2... ok
examples/awk/t.j.x... ok
examples/awk/t.longstr... ok
examples/awk/t.makef... ok
examples/awk/t.match... ok
examples/awk/t.match1... ok
examples/awk/t.max... ok
examples/awk/t.mod... ok
examples/awk/t.monotone... ok
examples/awk/t.nameval... ok
examples/awk/t.next... ok
examples/awk/t.NF... ok
examples/awk/t.not... ok
examples/awk/t.null0... ok
examples/awk/t.ofmt... ok
examples/awk/t.ofs... Warning: missing newline at end of file /tmp/doCa41e
Warning: missing newline at end of file examples/out/t.ofs
ok
examples/awk/t.ors... Warning: missing newline at end of file /tmp/dsCai2e
Warning: missing newline at end of file examples/out/t.ors
ok
examples/awk/t.pat... ok
examples/awk/t.pipe... ok
examples/awk/t.pp... ok
examples/awk/t.pp1... ok
examples/awk/t.pp2... ok
examples/awk/t.printf... ok
examples/awk/t.printf2... ok
examples/awk/t.quote... ok
examples/awk/t.randk... ok
examples/awk/t.re1... ok
examples/awk/t.re1a... ok
examples/awk/t.re2... ok
examples/awk/t.re3... ok
examples/awk/t.re4... ok
examples/awk/t.re5... ok
examples/awk/t.re7... ok
examples/awk/t.rec... ok
examples/awk/t.redir1... ok
examples/awk/t.reFS... ok
examples/awk/t.reg... ok
examples/awk/t.roff... ok
examples/awk/t.sep... ok
examples/awk/t.seqno... ok
examples/awk/t.set0... ok
examples/awk/t.set0a... ok
examples/awk/t.set0b... ok
examples/awk/t.set1... ok
examples/awk/t.set2... ok
examples/awk/t.set3... ok
examples/awk/t.split1... ok
examples/awk/t.split2... ok
examples/awk/t.split2a... ok
examples/awk/t.split3... ok
examples/awk/t.split4... ok
examples/awk/t.split8... ok
examples/awk/t.split9... ok
examples/awk/t.split9a... ok
examples/awk/t.stately... ok
examples/awk/t.strcmp... ok
examples/awk/t.strcmp1... ok
examples/awk/t.strnum... ok
examples/awk/t.sub0... ok
examples/awk/t.sub1... ok
examples/awk/t.sub2... ok
examples/awk/t.sub3... ok
examples/awk/t.substr... ok
examples/awk/t.substr1... ok
examples/awk/t.time... ok
examples/awk/t.vf... ok
examples/awk/t.vf1... ok
examples/awk/t.vf2... ok
examples/awk/t.vf3... ok
examples/awk/t.x... ok
a-format.awk... ok
concat-assign-same.awk... ok
decr-NF.awk... ok
fs-overflow.awk... ok
negative-nf.awk... ok
nf-self-assign.awk... ok
numeric-fs.awk... ok
numeric-output-seps.awk... Warning: missing newline at end of file /tmp/nawktest.1dt341/test.temp.out
Warning: missing newline at end of file /tmp/deMaOgf
ok
numeric-rs.awk... ok
numeric-subsep.awk... ok
ofs-rebuild.awk... ok
space.awk... ok
split-fs-from-array.awk... ok
string-conv.awk... ok
subsep-overflow.awk... ok
system-status.awk... ok
unary-plus.awk... ok

# One True AWK Test Programs

./T.-f-f... ok
./T.argv... ok
./T.builtin... ok
./T.chem... ok
./T.close... ok
./T.clv... ok
./T.delete... ok
./T.errmsg... ok
./T.expr... ok
./T.exprconv... ok
./T.flags... ok
./T.func... ok
./T.gawk... ok
./T.getline... ok
./T.latin1... ok
./T.lilly... ok
./T.main... ok
./T.misc... ok
./T.nextfile... ok
./T.overflow... ok
./T.re... ok
./T.recache... ok
./T.redir... ok
./T.rt... ok
./T.split... ok
./T.sub... ok
./T.system... ok

# Imported GAWK Test Programs

anchgsub.awk... ok
anchor.awk... ok
arrayind3.awk... ok
arrayprm2.awk... ok
arrayprm3.awk... ok
arrayref.awk... ok
arrymem1.awk... ok
arynasty.awk... ok
arynocls.awk... ok
aryprm9.awk... ok
arysubnm.awk... ok
aryunasgn.awk... ok
asgext.awk... ok
back89.awk... ok
childin.awk... ok
clobber.awk... ok
closebad.awk... ok
clsflnam.awk... ok
compare2.awk... ok
concat1.awk... ok
concat2.awk... ok
concat3.awk... ok
concat4.awk... ok
concat5.awk... ok
convfmt.awk... ok
datanonl.awk... ok
delarprm.awk... ok
dfastress.awk... ok
dynlj.awk... ok
eofsplit.awk... ok
exit2.awk... ok
exitval2.awk... ok
exitval3.awk... ok
fcall_exit.awk... ok
fcall_exit2.awk... ok
fldchg.awk... ok
fldchgnf.awk... ok
fldterm.awk... ok
fordel.awk... ok
forsimp.awk... ok
fsbs.awk... ok
fsrs.awk... ok
fsspcoln.awk... ok
fstabplus.awk... ok
funsemnl.awk... ok
funstack.awk... ok
gawksub.awk... ok
getline3.awk... ok
getlnbuf.awk... ok
getlnhd.awk... ok
getnr2tb.awk... ok
getnr2tm.awk... ok
gsubtest.awk... ok
gsubtst2.awk... ok
gsubtst5.awk... ok
gsubtst7.awk... ok
gsubtst8.awk... ok
hex.awk... ok
hex2.awk... ok
inpref.awk... ok
inputred.awk... ok
intest.awk... ok
intprec.awk... ok
iobug1.awk... ok
leadnl.awk... ok
longsub.awk... ok
longwrds.awk... ok
manglprm.awk... ok
math.awk... ok
membug1.awk... ok
minusstr.awk... ok
mtchi18n.awk... ok
nasty.awk... ok
nasty2.awk... ok
negexp.awk... ok
negrange.awk... ok
nested.awk... ok
nfldstr.awk... ok
nfloop.awk... ok
nlfldsep.awk... ok
nlinstr.awk... ok
nlstrina.awk... ok
noloop1.awk... ok
noloop2.awk... ok
nulrsend.awk... ok
numindex.awk... ok
numstr1.awk... ok
numsubstr.awk... ok
octsub.awk... ok
ofmt.awk... ok
ofmta.awk... ok
ofmtbig.awk... ok
ofmtfidl.awk... ok
ofmts.awk... ok
ofmtstrnum.awk... ok
onlynl.awk... ok
opasnidx.awk... ok
opasnslf.awk... ok
paramtyp.awk... ok
paramuninitglobal.awk... ok
parse1.awk... ok
pcntplus.awk... ok
pipeio1.awk... ok
pipeio2.awk... ok
prdupval.awk... ok
prec.awk... ok
printf0.awk... ok
printf1.awk... ok
printfchar.awk... ok
prmreuse.awk... ok
prt1eval.awk... ok
range1.awk... ok
rebrackloc.awk... ok
rebt8b1.awk... ok
rebuild.awk... ok
redfilnm.awk... ok
regeq.awk... ok
regexpbrack2.awk... ok
regexprange.awk... ok
reparse.awk... ok
resplit.awk... ok
rri1.awk... ok
rs.awk... ok
rscompat.awk... ok
rsnul1nl.awk... ok
rstest2.awk... ok
rstest3.awk... ok
rstest4.awk... ok
rstest5.awk... ok
rstest6.awk... ok
rswhite.awk... ok
setrec0.awk... ok
setrec1.awk... ok
sigpipe1.awk... ok
splitargv.awk... ok
splitarr.awk... ok
splitdef.awk... ok
splitvar.awk... ok
splitwht.awk... ok
status-close.awk... ok
strcat1.awk... ok
strnum1.awk... ok
strnum2.awk... ok
strtod.awk... ok
subamp.awk... ok
subi18n.awk... ok
subsepnm.awk... ok
subslash.awk... ok
substr.awk... ok
swaplns.awk... ok
tweakfld.awk... ok
uninit2.awk... ok
uninit5.awk... ok
uparrfs.awk... ok
wideidx.awk... ok
wideidx2.awk... ok
widesub.awk... ok
widesub2.awk... ok
widesub3.awk... ok
wjposer1.awk... ok
zero2.awk... ok
zeroe0.awk... ok
zeroflag.awk... ok
./compare.sh... ok
./fflush.sh... ok
./getline2.sh... ok
./leaddig.sh... ok
./localenl.sh... ok
./messages.sh... ok
./nors.sh... ok
./poundbang.sh... ok
./randtest.sh... ok
./rsnulbig.sh... ok
./rsnulbig2.sh... ok

# Imported GAWK Syntax Tests

arrayparm.awk... ok
arryref3.awk... ok
arryref4.awk... ok
arryref5.awk... ok
aryprm1.awk... ok
aryprm2.awk... ok
aryprm3.awk... ok
aryprm4.awk... ok
aryprm5.awk... ok
aryprm6.awk... ok
badassign1.awk... ok
badbuild.awk... ok
callparam.awk... ok
delfunc.awk... ok
fnarray.awk... ok
fnaryscl.awk... ok
fnmisc.awk... ok
funsmnam.awk... ok
gsubasgn.awk... ok
nastyparm.awk... ok
nfneg.awk... ok
noparms.awk... ok
nulinsrc.awk... ok
paramdup.awk... ok
parseme.awk... ok
readbuf.awk... ok
scalar.awk... ok
sclforin.awk... ok
sclifin.awk... ok
synerr1.awk... ok
synerr2.awk... ok
tradanch1.awk... ok
unterm.awk... ok

TOTAL: 475/475
#2

Updated by Cody Mello 5 months ago

Copying all of the util-tests and the test runner into /opt, all of the tests pass fine, including the AWK tests:

cpm@utu /opt/util-tests % export PATH=/opt/onbld/bin:/opt/onbld/bin/i386/:$PATH
cpm@utu /opt/util-tests % pfexec ./bin/utiltest -c ./runfiles/default.run               
Test: /opt/util-tests/tests/allowed-ips (run as root)             [00:00] [PASS]
Test: /opt/util-tests/tests/chown_test (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/date_test (run as root)               [00:00] [PASS]
Test: /opt/util-tests/tests/grep_test (run as root)               [00:00] [PASS]
Test: /opt/util-tests/tests/libjedec_test (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/libsff/libsff (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/mergeq/mqt (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/mergeq/wqt (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/printf_test (run as root)             [00:00] [PASS]
Test: /opt/util-tests/tests/set-linkprop (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/smbios (run as root)                  [00:00] [PASS]
Test: /opt/util-tests/tests/xargs_test (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/awk/runtests.sh (run as nobody)       [01:53] [PASS]
Test: /opt/util-tests/tests/ctf/precheck (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/ctf/ctftest (run as root)             [00:03] [PASS]
Test: /opt/util-tests/tests/demangle/afl-fast (run as root)       [00:01] [PASS]
Test: /opt/util-tests/tests/demangle/gcc-libstdc++ (run as root)  [00:00] [PASS]
Test: /opt/util-tests/tests/demangle/llvm-stdcxxabi (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_00_blank (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_01_boolean (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_02_numbers (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_03_empty_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_04_number_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_05_strings (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_06_nested (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_07_nested_arrays (run as root) [00:00] [PASS]

Results Summary
PASS      26

Running Time:   00:02:01
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20190821T154809
#3

Updated by Electric Monk 5 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 3ee4fc2aa6b5136515cc3eed32d3c6ef33e37471

commit  3ee4fc2aa6b5136515cc3eed32d3c6ef33e37471
Author: Cody Peter Mello <cody.mello@joyent.com>
Date:   2019-08-24T02:54:33.000Z

    11552 Want a more modern nawk(1)
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
    Reviewed by: Andy Fiddaman <andy@omniosce.org>
    Approved by: Gordon Ross <gwr@nexenta.com>

Also available in: Atom PDF