Project

General

Profile

Bug #10448

wsdiff explodes on encoding error

Added by Joshua M. Clulow 7 months ago. Updated 6 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
tools - gate/build tools
Start date:
2019-02-25
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

With the recent updates to wsdiff, it seems we've regressed on the ability to correctly diff certain files:

18:34:51 + /opt/onbld/bin/wsdiff -r artefacts/wsdiff.txt previous/proto/root_i386 current/proto/root_i386
18:37:48 platform/i86xpv/kernel/amd64/unix
18:38:27 platform/i86pc/kernel/amd64/unix
18:41:54 etc/motd
18:45:40 usr/sbin/iasl
18:46:14 usr/sbin/acpixtract
18:46:15 usr/sbin/acpidump
19:38:36 usr/lib/spell/hlistb
19:38:36 Exception in thread Thread-2:
19:38:36 Traceback (most recent call last):
19:38:36   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
19:38:36     self.run()
19:38:36   File "/opt/onbld/bin/wsdiff", line 1233, in run
19:38:36     compareOneFile(base, ptch, False)
19:38:36   File "/opt/onbld/bin/wsdiff", line 1277, in compareOneFile
19:38:36     return compareBasic(base, ptch, quiet, fileType)
19:38:36   File "/opt/onbld/bin/wsdiff", line 1112, in compareBasic
19:38:36     difference(fileName, fileType, diffs)
19:38:36   File "/opt/onbld/bin/wsdiff", line 195, in difference
19:38:36     log_difference(f, dtype, diffs)
19:38:36   File "/opt/onbld/bin/wsdiff", line 212, in log_difference
19:38:36     print(diffs[:diffs_sz_thresh], file=log)
19:38:36 UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-66: ordinal not in range(128)

Note that usr/lib/spell/hlistb, a binary data file, differs by two bytes between the two workspaces:

root@jenkins-oi0:~# diff \
    <(LANG=C od -t x1 /export/jenkins/workspace/wsdiff/previous/proto/root_i386/usr/lib/spell/hlistb) \
    <(LANG=C od -t x1 /export/jenkins/workspace/wsdiff/current/proto/root_i386/usr/lib/spell/hlistb)

416c416
< 0014760 d7 f1 1a 93 1a d8 7e e8 98 bf ff a9 7c de 89 b1
---
> 0014760 d7 f1 1a 93 1a d8 7e e8 98 7f ce a9 7c de 89 b1

Richard Lowe and I have previously noticed that hlistb (and hlista) occasionally differ by one or two bytes for no reason we've yet been able to determine -- but we noticed this because prior to the changes, wsdiff did correctly render the differences.


Related issues

Related to illumos gate - Bug #10225: wsdiff broken after 9979Closed2019-01-12

Actions
Related to illumos gate - Feature #9979: Support python3 for in-gate toolsClosed2019-02-06

Actions
Related to illumos gate - Bug #10449: wsdiff hangs forever after an exception is thrownNew2019-02-25

Actions

History

#1

Updated by Joshua M. Clulow 7 months ago

  • Related to Bug #10225: wsdiff broken after 9979 added
#2

Updated by Joshua M. Clulow 7 months ago

  • Related to Feature #9979: Support python3 for in-gate tools added
#3

Updated by Joshua M. Clulow 7 months ago

  • Description updated (diff)
#4

Updated by Joshua M. Clulow 7 months ago

  • Related to Bug #10449: wsdiff hangs forever after an exception is thrown added
#5

Updated by Joshua M. Clulow 7 months ago

Note that the error seen above occurs when wsdiff is run in the en_US.UTF-8 locale. If I instead run the program with LANG=C LC_ALL=C, it doesn't blow up. It's not clear that the diff is correct, though:

usr/lib/spell/hlistb
NOTE: ASCII difference detected.

523,3661c523,3661
< 0010120 032 032   ~   |   e 022   V 005   ~   ~   s   $   r   s   $   6
<         7c7e1a1a 05561265 24737e7e 36247372
< 0010140   e   u   * 032   M   3   q   $   7 025   N   | 002   P   L 037
<         1a2a7565 2471334d 7c4e1537 1f4c5002
< 0010160   K   {   A   S   #   Q   r   1   -   E   ~  \0  \0  \0  \0   '
<         53417b4b 31725123 007e452d 27000000
< 0010200       "   S   B   i   * 020   K 030   _   g   N   # 006  \b 033
<         42532220 4b102a69 4e675f18 1b080623
< 0010220   )   c   c   w   ]   ^ 027   9   \   t   {   k   Y   Z  \b   q
<         77636329 39175e5d 6b7b745c 71085a59
< 0010240  \n 007   @ 002   K   X   n  \b   S 016 025  \0 007   1   M   0
...
#6

Updated by Andy Fiddaman 6 months ago

  • Status changed from New to In Progress
  • Assignee set to Andy Fiddaman
#7

Updated by Electric Monk 6 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac

commit  2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2019-03-14T20:01:47.000Z

    10524 wsdiff much slower after move from deprecated commands module
    10448 wsdiff explodes on encoding error
    10525 wsdiff output is not correct for a binary file
    10526 wsdiff tries to spawn 4.8 threads
    Reviewed by: Gergő Mihály Doma <domag02@gmail.com>
    Reviewed by: Sebastian Wiedenroth <sebastian.wiedenroth@skylime.net>
    Approved by: Rich Lowe <richlowe@richlowe.net>

Also available in: Atom PDF