Bug #10448
wsdiff explodes on encoding error
Start date:
2019-02-25
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Description
With the recent updates to wsdiff, it seems we've regressed on the ability to correctly diff certain files:
18:34:51 + /opt/onbld/bin/wsdiff -r artefacts/wsdiff.txt previous/proto/root_i386 current/proto/root_i386 18:37:48 platform/i86xpv/kernel/amd64/unix 18:38:27 platform/i86pc/kernel/amd64/unix 18:41:54 etc/motd 18:45:40 usr/sbin/iasl 18:46:14 usr/sbin/acpixtract 18:46:15 usr/sbin/acpidump 19:38:36 usr/lib/spell/hlistb 19:38:36 Exception in thread Thread-2: 19:38:36 Traceback (most recent call last): 19:38:36 File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner 19:38:36 self.run() 19:38:36 File "/opt/onbld/bin/wsdiff", line 1233, in run 19:38:36 compareOneFile(base, ptch, False) 19:38:36 File "/opt/onbld/bin/wsdiff", line 1277, in compareOneFile 19:38:36 return compareBasic(base, ptch, quiet, fileType) 19:38:36 File "/opt/onbld/bin/wsdiff", line 1112, in compareBasic 19:38:36 difference(fileName, fileType, diffs) 19:38:36 File "/opt/onbld/bin/wsdiff", line 195, in difference 19:38:36 log_difference(f, dtype, diffs) 19:38:36 File "/opt/onbld/bin/wsdiff", line 212, in log_difference 19:38:36 print(diffs[:diffs_sz_thresh], file=log) 19:38:36 UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-66: ordinal not in range(128)
Note that usr/lib/spell/hlistb, a binary data file, differs by two bytes between the two workspaces:
root@jenkins-oi0:~# diff \
<(LANG=C od -t x1 /export/jenkins/workspace/wsdiff/previous/proto/root_i386/usr/lib/spell/hlistb) \
<(LANG=C od -t x1 /export/jenkins/workspace/wsdiff/current/proto/root_i386/usr/lib/spell/hlistb)
416c416
< 0014760 d7 f1 1a 93 1a d8 7e e8 98 bf ff a9 7c de 89 b1
---
> 0014760 d7 f1 1a 93 1a d8 7e e8 98 7f ce a9 7c de 89 b1
Richard Lowe and I have previously noticed that hlistb (and hlista) occasionally differ by one or two bytes for no reason we've yet been able to determine -- but we noticed this because prior to the changes, wsdiff did correctly render the differences.
Related issues
History
Updated by
Updated by Joshua M. Clulow 12 months ago
- Related to Feature #9979: Support python3 for in-gate tools added
Updated by Joshua M. Clulow 12 months ago
- Related to Bug #10449: wsdiff hangs forever after an exception is thrown added
Updated by Joshua M. Clulow 12 months ago
Note that the error seen above occurs when wsdiff is run in the en_US.UTF-8 locale. If I instead run the program with LANG=C LC_ALL=C, it doesn't blow up. It's not clear that the diff is correct, though:
usr/lib/spell/hlistb
NOTE: ASCII difference detected.
523,3661c523,3661
< 0010120 032 032 ~ | e 022 V 005 ~ ~ s $ r s $ 6
< 7c7e1a1a 05561265 24737e7e 36247372
< 0010140 e u * 032 M 3 q $ 7 025 N | 002 P L 037
< 1a2a7565 2471334d 7c4e1537 1f4c5002
< 0010160 K { A S # Q r 1 - E ~ \0 \0 \0 \0 '
< 53417b4b 31725123 007e452d 27000000
< 0010200 " S B i * 020 K 030 _ g N # 006 \b 033
< 42532220 4b102a69 4e675f18 1b080623
< 0010220 ) c c w ] ^ 027 9 \ t { k Y Z \b q
< 77636329 39175e5d 6b7b745c 71085a59
< 0010240 \n 007 @ 002 K X n \b S 016 025 \0 007 1 M 0
...
Updated by Andy Fiddaman 12 months ago
Updated by - Status changed from New to In Progress
- Assignee set to Andy Fiddaman
Updated by Electric Monk 12 months ago
Updated by - Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit 2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac
commit 2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date: 2019-03-14T20:01:47.000Z
10524 wsdiff much slower after move from deprecated commands module
10448 wsdiff explodes on encoding error
10525 wsdiff output is not correct for a binary file
10526 wsdiff tries to spawn 4.8 threads
Reviewed by: Gergő Mihály Doma <domag02@gmail.com>
Reviewed by: Sebastian Wiedenroth <sebastian.wiedenroth@skylime.net>
Approved by: Rich Lowe <richlowe@richlowe.net>