Bug #10448
wsdiff explodes on encoding error
Start date:
2019-02-25
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Description
With the recent updates to wsdiff, it seems we've regressed on the ability to correctly diff certain files:
18:34:51 + /opt/onbld/bin/wsdiff -r artefacts/wsdiff.txt previous/proto/root_i386 current/proto/root_i386 18:37:48 platform/i86xpv/kernel/amd64/unix 18:38:27 platform/i86pc/kernel/amd64/unix 18:41:54 etc/motd 18:45:40 usr/sbin/iasl 18:46:14 usr/sbin/acpixtract 18:46:15 usr/sbin/acpidump 19:38:36 usr/lib/spell/hlistb 19:38:36 Exception in thread Thread-2: 19:38:36 Traceback (most recent call last): 19:38:36 File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner 19:38:36 self.run() 19:38:36 File "/opt/onbld/bin/wsdiff", line 1233, in run 19:38:36 compareOneFile(base, ptch, False) 19:38:36 File "/opt/onbld/bin/wsdiff", line 1277, in compareOneFile 19:38:36 return compareBasic(base, ptch, quiet, fileType) 19:38:36 File "/opt/onbld/bin/wsdiff", line 1112, in compareBasic 19:38:36 difference(fileName, fileType, diffs) 19:38:36 File "/opt/onbld/bin/wsdiff", line 195, in difference 19:38:36 log_difference(f, dtype, diffs) 19:38:36 File "/opt/onbld/bin/wsdiff", line 212, in log_difference 19:38:36 print(diffs[:diffs_sz_thresh], file=log) 19:38:36 UnicodeEncodeError: 'ascii' codec can't encode characters in position 64-66: ordinal not in range(128)
Note that usr/lib/spell/hlistb, a binary data file, differs by two bytes between the two workspaces:
root@jenkins-oi0:~# diff \
<(LANG=C od -t x1 /export/jenkins/workspace/wsdiff/previous/proto/root_i386/usr/lib/spell/hlistb) \
<(LANG=C od -t x1 /export/jenkins/workspace/wsdiff/current/proto/root_i386/usr/lib/spell/hlistb)
416c416
< 0014760 d7 f1 1a 93 1a d8 7e e8 98 bf ff a9 7c de 89 b1
---
> 0014760 d7 f1 1a 93 1a d8 7e e8 98 7f ce a9 7c de 89 b1
Richard Lowe and I have previously noticed that hlistb (and hlista) occasionally differ by one or two bytes for no reason we've yet been able to determine -- but we noticed this because prior to the changes, wsdiff did correctly render the differences.
Related issues
History
Updated by
Updated by Joshua M. Clulow 7 months ago
- Related to Feature #9979: Support python3 for in-gate tools added
Updated by Joshua M. Clulow 7 months ago
- Related to Bug #10449: wsdiff hangs forever after an exception is thrown added
Updated by Joshua M. Clulow 7 months ago
Note that the error seen above occurs when wsdiff is run in the en_US.UTF-8 locale. If I instead run the program with LANG=C LC_ALL=C, it doesn't blow up. It's not clear that the diff is correct, though:
usr/lib/spell/hlistb
NOTE: ASCII difference detected.
523,3661c523,3661
< 0010120 032 032 ~ | e 022 V 005 ~ ~ s $ r s $ 6
< 7c7e1a1a 05561265 24737e7e 36247372
< 0010140 e u * 032 M 3 q $ 7 025 N | 002 P L 037
< 1a2a7565 2471334d 7c4e1537 1f4c5002
< 0010160 K { A S # Q r 1 - E ~ \0 \0 \0 \0 '
< 53417b4b 31725123 007e452d 27000000
< 0010200 " S B i * 020 K 030 _ g N # 006 \b 033
< 42532220 4b102a69 4e675f18 1b080623
< 0010220 ) c c w ] ^ 027 9 \ t { k Y Z \b q
< 77636329 39175e5d 6b7b745c 71085a59
< 0010240 \n 007 @ 002 K X n \b S 016 025 \0 007 1 M 0
...
Updated by Andy Fiddaman 6 months ago
Updated by - Status changed from New to In Progress
- Assignee set to Andy Fiddaman
Updated by Electric Monk 6 months ago
Updated by - Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit 2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac
commit 2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date: 2019-03-14T20:01:47.000Z
10524 wsdiff much slower after move from deprecated commands module
10448 wsdiff explodes on encoding error
10525 wsdiff output is not correct for a binary file
10526 wsdiff tries to spawn 4.8 threads
Reviewed by: Gergő Mihály Doma <domag02@gmail.com>
Reviewed by: Sebastian Wiedenroth <sebastian.wiedenroth@skylime.net>
Approved by: Rich Lowe <richlowe@richlowe.net>