Project

General

Profile

Bug #10524

wsdiff much slower after move from deprecated commands module

Added by Andy Fiddaman 4 months ago. Updated 4 months ago.

Status:
Closed
Priority:
High
Assignee:
Andy Fiddaman
Category:
tools - gate/build tools
Start date:
2019-03-10
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:

Description

wsdiff was modified under #9979 so that it could be run under wither Python2 or Python3. Part of this work was to replace code that used the deprecated commands module with the newer subprocess module. This change has inadvertently caused a performance regression.

Here are some timings for running wsdiff against usr/bin with both python2 and python3:

python2 =wsdiff proto/root_i386{,-nd}/usr/bin  467.68s user 238.26s system 281% cpu 4:10.37 total
python2 =wsdiff proto/root_i386{,-nd}/usr/bin  463.40s user 236.09s system 280% cpu 4:09.71 total
python3 =wsdiff proto/root_i386{,-nd}/usr/bin  27.50s user 129.93s system 109% cpu 2:23.35 total
python3 =wsdiff proto/root_i386{,-nd}/usr/bin  27.73s user 129.84s system 110% cpu 2:22.86 total

And with python2 using the version before #9979

wsdiff proto/root_i386{,-nd}/usr/bin  11.84s user 11.74s system 69% cpu 33.974 total

Related issues

Related to illumos gate - Feature #9979: Support python3 for in-gate toolsClosed2019-02-06

Actions

History

#1

Updated by Andy Fiddaman 4 months ago

  • Related to Feature #9979: Support python3 for in-gate tools added
#2

Updated by Andy Fiddaman 4 months ago

Proposed patch at:

https://illumos.org/rb/r/1554/

Testing notes:

With the patch, python2 wsdiff is much faster (back to previous speed), python3 remains slower.

python2 =wsdiff proto/root_i386{,-nd}/usr/bin  9.22s user 20.69s system 111% cpu 26.866 total
python3 =wsdiff proto/root_i386{,-nd}/usr/bin  31.24s user 101.09s system 110% cpu 2:00.04 total
#3

Updated by Electric Monk 4 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac

commit  2f7dba3e6747cbaaf1deb86e6ca1e2a5c96332ac
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2019-03-14T20:01:47.000Z

    10524 wsdiff much slower after move from deprecated commands module
    10448 wsdiff explodes on encoding error
    10525 wsdiff output is not correct for a binary file
    10526 wsdiff tries to spawn 4.8 threads
    Reviewed by: Gergő Mihály Doma <domag02@gmail.com>
    Reviewed by: Sebastian Wiedenroth <sebastian.wiedenroth@skylime.net>
    Approved by: Rich Lowe <richlowe@richlowe.net>

Also available in: Atom PDF