Project

General

Profile

Actions

Bug #9695

closed

Slow crash dumps, significantly slower than live core

Added by Yuri Pankov about 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
driver - device drivers
Start date:
2018-08-02
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:
Gerrit CR:

Description

The table below showscomparative data for approx 10.5G crash dumps, with various different dump devices (all zvol). The compression ratio for this scenario is 5.6, so about 2G of data is written to the dump device. The IO Size is 128k.

driver Parallel / Serial dump time (s) IORate device write (ms) write ns/pg pages /sec
mr_sas S 192 13.04 9.59 56572 15090
mpt_sas S 42 111.95 1.12 5951 66940
mr_sas P 148 13.03 9.59 53310 18797
mpt_sas P 17 114.19 1.09 6113 163469
sata S 357 6.53 19.14 107711 7802
mpt_sas S 73 119.33 1.05 6102 38227
sata P 301 6.53 19.14 107807 9263
mpt_sas P 18 113.78 1.1 6343 154874

What we can see from above is that the time for each write to the dump device (125 / IO Rate) is approx:

driver device write (ms)
mpt_sas 1
mr_sas 10
sata 20

It appears that the observed dump performance issues are when using SATA or MR_SAS drivers. This is pretty much specific to dump, in that it is the SYNC/POLLED write path. Tests with savecore -L have not shown such differences (although SATA still about 2x slower than SAS, but that's probably closer to what we'd expect).

The conclusion is that the SYNC/POLLED write path (as used by dump) in SATA and MR_SAS is too slow.

The table also shows the comparison between Serial and Parallel dump times. The IORate is unaffected, as expected, but in cases with a reasonable IORate (mpt_sas) the compression parallelism provides significant benefit.

Investigating mr_sas:
It turns out that the mr_sas driver checks for polled write completion after 10ms, which would explain why a write appears to take 10ms. By reducing this to 1ms (which is what is used if the command were issued via bolt_issue_cmd_in_poll_mode) dump times can be reduced significantly. The above 192s (serial) / 148s (parallel) times are reduced to 39s and 15s.

Investigating sata:
There is a similar issue in the ahci SATA driver. The poll wait time is 10ms. Reducing this to 1ms improves IORate by approximately a factor of 10. However, in the SATA case the IORate was about 20x lower, so dump writes still take twice as long as SAS. The reason for this is that the design of this driver causes the drv_usecwait() to be done twice on every call.

Actions #1

Updated by Electric Monk almost 3 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 50 to 100

git commit d8873b3136f0985c9e33ff8801644e1b4253b36c

commit  d8873b3136f0985c9e33ff8801644e1b4253b36c
Author: Joyce McIntosh <joyce.mcintosh@nexenta.com>
Date:   2018-08-07T19:46:09.000Z

    9695 Slow crash dumps, significantly slower than live core
    Reviewed by: Dan Fields <dan.fields@nexenta.com>
    Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
    Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
    Reviewed by: Toomas Soome <tsoome@me.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Actions

Also available in: Atom PDF