Project

General

Profile

Bug #9291

NVMe driver is slower than expected

Added by Adam Stylinski over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
driver - device drivers
Start date:
2018-03-16
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Using an Intel Optane 800p, rated for 18us write latency, I'm seeing latencies on average in the hundreds of microseconds. Using iostat, 400us is typical, and using the dtrace provider and filtering on the path_name, I'm seeing a distribution that looks like this:

disk I/O latency (ns)                            
           value  ------------- Distribution ------------- count    
            4096 |                                         0        
            8192 |                                         18      
           16384 |@                                        242      
           32768 |@                                        368      
           65536 |@                                        395      
          131072 |@@@@@@@@@@@@@@@@@@@@                     5760    
          262144 |@@@@@@                                   1778    
          524288 |@@@@@@@@                                 2190    
         1048576 |@@@                                      725      
         2097152 |                                         0        
         4194304 |                                         8        
         8388608 |                                         0

Typical write throughput when I made the device its own zpool was around 355MB/sec for a device that should be pushing 680MB/sec. Right now this device is a log device in a pool, and the above workload was generated using a synthetic benchmark with mysql (sysbench).


Related issues

Related to illumos gate - Feature #11827: Increase concurrency through blkdevClosed

Actions

History

#1

Updated by Adam Stylinski over 1 year ago

Dtrace script that generated the above:

#!/usr/sbin/dtrace -s
/*
 * iolatency.d  Show distribution of disk I/O latency.
 *
 * From Chapter 4 of the DTrace book.
 */
#pragma D option quiet

io:::start
/args[1]->dev_pathname=="/devices/pci@0,0/pci8086,155@1,1/pci8086,3802@0/blkdev@w<redacted>,0:a"/
{
        start[arg0] = timestamp;
}

io:::done
/this->start = start[arg0]/
{
        @time["disk I/O latency (ns)"] = quantize(timestamp - this->start);
        start[arg0] = 0;
}
#2

Updated by Marcel Telka over 1 year ago

  • Description updated (diff)
  • Category set to driver - device drivers
#3

Updated by Adam Stylinski 27 days ago

  • Related to Feature #11827: Increase concurrency through blkdev added

Also available in: Atom PDF