illumos gate

The attached dtrace script attempts to measure NVMe device latency. The drives have a physical blocksize 4096 bytes and logical blocksize 512. An example command is below:

nvme_latency.d $lun -c "/usr/bin/dd if=/dev/zero of=/dev/rdsk/$lun bs=512 count=10000000"

where lun=c7tED41011480CF0400d0

Note: With larger blocksizes the count is reduced according to the size of the drive (ie for a 480GB drive with bs=1048576 count=457854)

With bs < 4096 (ie 512, 1024 and 2048) the standard deviation is reported as -1 usec (with %@d) or the equivalent 18446744073709551615 with %@u. See the results below run on a 480GB drive:

bs=1048576: LATENCY DEVICE: 1580b60200cf0400: 465 +/- 772 usec (1831463 count)
bs=524288: LATENCY DEVICE: 1580b60200cf0400: 463 +/- 761 usec (1831471 count)
bs=262144: LATENCY DEVICE: 1580b60200cf0400: 462 +/- 770 usec (1831464 count)
bs=131072: LATENCY DEVICE: 1580b60200cf0400: 221 +/- 499 usec (3662884 count)
bs=65536: LATENCY DEVICE: 1580b60200cf0400: 105 +/- 313 usec (7325728 count)
bs=32768: LATENCY DEVICE: 1580b60200cf0400: 54 +/- 154 usec (10000047 count)
bs=16384: LATENCY DEVICE: 1580b60200cf0400: 36 +/- 92 usec (10000030 count)
bs=8192: LATENCY DEVICE: 1580b60200cf0400: 28 +/- 62 usec (10000032 count)
bs=4096: LATENCY DEVICE: 1580b60200cf0400: 26 +/- 45 usec (10000022 count)
bs=2048: LATENCY DEVICE: 1580b60200cf0400: 997 +/- 18446744073709551615 usec (10000503 count)
bs=1024: LATENCY DEVICE: 1580b60200cf0400: 1434 +/- 18446744073709551615 usec (10000716 count)
bs=512: LATENCY DEVICE: 1580b60200cf0400: 1651 +/- 18446744073709551615 usec (10000793 count)

The output above was with printa() format %@u. With printa() format %@d the last three standard deviations are -1.