Project

General

Profile

Bug #11952

Updated by Joshua M. Clulow over 2 years ago

Using a large USB hard disk (most likely larger than 2TB) can result in DMA errors and ultimately a stuck device; e.g., 

 <pre> 
 Nov    9 09:21:36 newcastle rootnex: [ID 561485 kern.warning] WARNING: xhci: coding error detected, the driver is using ddi_dma_attr(9S) incorrectly. There is a small risk of data corruption in particular with large I/Os. The driver should be replaced with a corrected version for proper system operation. To disable this warning, add 'set rootnex:rootnex_bind_warn=0' to /etc/system(4). 
 Nov    9 09:21:36 newcastle xhci: [ID 197104 kern.info] NOTICE: xhci0: failed to bind DMA memory: -3 
 Nov    9 09:21:36 newcastle xhci: [ID 902155 kern.info] NOTICE: xhci0: xhci stop endpoint command (2)/slot (3) in wrong state: 19 
 Nov    9 09:21:36 newcastle xhci: [ID 617155 kern.info] NOTICE: xhci0: endpoint is in state 3 
 Nov    9 09:21:36 newcastle xhci: [ID 902155 kern.info] NOTICE: xhci0: xhci stop endpoint command (3)/slot (3) in wrong state: 19 
 Nov    9 09:21:36 newcastle xhci: [ID 617155 kern.info] NOTICE: xhci0: endpoint is in state 3 
 Nov    9 09:21:36 newcastle scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci8086,2064@14/storage@11/disk@0,0 (sd1): 
 Nov    9 09:21:36 newcastle         Command failed to complete...Device is gone 
 </pre> 

 This turns out to be somewhat complicated.    The device shown above is presenting 512 byte sectors (whether they're emulated or not, under the covers) which means the logical block address (LBA) range spans from @0x0@ up to @0x1d1a94a20@; critically, this value is larger than the 32-bit LBA field in a SCSI @READ (12)@ or @WRITE (12)@ (aka @SCMD_READ_G5@ @SCMD_READ_G5 or @SCMD_WRITE_G5@) command.    Block addresses beyond the 32 bit boundary must be encoded in a command that supports a wider LBA, so @sd@ moves up to @READ (16)@ and @WRITE (16)@ (aka @SCMD_READ_G4@ and @SCMD_WRITE_G4@). 

 USB host controllers have a relatively small transfer size limit for an individual request.    As such, we have logic in @scsa2usb_rw_transport()@ to break a larger SCSI read or write command into multiple smaller commands in sequence to satisfy the upstack I/O request.    Today this logic only covers the Group 1 and Group 5 SCSI read and write commands, leaving out the Group 4 SCSI commands which only come into effect for larger logical addresses.    This has only become an issue since the advent of 512 byte sector drives of a capacity larger than about 2TB. 

 The intermittent nature of the I/O failures (e.g., a pool will import, but heavy I/O will subsequently fail and take the pool offline) seems to be on account of at least two things: 

 * ZFS issuing I/O requests that fit within the transfer limit at least during import -- but not later 
 * I/O requests for logical addresses prior to the 2TB boundary appear to be sent as Group 5 commands; only writes to the latter half of a large disk will be Group 4

Back