Project

General

Profile

Actions

Bug #14743

closed

new vioscsi driver

Added by Garrett D'Amore 21 days ago. Updated 7 days ago.

Status:
Closed
Priority:
Normal
Category:
driver - device drivers
Start date:
Due date:
% Done:

100%

Estimated time:
60.00 h
Difficulty:
Hard
Tags:
Gerrit CR:

Description

There are currently two public drivers for virtio-scsi floating about -- one from Nexenta and another from Joshua Clulow.

Unfortunately they both suffer from being somewhat incomplete, and there are a number of bugs in the Nexenta driver. I've spent some time lately and have put together a third effort, based originally upon Nexenta's effort, but with rather substantial improvements:

  • iport(9) based (and completely DDI compliant, apart from the virtio interfaces documented in the virtio header)
  • supports dynamic disk discovery (and removal) -- i.e. hotplug
  • supports reset of target or lun (but not bus)
  • supports abort (scsi_abort)
  • honors pkt_time for SCSI request timeouts
  • supports tagged queuing and untagged queuing
  • honors virtio limit on commands issued to a single LUN

The resulting code now bears very little resemblance to the original Nexenta code. (Owing to use of iports, it may more closely resemble Joshua's version.)

I have tested with KVM on CentOS 8 with some degree of success. I will be posting this up for general review soon.

Actions #1

Updated by Electric Monk 21 days ago

  • Gerrit CR set to 2185
Actions #2

Updated by Joshua M. Clulow 21 days ago

It would be good to make sure this works with Google Cloud (GCE) too, which was my original motivation for my vioscsi driver. Their control plane allows for hot addition and removal of disks, I believe, as well, which I had managed to get going there.

Actions #3

Updated by Andy Fiddaman 21 days ago

This gives me some incentive to see about getting the bhyve vioscsi support wired up too, if only for testing and then subsequently adding support for dynamic LUN resizing, automatic zpool expansion etc. (we may get quite a lot of that for free if this is using sd)

Actions #4

Updated by Garrett D'Amore 13 days ago

  • Status changed from New to Pending RTI
Actions #5

Updated by Garrett D'Amore 10 days ago

It is indeed using sd.

This driver has been tested in the following environments:

CentOS 8 based KVM using libvirt
qemu based system on OmniOS
UTM based KVM (on a mac m1!)
Nutanix
Google Cloud (e2 instance)

We have tested use for booting (no longer any need to use IDE for these deployments!), and hot plug attach and detach. Some environments attach new drives as logical units under target 0 (e.g. libvirt, Nutanix) and others attach new drives as new targets (e.g. GCE). Both work well, although the code paths by which they work are totally different.

I've also verified that various SCSI data like page 83 WWN information is passed through correctly, assuming that it has been correctly configured in the hypervisor.

All of this testing has been done using RackTop's distribution of illumos -- BrickStorOS.

And I would really love to see bhyve wired up for this as well. :-)

Actions #6

Updated by Garrett D'Amore 10 days ago

One more thing. The driver that I have out for RTI was also explicitly tested for timeout, abort, and reset handling.

The only testing I have not performed that I'd like to see done is the passthrough of physical disks. I've not done this because I don't really have a configuration at hand suitable for such testing. However, I have every reason to believe it will also Just Work.

Note that follow up work is coming which I hope will simplify the LUN based hot attach of disks, by adding better support for this to the iport / target map framework in SCSA itself. But I'd like to merge this first, so that folks can start playing with it right away.

Actions #7

Updated by Joshua M. Clulow 7 days ago

Testing Notes (Supplemental)

After fixing the comment formatting, I built the change and tested it under QEMU on my Ubuntu 21.10 workstation:

$ qemu-system-x86_64 --version
QEMU emulator version 6.0.0 (Debian 1:6.0+dfsg-2expubuntu1.3)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

With an appropriate Virtio SCSI device, as specified in the libvirt domain XML:

  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/tmp/vioscsi.raw' index='1'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
...

I was able to see the disk:

ok set disable-xhci=true
ok set nmi=kmdb
ok boot -k
Loading unix...
Loading /platform/i86pc/amd64/boot_archive...
Loading /platform/i86pc/amd64/boot_archive.hash...
Booting...
Loading kmdb...
Oxide Helios Version garrett/vioscsi-0-gdbf5a1cc29 64-bit (onu)
NOTICE: driver xhci disabled
NOTICE: Performing full ZFS device scan!
NOTICE: Original /devices path (/pseudo/lofi@2:a) not available; ZFS is trying an alternate path (/pci@0,0/pci1af4,8@6/iport@iport0/disk@0,0:a)
NOTICE: vdev_disk_open /dev/dsk/c2t2d0s0: update devid from '<none>' to 'id1,sd@f0ba951d562bb8e030002d96b0000/a'
NOTICE: vdev_disk_open /dev/dsk/c2t2d0s0: update devid from '<none>' to 'id1,sd@f0ba951d562bb8e030002d96b0000/a'
Configuring devices.
Hostname: unknown

unknown console login: root
Password:
Jun 28 23:26:26 unknown login: ROOT LOGIN /dev/console
The illumos Project     garrett/vioscsi-0-gdbf5a1cc29   June 2022
root@unknown:~# diskinfo
TYPE    DISK                    VID      PID              SIZE          RMV SSD
SCSI    c1t0d0                  QEMU     QEMU HARDDISK       0.98 GiB   no  no
root@unknown:~# ls -l /dev/dsk/c1t0d0
lrwxrwxrwx   1 root     root          58 Jun 28 23:26 /dev/dsk/c1t0d0 -> ../../devices/pci@0,0/pci1af4,8@6/iport@iport0/disk@0,0:wd
Actions #8

Updated by Electric Monk 7 days ago

  • Status changed from Pending RTI to Closed
  • % Done changed from 90 to 100

git commit 99e2a6f8e952fd927a72b75323d3e56bcbcda40a

commit  99e2a6f8e952fd927a72b75323d3e56bcbcda40a
Author: Garrett D'Amore <garrett@damore.org>
Date:   2022-06-28T23:33:08.000Z

    14743 new vioscsi driver
    Reviewed by: Gordon Ross <Gordon.W.Ross@gmail.com>
    Reviewed by: Albert Lee <trisk@forkgnu.org>
    Reviewed by: Igor Kozhukhov <igor@dilos.org>
    Reviewed by: Jerry Jelinek <gjelinek@gmail.com>
    Approved by: Joshua M. Clulow <josh@sysmgr.org>

Actions

Also available in: Atom PDF