illumos gate

When launching illumos as an HVM Guest, with Xen 4.x as the hypervisor, you can only attach IDE devices (which are visible and works). Attaching SCSI/block devices aren't visible to the operating system.

WARNING: pv driver failed to connect: /xpvd/xdf@5632
WARNING: PV access to device disabled: /pci@0,0/pci-ide@1,1/ide@1/sd@0,0

The code in https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/i86pc/i86hvm is > 5 years old and needs some rework. This would have last been working with Xen 3.0/3.1 in 2009.

Note that illumos works well as a PV guest with Xen 4.x.

Console output:

module /platform/i86pc/kernel/amd64/unix: text at [0xfffffffffb800000, 0xfffffffffb96563b] data at 0xfffffffffbc00000
module /kernel/amd64/genunix: text at [0xfffffffffb965640, 0xfffffffffbbedfaf] data at 0xfffffffffbc95640
Loading kmdb...
module /kernel/misc/amd64/kmdbmod: text at [0xfffffffffbd05610, 0xfffffffffbdbc577] data at 0xfffffffffbdbc580
module /kernel/misc/amd64/ctf: text at [0xfffffffffbbedfb0, 0xfffffffffbbf8907] data at 0xfffffffffbdd85c0

Welcome to kmdb
kmdb: unable to determine terminal type: assuming `vt100'
Loaded modules: [ unix krtld genunix ]
[0]> :c
OpenIndiana Build oi_151a8 64-bit (illumos 7256a34efe)
SunOS Release 5.11 - Copyright 1983-2010 Oracle and/or its affiliates.
All rights reserved. Use is subject to license terms.
x86_feature: lgpg
x86_feature: tsc
x86_feature: msr
x86_feature: mtrr
x86_feature: pge
x86_feature: de
x86_feature: cmov
x86_feature: mmx
x86_feature: mca
x86_feature: pae
x86_feature: cv8
x86_feature: pat
x86_feature: sep
x86_feature: sse
x86_feature: sse2
x86_feature: htt
x86_feature: asysc
x86_feature: nx
x86_feature: sse3
x86_feature: cx16
x86_feature: cmp
x86_feature: tscp
x86_feature: cpuid
x86_feature: ssse3
x86_feature: sse4_1
x86_feature: sse4_2
x86_feature: clfsh
x86_feature: 64
x86_feature: pclmulqdq
x86_feature: xsave
x86_feature: rdrand
mem = 1740408K (0x6a39e000)
SMBIOS v2.4 loaded (300 bytes)root nexus = i86pc
pseudo0 at root
pseudo0 is /pseudo
scsi_vhci0 at root
scsi_vhci0 is /scsi_vhci
Reading Intel IOMMU boot options
pci0 at root: space 0 offset 0
pci0 is /pci@0,0
PCI-device: isa@1, isa0
PCI-device: pci5853,1@3, xpv0
xpv0 is /pci@0,0/pci5853,1@3
xpvd0 at root
xdf@768, xdf0
xdf0 is /xpvd/xdf@768
/xpvd/xdf@768 (xdf0) online
xdf@768: 16777216 blocksxdf@51728, xdf4
xdf4 is /xpvd/xdf@51728
/xpvd/xdf@51728 (xdf4) online
xdf@51728: 16384000 blocksxdf@5632, xdf1
xdf1 is /xpvd/xdf@5632
/xpvd/xdf@5632 (xdf1) online
xdf@5632: 1016824 blocks/xpvd/xnf@0 (xnf0) online
pseudo-device: acpippm0
acpippm0 is /pseudo/acpippm@0
pseudo-device: ppm0
ppm0 is /pseudo/ppm@0
IDE device at targ 0, lun 0 lastlun 0x0
model QEMU HARDDISK
ATA/ATAPI-7 supported, majver 0xf0 minver 0x16
PCI-device: ide@0, ata0
ata0 is /pci@0,0/pci-ide@1,1/ide@0
UltraDMA mode 5 selected
cmdk0 at ata0 target 0 lun 0
cmdk0 is /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
root on rpool/ROOT/openindiana fstype zfs
/cpus (cpunex0) online
pseudo-device: dld0
dld0 is /pseudo/dld@0
ISA-device: asy0
asy0 is /pci@0,0/isa@1/asy@1,3f8
8042 device: keyboard@0, kb8042 # 0
kb80420 is /pci@0,0/isa@1/i8042@1,60/keyboard@0
8042 device: mouse@1, mouse8042 # 0
mouse80420 is /pci@0,0/isa@1/i8042@1,60/mouse@1
cpu0: x86 (chipid 0x0 GenuineIntel 306C3 family 6 model 60 step 3 clock 2993 MHz)
cpu0: Intel(r) Pentium(r) CPU G3220 @ 3.00GHz
NOTICE: System detected 256 cpus, but only 1 cpu(s) were enabled during boot.
NOTICE: Use "boot-ncpus" parameter to enable more CPU. See eeprom(1M).
pseudo-device: stmf_sbd0
stmf_sbd0 is /pseudo/stmf_sbd@0
pseudo-device: audio0
audio0 is /pseudo/audio@0
[ network/ip-interface-management:default starting (IP interface management) ]
[ network/netcfg:default starting (Network configuration data management) ]
[ network/datalink-management:default starting (data-link management daemon) ]
[ network/loopback:default starting (loopback network interface) ]
[ network/physical:nwam starting (physical network interface autoconfiguration) ]
[ system/identity:node starting (system identity (nodename)) ]
[ network/iptun:default starting (IP tunnel configuration) ]
Hostname: openindiana
[ system/metainit:default starting (SVM initialization) ]
[ system/filesystem/root:default starting (root file system mount) ]
pseudo-device: zfs0
zfs0 is /pseudo/zfs@0
[ system/scheduler:default starting (default scheduling class configuration) ]
[ system/cryptosvc:default starting (cryptographic services) ]
[ system/auditset:default starting (Set non-/attributable audit flags
in the kernel context.) ]
[ system/boot-archive:default starting (check boot archive content) ]
[ system/filesystem/usr:default starting (read/write root file systems mounts) ]
This Solaris instance has UUID c86aed1c-9f8b-e1e2-d78e-ed38889e22ed
dump on /dev/zvol/dsk/rpool/dump size 850 MB
[ system/pfexec:default starting (Supports in-kernel pfexec and forced privileges.) ]
[ system/keymap:default starting (keyboard defaults) ]
[ system/device/local:default starting (Standard Solaris device configuration.) ]
[ network/ipsec/ipsecalgs:default starting (IPsec algorithm initialization) ]
[ system/filesystem/minimal:default starting (minimal file system mounts) ]
[ system/sysevent:default starting (system event notification) ]
[ system/logadm-upgrade:default starting (logadm upgrade) ]
[ system/metasync:default starting (SVM metasync) ]
[ system/hostid:default starting (hostid) ]
[ system/resource-mgmt:default starting (Global zone resource management settings) ]
[ system/pkgserv:default starting (Flush package command database to disk (see pkgadm(1m)).) ]
[ system/rmtmpfiles:default starting (remove temporary files) ]
[ system/rbac:default starting (Assemble the RBAC *attr files.) ]
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
[ network/npiv_config:default starting ]
[ system/picl:default starting (platform information and control) ]
[ platform/i86pc/acpihpd:default starting (Intel ACPI hot-plug daemon) ]
acpinex0 at root
acpinex0 is /fw
iscsi0 at root
iscsi0 is /iscsi
acpinex: sb@0, acpinex1
acpinex1 is /fw/sb@0
ATAPI device at targ 0, lun 0 lastlun 0x0
model QEMU DVD-ROM
ATA/ATAPI-4 supported, majver 0x1e minver 0x0
PCI-device: ide@1, ata1
ata1 is /pci@0,0/pci-ide@1,1/ide@1
UltraDMA mode 5 selected
pseudo-device: pseudo1
pseudo1 is /pseudo/zconsnex@1
/xpvd/xencons@0 (xencons0) offline
ISA-device: fdc0
ISA-device: ecpp0
ecpp0 is /pci@0,0/isa@1/lp@1,378
WARNING: pv driver failed to connect: /xpvd/xdf@5632
WARNING: PV access to device disabled: /pci@0,0/pci-ide@1,1/ide@1/sd@0,0
ISA-device: pit_beep0
pit_beep0 is /pci@0,0/isa@1/pit_beep
pseudo-device: dcpc0
dcpc0 is /pseudo/dcpc@0
pseudo-device: dtrace0
dtrace0 is /pseudo/dtrace@0
sd0 at ata1: target 0 lun 0
sd0 is /pci@0,0/pci-ide@1,1/ide@1/sd@0,0
/xpvd/xenbus@0 (xenbus0) offline
pseudo-device: fasttrap0
fasttrap0 is /pseudo/fasttrap@0
/xpvd/domcaps@0 (domcaps0) offline
pseudo-device: fbt0
fbt0 is /pseudo/fbt@0
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat@0
/xpvd/balloon@0 (balloon0) offline
pseudo-device: profile0
profile0 is /pseudo/profile@0
pseudo-device: sdt0
sdt0 is /pseudo/sdt@0
pseudo-device: systrace0
systrace0 is /pseudo/systrace@0
pseudo-device: nvidia255
nvidia255 is /pseudo/nvidia@255
pseudo-device: fcp0
fcp0 is /pseudo/fcp@0
pseudo-device: srn0
srn0 is /pseudo/srn@0
pseudo-device: fct0
fct0 is /pseudo/fct@0
pseudo-device: stmf0
stmf0 is /pseudo/stmf@0
pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm@0
pseudo-device: llc10
llc10 is /pseudo/llc1@0
pseudo-device: lofi0
lofi0 is /pseudo/lofi@0
pseudo-device: power0
power0 is /pseudo/power@0
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk@1024
pseudo-device: ucode0
ucode0 is /pseudo/ucode@0
pseudo-device: pool0
pool0 is /pseudo/pool@0
pseudo-device: fssnap0
fssnap0 is /pseudo/fssnap@0
pseudo-device: nsmb0
nsmb0 is /pseudo/nsmb@0
pseudo-device: pm0
pm0 is /pseudo/pm@0
pseudo-device: bpf0
bpf0 is /pseudo/bpf@0
[ system/device/fc-fabric:default starting (Solaris FC fabric device configuration.) ]
[ milestone/devices:default starting (device configuration milestone) ]
[ milestone/name-services:default starting (name services milestone) ]
[ system/manifest-import:default starting (service manifest import) ]
[ application/print/service-selector:default starting (print service selector) ]
[ network/location:default starting (network interface configuration) ]
[ system/coreadm:default starting (system-wide core file configuration) ]
[ milestone/network:default starting (Network milestone) ]
[ milestone/single-user:default starting (single-user milestone) ]
[ system/fcoe_initiator:default starting (fcoe initiator service) ]
[ network/routing-setup:default starting (Initial routing-related configuration.) ]
[ network/initial:default starting (initial network services) ]
[ system/filesystem/local:default starting (local file system mounts) ]
[ network/service:default starting (layered network services) ]
[ network/netmask:default starting (Reset the netmask and broadcast address) ]
[ system/boot-archive-update:default starting (update boot archive if necessary) ]
[ system/dumpadm:default starting (system crash dump configuration) ]
[ network/shares/group:zfs starting (Share Group) ]
[ network/shares/group:default starting (Share Group) ]
[ system/cron:default starting (clock daemon (cron)) ]
[ network/physical:nwam starting (physical network interface autoconfiguration) ]
[ network/dns/client:default starting (DNS resolver) ]
[ system/identity:domain starting (system identity (domainname)) ]
[ system/sysidtool:net starting (sysidtool) ]
[ network/ipfilter:default starting (IP Filter) ]
[ system/sysidtool:system starting (sysidtool) ]
[ milestone/sysconfig:default starting (Basic system configuration milestone) ]
[ system/system-log:default starting (system log) ]
[ system/utmp:default starting (utmpx monitoring) ]
[ system/sac:default starting (SAF service access controller) ]
[ system/console-login:default starting (Console login) ]
[ network/ssh:default starting (SSH server) ]

openindiana console login:

root@uf8bc129adebf5416c5a8:/home/local/ANT/mjanthon#
root@uf8bc129adebf5416c5a8:/home/local/ANT/mjanthon#
root@uf8bc129adebf5416c5a8:/home/local/ANT/mjanthon#
root@uf8bc129adebf5416c5a8:/home/local/ANT/mjanthon# cd
root@uf8bc129adebf5416c5a8:~# screen -r
There is no screen to be resumed.
root@uf8bc129adebf5416c5a8:~# screen

pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm@0
pseudo-device: llc10
llc10 is /pseudo/llc1@0
pseudo-device: lofi0
lofi0 is /pseudo/lofi@0
pseudo-device: power0
power0 is /pseudo/power@0
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk@1024
pseudo-device: ucode0
ucode0 is /pseudo/ucode@0
pseudo-device: pool0
pool0 is /pseudo/pool@0
pseudo-device: fssnap0
fssnap0 is /pseudo/fssnap@0
pseudo-device: nsmb0
nsmb0 is /pseudo/nsmb@0
pseudo-device: pm0
pm0 is /pseudo/pm@0
pseudo-device: bpf0
bpf0 is /pseudo/bpf@0
[ system/device/fc-fabric:default starting (Solaris FC fabric device configuration.) ]
[ milestone/devices:default starting (device configuration milestone) ]
[ milestone/name-services:default starting (name services milestone) ]
[ system/manifest-import:default starting (service manifest import) ]
[ application/print/service-selector:default starting (print service selector) ]
[ network/location:default starting (network interface configuration) ]
[ system/coreadm:default starting (system-wide core file configuration) ]
[ milestone/network:default starting (Network milestone) ]
[ milestone/single-user:default starting (single-user milestone) ]
[ system/fcoe_initiator:default starting (fcoe initiator service) ]
[ network/routing-setup:default starting (Initial routing-related configuration.) ]
[ network/initial:default starting (initial network services) ]
[ system/filesystem/local:default starting (local file system mounts) ]
[ network/service:default starting (layered network services) ]
[ network/netmask:default starting (Reset the netmask and broadcast address) ]
[ system/boot-archive-update:default starting (update boot archive if necessary) ]
[ system/dumpadm:default starting (system crash dump configuration) ]
[ network/shares/group:zfs starting (Share Group) ]
[ network/shares/group:default starting (Share Group) ]
[ system/cron:default starting (clock daemon (cron)) ]
[ network/physical:nwam starting (physical network interface autoconfiguration) ]
[ network/dns/client:default starting (DNS resolver) ]
[ system/identity:domain starting (system identity (domainname)) ]
[ system/sysidtool:net starting (sysidtool) ]
[ network/ipfilter:default starting (IP Filter) ]
[ system/sysidtool:system starting (sysidtool) ]
[ milestone/sysconfig:default starting (Basic system configuration milestone) ]
[ system/system-log:default starting (system log) ]
[ system/utmp:default starting (utmpx monitoring) ]
[ system/sac:default starting (SAF service access controller) ]
[ system/console-login:default starting (Console login) ]
[ network/ssh:default starting (SSH server) ]

openindiana console login:

===
xen configuration file:

name = "hvm"
vcpus = 1
memory = "1700"
builder='hvm'
kernel = "/usr/lib/xen/boot/hvmloader"
bootloader = "/usr/lib/xen/bin/pygrub"
xen_platform_pci=1
disk = [ 'file:/root/openindiana/openindiana-working.img,hda,w', 'file:/root/openindiana/disk2.img,xvdb,w', 'file:/root/oi-dev-151a8-text-x86.iso,hdc:cdrom,r' ]
vif = ['bridge=xenbr0']
boot="cd"
serial="pty"
vnc=1
vnclisten="0.0.0.0"
vncdisplay=1
apic=0 ===