illumos gate

Environment:

• XenServer 5.6.100 SP2 (Xen 3.4.2)
• PyGrub & libfsimage from Xen 4.1 (ZFS_VERSION bumped to 28)
• Host: Xeon X3360, 8 GB RAM, 250 GB HDD
• DomU: 1 CPU, 1 GB RAM, 8 GB HDD

How to repeat:

• Install from server ISO (http://dlc-int.openindiana.org/151/oi-dev-151-text-x86-20110526-1.iso)
• Reboot (to illumos d89753efabc8, May 2011)
• pkg image-update
• Reboot (to illumos a4e1558c0599, July 2011)

I am using a somewhat modified Xen toolchain consisting of:
Custom installation template for OI (32-/64-bit).
Modified eliloader (to install with DomU kernel from ISO).
Modified pygrub (to load kernel and ramdisk from DomU rather than from Dom0).
This setup has been working fine with OI since several months now.

I have verified that the Dom0 loads the correct files from the DomU image.
The md5sum's match what is inside the image, so I am pretty sure the toolchain
works as expected.

I have tried various incremental pkg update's until the reboot fails and found
the offending package to be: pkg:/system/kernel.

Further observations:

/platform/i86xpv/kernel/amd64/unix

has the same md5sum in both boot environments.

/platform/i86pc/amd64/boot_archive

(in the new be) contains the following changed files:

kernel/amd64/genunix
kernel/crypto/amd64/arcfour
kernel/drv/amd64/tl
kernel/misc/amd64/bignum

Booting from ISO and rebuilding the boot_archive seems to have no effect.
The crash occurs in both 32 and 64-bit mode.

Let me know, if you need any further information.

Stacktrace:

v3.4.2 chgset 'trunk.455c6392747f.2bcefc51add5 (3.4.2 5.6.100.704.20055)'
OpenIndiana Build oi_151 64-bit (illumos a4e1558c0599)
SunOS Release 5.11 - Copyright 1983-2010 Oracle and/or its affiliates.
All rights reserved. Use is subject to license terms.

panic[cpu0]/thread=fffffffffbc609e0: BAD TRAP: type=e (#pf Page fault) rp=fffffffffbcb42d0 addr=fffffe000406d838

#pf Page fault
Bad kernel fault at addr=0xfffffe000406d838
pid=0, pc=0xfffffffffb8c03f3, sp=0xfffffffffbcb43c8, eflags=0x10246
cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 2660<vmxe,xmme,fxsr,mce,pae>
cr2: fffffe000406d838
        rdi: fffffe000406d838 rsi:                0 rdx:        2193bd273
        rcx:                3  r8:                0  r9: ffffff00c6707000
        rax:                0 rbx:        2193bd273 rbp: fffffffffbcb4460
        r10: ffffff00c554b000 r11: ffffff0000000000 r12:                0
        r13:                1 r14: fffffe000406d838 r15:        2193bd273
        fsb:        200000000 gsb: fffffffffbc61c20  ds:                0
         es:                0  fs:                0  gs:                0
        trp:                e err:                3 rip: fffffffffb8c03f3
         cs:             e030 rfl:            10246 rsp: fffffffffbcb43c8
         ss:             e02b

Warning - stack not written to the dump buffer
fffffffffbcb41b0 unix:die+10f ()
fffffffffbcb42c0 unix:trap+17af ()
fffffffffbcb42d0 unix:cmntrap+12b ()
fffffffffbcb4460 unix:atomic_cas_64+3 ()
fffffffffbcb4500 unix:hati_pte_map+18c ()
fffffffffbcb4580 unix:hati_load_common+15d ()
fffffffffbcb4640 unix:hat_devload+198 ()
fffffffffbcb46a0 unix:kmem_override_cache_attrs+be ()
fffffffffbcb4720 unix:i_ddi_mem_alloc+1a4 ()
fffffffffbcb47e0 genunix:ddi_dma_mem_alloc+1a3 ()
fffffffffbcb48a0 unix:xvdi_alloc_ring+d4 ()
fffffffffbcb48f0 xdf:xdf_setstate_init+f5 ()
fffffffffbcb4980 xdf:xdf_attach+387 ()
fffffffffbcb49e0 genunix:devi_attach+80 ()
fffffffffbcb4a10 genunix:attach_node+95 ()
fffffffffbcb4a50 genunix:i_ndi_config_node+c4 ()
fffffffffbcb4a70 genunix:i_ddi_attachchild+40 ()
fffffffffbcb4ab0 genunix:devi_attach_node+ac ()
fffffffffbcb4b10 genunix:ndi_devi_online+c7 ()
fffffffffbcb4bd0 unix:xvdi_create_dev+258 ()
fffffffffbcb4c40 xpvd:xpvd_bus_config+91 ()
fffffffffbcb4cc0 genunix:ndi_devi_config_one+bd ()
fffffffffbcb4d70 genunix:resolve_pathname+19c ()
fffffffffbcb4dd0 genunix:ldi_vp_from_name+ef ()
fffffffffbcb4e40 genunix:ldi_open_by_name+57 ()
fffffffffbcb4ed0 zfs:vdev_disk_read_rootlabel+a0 ()
fffffffffbcb4f20 zfs:spa_generate_rootconf+26 ()
fffffffffbcb4f90 zfs:spa_import_rootpool+31 ()
fffffffffbcb4ff0 zfs:zfs_mountroot+eb ()
fffffffffbcb5010 genunix:fsop_mountroot+1a ()
fffffffffbcb5040 genunix:rootconf+133 ()
fffffffffbcb5080 genunix:vfs_mountroot+63 ()
fffffffffbcb50b0 genunix:main+136 ()
fffffffffbcb50c0 unix:_locore_start+7e ()