illumos gate

Running ztest I hit this failure:

$c

libc.so.1`_lwp_kill+0x15(ea565680, ea565680, 6d, ea5656e8, fea651d0, 0)
libc.so.1`_assfail+0x26c(ea5659e8, 806733f, 4d3, ea5659fe, 1c, 0)
libc.so.1`assfail3+0x115(8067363, c, 0, 8067360, 1c, 0)
ztest_tx_assign+0x81(b6c8b00, 1, 0, 8066b42)
ztest_replay_remove+0x173(ea566588, 8729e48, 0, 35000200, 0, 0)
ztest_remove+0x9f(ea566588, ea565f50, 4, 3)
ztest_object_init+0x78(ea566588, ea565f50, 4e0, 1)
ztest_dmu_object_alloc_free+0x71(ea566588, 10)
ztest_dmu_objset_create_destroy+0x224(80ca2d0, 10, 0, 805d2c3, 1ec15983, 0)
ztest_execute+0x89(a, 807c720, 10, 0)
ztest_thread+0xea(10, 0, 0, 0)
libc.so.1`_thrp_setup+0x88(f0981a40)
libc.so.1`_lwp_start(f0981a40, 0, 0, 0, 0, 0)

*panicstr/s

0xea565680: assertion failed for thread 0xf0981a40, thread-id 567: error
28 (0xc 0x1c), file ../ztest.c, line 1235
The arc is returning back ENOMEM here:
arc_tempreserve_space(uint64_t reserve, uint64_t txg) {
int error;
uint64_t anon_size;

if (reserve > arc_c/4 && !arc_no_grow)
                arc_c = MIN(arc_c_max, reserve * 4);
        if (reserve > arc_c)
                return (SET_ERROR(ENOMEM));
Walking the tx_holds we see that the txh_memory_tohold is larger than arc_c:

b6c8b00::print dmu_tx_t tx_holds | ::walk list | ::print -d dmu_tx_hold_t txh_

memory_tohold
txh_memory_tohold = 0
txh_memory_tohold = 0t224165888

::arc ! grep ^c

c = 191 MB
c_min = 191 MB
c_max = 1535 MB
We're trying to remove an object that has a 1MB block size:

ea565f50::print ztest_od_t

{
od_dir = 0x1
od_object = 0x2
od_type = 0t26 (DMU_OT_UINT64_OTHER)
od_crtype = 0t26 (DMU_OT_UINT64_OTHER)
od_blocksize = 0x100000
od_crblocksize = 0x2000
od_gen = 0x1
od_crgen = 0
od_name = [ "ztest_dmu_object_alloc_free(16)[0]" ]
}

Fix is to not let arc_c get so tiny.