illumos gate

I ran into an issue with COMSTAR's UNMAP handling. COMSTAR uses zvol DKIOCFREE for UNMAP and its involvement ends there.

A thread that made it to the DKIOCFREE case creates a transaction and then assigns it to the next available transaction group. Then, it updates the zil (not interesting) and calls dmu_free_long_range. That's just a wrapper around dmu_free_long_range_impl which more or less does:

while(...) {
dmu_tx_create(...);
dmu_tx_assign(..., TXG_WAIT);
<free a bit of the range>
dmu_tx_commit(...);
}

Now, the thread that's executing the UNMAP is busy creating & committing these transactions. Then, at some point, txg_quiesce_thread runs and tries quiesce the txg. It bumps the tx_open_txg, and then waits for every transaction in the quiescing txg to commit. This forces the thread executing the UNMAP to hang in dmu_tx_assign because it is waiting for the open txg to be ready. Since that thread went to sleep, there's no way it can return from dmu_free_long_range_impl and do the dmu_tx_commit on the top-level tx.

Matthew Ahrens confirmed on the mailing list that nested DMU transactions are not supposed to happen.

Related issue: #1845