illumos gate

For users using 512e disks, the sd driver maintains a kmem cache for blocks that might be read-modify-write operations.

During close(), the sd driver attempts to possibly destroy that cache, but it checks for the list of outstanding commands, which may include commands that have nothing to do with actual I/O. (For example pending queries to drive lock states, inquiries, and so forth.)

During that close, there is an attempt to reap the kmem_cache. Which is a nice optimization. But if the driver has commands outstanding (which might not be linked to active file descriptors), it complains that it can't "clean up memory". The thing is, this circumstance is utterly benign, and depending on the state of other accesses to the disk, might be completely normal.

This message is confusing, and distressing. And ultimately, what it is complaining about is merely a failure to reap memory that will later be cleaned up if the driver detaches. (But in all likelihood, will simply be reused the next time the driver is opened.)

We've seen this message when doing, for example, zfs reopen. Or during early boot when our logic which probes drives (without actually keeping the drive open) opens and then exits.

As there is nothing really problematic here, the best thing to do is to simply remove the message. It brings no value, and raises concerns that are likely irrelevant.