port_fop_femuninstall is too loose with its locks
Thanks to folks at Mathworks for altering us to this and their help in root causing the issue. They following is what they provided:
list_remove+0x1b(ffffff49a25c7a50, ffffff443c6c3740) port_fop_femuninstall+0xa3(ffffff4447c6f500) port_remove_fop+0xca() port_dissociate_fop+0xa0(ffffff83fb8ba880, fffffd7fffdffbf0) portfs+0x2d1(2, 3, 7, fffffd7fffdffbf0, 0, 0) sys_syscall+0x17a()
Note: Special thanks to Ira Cooper and Youzhong Yang for identifying, describing, isolating and reproducing this!
Given "https://github.com/joyent/illumos-joyent/blob/master/usr/src/uts/common/fs/portfs/port_fop.c", Ira had a theory that the mutex released at line 582 ("mutex_exit(&pvp->pvp_mutex");), is allowing another thread to carry out the same list removal operation for the same "pvp" as found on line 599 ("list_remove(&pvfsp->pvfs_pvplist, pvp);) - It appears this is correct.