illumos gate

Analysis (and fix) from Kevin Crowe

With a debug kernel this is panicing in vnode.c in an assertion on line 1494: assertion failed in_crit == 0
That's in the function vn_createat()
Specifically, in this section of code that would bother the code review nazi's:

        /*
         * In general we want to generate EROFS if the file system is
         * readonly.  However, POSIX (IEEE Std. 1003.1) section 5.3.1
         * documents the open system call, and it says that O_CREAT has no
         * effect if the file already exists.  Bug 1119649 states
         * that open(path, O_CREAT, ...) fails when attempting to open an
         * existing file on a read only file system.  Thus, the first part
         * of the following if statement has 3 checks:
         *      if the file exists &&
         *              it is being open with write access &&
         *              the file system is read only
         *      then generate EROFS
         */
        if ((*vpp != NULL && (mode & VWRITE) && ISROFILE(*vpp)) ||
            (*vpp == NULL && dvp->v_vfsp->vfs_flag & VFS_RDONLY)) {
                if (*vpp)
                        VN_RELE(*vpp);
                error = EROFS;
        } else if (excl == NONEXCL && *vpp != NULL) {

it's the "else if" branch. Interesting, there's spots for it to bail if nbmand locks are taken:

                /*
                 * File already exists.  If a mandatory lock has been
                 * applied, return error.
                 */
                vp = *vpp;
                if (VOP_REALVP(vp, &rvp, NULL) != 0)
                        rvp = vp;
                if ((vap->va_mask & AT_SIZE) && nbl_need_check(vp)) {
                        nbl_start_crit(vp, RW_READER);
                        in_crit = 1;                    <<<<<  not sure if in_crit is getting set here or elsewhere yet....
                }
                if (rvp->v_filocks != NULL || rvp->v_shrlocks != NULL) {
                        vattr.va_mask = AT_MODE|AT_SIZE;
                        if (error = VOP_GETATTR(vp, &vattr, 0, CRED(), NULL)) {
                                goto out;
                        }
                        if (MANDLOCK(vp, vattr.va_mode)) {
                                error = EAGAIN;
                                goto out;
                        }
                        /*
                         * File cannot be truncated if non-blocking mandatory
                         * locks are currently on the file.
                         */
                        if ((vap->va_mask & AT_SIZE) && in_crit) {
                                u_offset_t offset;
                                ssize_t length;

                                offset = vap->va_size > vattr.va_size ?
                                    vattr.va_size : vap->va_size;
                                length = vap->va_size > vattr.va_size ?
                                    vap->va_size - vattr.va_size :
                                    vattr.va_size - vap->va_size;
                                if (nbl_conflict(vp, NBL_WRITE, offset,
                                    length, 0, NULL)) {
                                        error = EACCES;
                                        goto out;
                                }
                        }
                }

Regardless it hits the 2nd assert in this conditional block:

                /*
                 * If the file is the root of a VFS, we've crossed a
                 * mount point and the "containing" directory that we
                 * acquired above (dvp) is irrelevant because it's in
                 * a different file system.  We apply VOP_CREATE to the
                 * target itself instead of to the containing directory
                 * and supply a null path name to indicate (conventionally)
                 * the node itself as the "component" of interest.
                 *
                 * The intercession of the file system is necessary to
                 * ensure that the appropriate permission checks are
                 * done.
                 */
                if (vp->v_flag & VROOT) {
                        ASSERT(why != CRMKDIR);
                        error = VOP_CREATE(vp, "", vap, excl, mode, vpp,
                            CRED(), flag, NULL, NULL);
                        /*
                         * If the create succeeded, it will have created
                         * a new reference to the vnode.  Give up the
                         * original reference.  The assertion should not
                         * get triggered because NBMAND locks only apply to
                         * VREG files.  And if in_crit is non-zero for some
                         * reason, detect that here, rather than when we
                         * deference a null vp.
                         */
                        ASSERT(in_crit == 0);
                        VN_RELE(vp);
                        vp = NULL;
                        goto out;
                }

More from Kevin:

OK, the non-debug panic makes sense too.
In vn_createat() it's in this code where it tried to create the file (VOP_CREATE) and because of earlier code in vn_createat() in_crit is set to 1.
When it tries to create the file - which just so happens to be the mount point of the zfs fs an error is returned. Ultimately the vnode is
released (VN_RELE) and vp gets set to NULL and it does a 'goto out':

                if (vp->v_flag & VROOT) {
                        ASSERT(why != CRMKDIR);
                        error = VOP_CREATE(vp, "", vap, excl, mode, vpp,
                            CRED(), flag, NULL, NULL);
                        /*
                         * If the create succeeded, it will have created
                         * a new reference to the vnode.  Give up the
                         * original reference.  The assertion should not
                         * get triggered because NBMAND locks only apply to
                         * VREG files.  And if in_crit is non-zero for some
                         * reason, detect that here, rather than when we
                         * deference a null vp.
                         */
                        ASSERT(in_crit == 0);
                        VN_RELE(vp);
                        vp = NULL;
                        goto out;
                }

the code at the out label does this, the pc is just past where things went south, specifically, it took the if(in_crit) condition and called nbl_end_crit(vp):

out:
        if (auditing)
                audit_vncreate_finish(*vpp, error);
        if (in_crit) {
                nbl_end_crit(vp);    <<<<<<<<<<<<<<<<<<<<<  (panic)
                in_crit = 0;
        }
        if (vp != NULL) {
                VN_RELE(vp);
                vp = NULL;
        }

and in nbl_end_crit() the code tries to exit the rw lock and we panic with addr = 60 (see description field of this bug), offset 60 in a vnode_t is the rwlock: v_nbllock
So, both the debug & non-debug panic's make sense.

Virtually the same code path is executed with a zfs fs that does not have nbmand set on it, the only difference is that in_crit is not set to 1, and at the out label, the code does not execute the nbl_end_crit(vp) call that panics things due to vp being NULL.