illumos gate

Replicating downstream bug OS-6116

Several customers, internal and otherwise, have reported seeing nginx periodically spew errors related to epoll_wait yielding EPERM. This is a little bit odd, as there isn't much in the way of access checking in the guts of epoll/devpoll itself. That logic does, however, allow errors from VOP_POLL handlers to trickle up. With the merging of OS-5941, attempts to epoll on regular files and directories will result in EPERM.

Since epoll_wait = EPERM is the expectation if a normal file end up in the set, why then is nginx stumbling over the problem. Certainly it wouldn't add a plain fd intentionally, only to complain about it later. Another possibility does exist in this case: An active fd was closed, without first being removed from the set, followed by it being quickly reassigned when nginx went to open a regular file. This touches on one of the wrinkles in the core design of epoll: it expects to act on the underlying struct file (our vnode_t would likely be the closes analog) rather than the fd itself. This places a premium in detecting when the resource backing of an fd changes.