illumos gate

It turns out that event ports are slightly broken for FIFOs.

Specifically, it is possible for port_getn to return an event on a FIFO fd (registered via PORT_SOURCE_FD and POLLIN) even though there is nothing available for reading.

Attached is a minimal test case.

If you run it with stdin being a file or a tty, port_getn blocks until bytes are available for reading:

$ ./a.out
port = 3, errno = 0
ret = 0, errno = 0
ret = -1, errno = 48
ucred = 803bc88
<waiting>

If, however, stdin is a pipe, port_getn returns immediately because the fd is marked as readable by the getpeerucred call.

$ cat | ./a.out
port = 3, errno = 0
ret = 0, errno = 0
ret = -1, errno = 48
ucred = 803e258
ret = 0, errno = 48
ev.portev_source = 4
ev.portev_events = 1
ev.portev_object = 0
ev.portev_user = 0

Note that pfiles(1) injects a getpeerucred call into the target with the help of the agent lwp.

The getpeerucred call ends up issuing an ioctl on the FIFO. Since the _I_GETPEERCRED command is not handled by fastpath mode (http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/fifofs/fifovnops.c#1160), the FIFO is converted to a streams mode FIFO. During this conversion, all poll and event ports waiters are awoken to cause them to re-register with the newly converted FIFO (http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/fifofs/fifosubr.c#1108). The kernel stack of a thread waking up all waiters because of the FIFO conversion looks like this:

              genunix`port_send_event+0x131
              genunix`pollwakeup+0x86
              genunix`strpollwakeup+0x20
              fifofs`fifo_fastturnoff+0xae
              fifofs`fifo_fastoff+0x92
              fifofs`fifo_fastioctl+0x19d
              fifofs`fifo_ioctl+0x3d
              genunix`fop_ioctl+0x55
              genunix`getpeerucred+0xff
              genunix`ucredsys+0x52
              genunix`ucredsys32+0x1b
              unix`sys_syscall32+0xff

This spurious wakeup can cause userspace applications to end up blocking on a subsequent read/write because they were notified that the file descriptor was ready. (This problem has been spotted in the wild: https://github.com/PowerDNS/pdns/issues/2925) The manpages for event ports don't specify whether spurious events are something an application needs to guard against.

As far as I can tell, there appears to be no good reason for fastpath FIFOs to not handle _I_GETPEERCRED ioctls directly. Attached is a quick patch which appears to work with my quick test.