illumos gate

[Investigation]
There are 30 threads which in general making the following steps to handle incoming IO requests:
1. Get the event from the port (port_get()).
2. If it is a read or write request, then use aio_read() or aio_write() calls to handle it.
When AIO request is competed the port is notified, as a result the aio_port_callback() routine is called, the backtrace is something like this:
port_getn()
port_copy_event()
aio_port_callback()

This aio_port_callback() (http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/os/aio.c#3806) routine performs some clean up changes and then calls aio_copyout_result_port() to store the error and return value. When the aio_copyout_result_port() (http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/os/aio_subr.c#1138) routine is executed. It was found out that sometimes the error and return value is really 0 in case of stress test:
errno = 0;
retval = iov->iov_len - bp->b_resid;
Both variables iov->iov_len and bp->b_resid are set to 0. That is correct for b_resid, but not for the iov_len field.
The issue for it is the race between the threads:
1. THREAD1:
1.1. AIO request completed
1.2. Notify thread using port
1.3. Call the aio_port_callback
1.4. mutex_enter(&aiop->aio_mutex);
1.5. Make some cleaning. Call the aio_req_free_port() routine (http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/os/aio_subr.c#573). Which performs the following action:
aiop->aio_free = reqp;
1.6. mutex_exit(&aiop->aio_mutex);
1.7. Call the aio_copyout_result_port() routine to store error/return values from the reqp variable.

2. THREAD2:
2.1. Get a new AIO request.
2.2. Call aio_read or aio_write.
2.3. Waiting on aiop->aio_mutex.
2.4. As soon as it is free get it. Sometimes it is the mutex before calling the aio_req_alloc() routine (http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/os/aio.c#2493). The following lines of code are executed:
if ((reqp = aiop->aio_free) != NULL) {
aiop->aio_free = reqp->aio_req_next;
bzero(reqp, sizeof (*reqp));
} else {
Since aio_free is not NULL (see p1.5) the bzero() call will be made.

So here we got the race condition between storing values from the reqp structure (p1.7) and set to 0 by calling bzero() (p2.5).

[Design]
The idea of the fix is pretty straight forward:
Call the aio_copyout_result_port() routine before the aio_req_free_port() call.

The aio_copyout_result_port() function is called in one place which we are going to fix.
The aio_copyout_result() routine is called from several places (aio.c and aio_subr.c), but it is always called before the aio_req_free() call. So only one place should be fixed.

New rb request is created for review: https://www.illumos.org/rb/r/357/