illumos gate

As part for the Solaris door_call, the client/caller thread specifies a buffer to hold the results. If the buffer is not big enough to hold the results, the server/callee thread creates a new buffer in the caller's address space. See the door_call manual page for more specifics.

The problem we discovered was when the caller (in our case a 64-bit JVM with more than 4G vmem allocated) made a door_call to a callee (32-bit ipmgmtd); a check was made against the virtual memory size limit of the callee even though the memory is being allocated in the caller's address space. This triggered an error which effectively prevents our 64-bit JVM process from using more than 3GB for the Java heap. Although in this case, the issue happened between 64-bit and 32-bit processes, I believe it would also occur if both are 32-bit, but the callee has a lower "ulimit -v".

It appears this can be fixed by using the process that owns the address space we are allocating memory in, as->a_proc, instead of the current process. All p is used for in as_map_loced is process limitation checks (p_vmem_ctl and p_rctls) so it appears safe and correct to instead use the address space's process.