Checking vmem limit of wrong process for door_calls
As part for the Solaris door_call, the client/caller thread specifies a buffer to hold the results. If the buffer is not big enough to hold the results, the server/callee thread creates a new buffer in the caller's address space. See the door_call manual page for more specifics.
The problem we discovered was when the caller (in our case a 64-bit JVM with more than 4G vmem allocated) made a door_call to a callee (32-bit ipmgmtd); a check was made against the virtual memory size limit of the callee even though the memory is being allocated in the caller's address space. This triggered an error which effectively prevents our 64-bit JVM process from using more than 3GB for the Java heap. Although in this case, the issue happened between 64-bit and 32-bit processes, I believe it would also occur if both are 32-bit, but the callee has a lower "ulimit -v".
It appears this can be fixed by using the process that owns the address space we are allocating memory in, as->a_proc, instead of the current process. All p is used for in as_map_loced is process limitation checks (p_vmem_ctl and p_rctls) so it appears safe and correct to instead use the address space's process.
Updated by Electric Monk over 5 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
commit 6430daec31473902a374dd7a244e1f67ce33191d Author: Brandon Baker <firstname.lastname@example.org> Date: 2016-09-07T03:15:32.000Z 7268 Checking vmem limit of wrong process for door_calls Reviewed by: Dan Kimmel <email@example.com> Reviewed by: George Wilson <firstname.lastname@example.org> Reviewed by: Sebastien Roy <email@example.com> Reviewed by: Robert Mustacchi <firstname.lastname@example.org> Approved by: Dan McDonald <email@example.com>