Bug #12306


XPG4v2 slave pty behaviour should generally be disabled

Added by Andy Fiddaman over 2 years ago. Updated over 2 years ago.

lib - userland libraries
Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:
External Bug:


As part of the work switching OmniOS userland to build with gcc9, and also while working on gcc9 patches for a proposed compiler to build a future illumos-gate, we have hit several problems around the fact that the behaviour of slave PTYs changes as soon as either xpg4-values.o or xpg6-values.o is linked into a program.

The default behaviour in gcc8 and gcc9 is to link in one of these at all times - xpg4 when building with the C90 standard, and xpg6 otherwise.

Both of these values files set the global __xpg4 variable to 1, which modifies the behaviour of libc.

In particular, this causes the following additional actions when a slave PTY is opened through the open(2) call:
  • The ptem, ldterm and ttcompat modules are automatically pushed (Note 1);
  • A flag is set in ptem that slightly modifies the behaviour. In particular, this causes empty mblks to be sent up the stream.

The second action means that a read(2) call can legitimately return 0. Most applications (including some in gate like zlogin) treat this as an EOF marker and fail in some way.

I tested this using this simple program:

#include <unistd.h>

int main() {
    write(1, NULL, 0);
    return 0;

When accessing a system via an OpenSSH daemon which has xpgX-values linked, running this immediately kills the terminal and truss shows read() returning 0.
Rebuilding OpenSSH without xpgX-values does not exhibit this problem.

Following discussion on IRC, it appears that this modified behaviour was added to Solaris specifically to resolve an XPG test suite failure (since Solaris/illumos PTYs are STREAMs based, there are extra requirements in the standard; I don't have the detail).

Since most applications do not expect this (and often break), the proposal is to enable this only if an application is built in strict XPG4v2 compliance mode -i.e.
if defined(_XPG4_2) && !defined(__EXTENSIONS__)

Note 1: This often caused duplicate modules to be pushed onto slave PTY streams but was resolved in 9042 multiples of tty streams modules cause weirdness


itest.c (2.18 KB) itest.c pty test program Andy Fiddaman, 2020-02-16 11:23 AM

Related issues

Related to illumos gate - Bug #9042: multiples of tty streams modules cause weirdnessClosed2018-02-05

Related to illumos gate - Bug #12057: Writing part of the string to stderr makes zlogin exitClosed

Related to illumos gate - Bug #12779: mdb str_flags is missing STRXPG4TTYClosedAndy Fiddaman


Also available in: Atom PDF