implement openpty(3C), forkpty(3C), and login_tty(3C)
Other systems have openpty() and forkpty() functions to make dealing with pty's considerably easier and more foolproof than they are for us.
The way it must be done on illumos is basically to implement whichever of those two routines you want, completely, and yourself. This should not be.
A correct implementation of each (and anything related I don't know of) should be in libc.