illumos gate

While working on adding SMB2 support to the SMB client, I found the existing design to be somewhat inconvenient. This issue tracks a minor redesign of SMB client connection setup. this is a separate issue from "add SMB2" (#9735) because it's testable independently from that work, and changes in small, testable chunks are almost always better than larger chunks.

The current SMB client design implements connection setup doing as much as possible in user space. That's normally (in general) a good approach, but in this design it causes some code duplication because it implies dealing with SMB messages both in the user level library and in the kernel. That code duplication wasn't excessive with just SMB1 messages, but I found that once I started adding SMB2 support, it became apparent that it would be easier to change that design so that SMB messages (SMB1 or SMB2) are handled only in the driver.

In the new design, connection setup is still driven by the user-level library (instantiated in the smbiod service program), but the actual SMB message handling is all in the kernel driver. The user level code creates an IO Deamon (IOD) thread which calls a series of ioctl functions to sequence the driver through SMB (or SMB2) negotiate, SMB (or SMB2) session setup. After that point, the design is the same as before. If the session setup is successful, the IOD thread enters the driver via a "work" ioctl and stays there as long as the connection is usable and needed.

Also unchanged is where the authentication objects are built and parsed, with only the transport of those objects changed in this design. Previously, the library code extracted an incoming authentication object (SPNEGO) from an SMB reply, built an outgoing object, wrapped it in an SMB session setup request and sent it out. In the new design, the driver extracts the authentication object from the SMB (or SMB2) message and copies it out into a buffer provided by the user-level library in the ioctl call. The library code builds a response authentication object and passes that buffer to the next ioctl call, which then (in the driver) wraps that auth. object in an SMB (or SMB2) session setup request and sends it out.

One simplification was made in SMB1 for this design: In SMB1 we used to allow either "extended security" (SPNEGO) or old-style (non-extended) authentication. We could support SMB1 non-extended in this design, but that would add a bunch of complication and it's not needed to talk to any reasonably modern server, so this implements only "extended-security" in SMB1. (In SMB2 there is only extended security, so this is never an issue there.)