illumos gate

Matt's design write-up:

Auditing in Illumos: How it Works

(fun fact: FreeBSD and Mac OS X have an implementation of this called "OpenBSM"!)
At startup, main calls audit_init(). If c2audit hasn't been specifically excluded in /etc/system, audit_init() initializes the auditing subsystem. This includes adding a per-thread auditing structure to all existing threads, and initializing "audit_active", so that all future threads include this auditing structure. At this point, no auditing configuration is in use.
When auditd() starts up, it uses the auditon() SETCOND ioctl to turn on auditing in the kernel. This tells the kernel to begin auditing based on its (not-yet-set) configuration. The kernel then generates a 'SYSTEMBOOT' audit record. (This appears to be a relic of when auditing couldn't be reconfigured without a reboot.) auditd then reads in the audit configuration from SMF, and the event-to-class mappings from /etc/security/audit_event.txt, and passes the relevant configuration to the kernel using more auditon() ioctls.
auditd() then initializes and configures the auditd_plugins (loaded from SMF). From then on, auditd()'s job is to take audit records passed to it by the kernel (via doorcall) and enqueue it onto each active plugin's queue. Each plugin has its own thread that dequeues these records for processing by the plugin.
audit records can originate either from user space via the audit() syscall, or from the kernel, either due to syscall auditing or a 'non-attributable' event (e.g. SYSTEMBOOT, PROMENTER/PROMEXIT). Audit events are generated either directly due to the actions of a user, known as the 'process model', or on behalf of a user, known as the 'session model'. All auditing begins as a session model: a program requests (using libbsm) that it be granted an auditing session handle. It then initializes that handle with the necessary information to audit that user's actions, including the user-specific audit mask and its auditing IDs. The program can then generate audit events on that handle. As an example, SMBD uses this to generate login/logout audit events for SMB users (see smbd_user_auth_logon()). A session can then be converted to the 'process model' via libbsm. libbsm uses another syscall to copy the auditing information from the session handle into the user process's credential. When that process performs an auditable syscall, the kernel uses the information stored in the process credential to generate an audit record. Auditable syscalls first get redirected to c2audit for initialization, including initializing the per-thread audit structure. Syscalls then store auditable information in that structure, and once the syscall is complete, c2audit determines whether or not to commit the record. PAM_unix uses this model to propogate auditing configuration through all of a user's processes forked after login. The syscall hook then pushes the written record onto a queue, and the writer thread eventually dequeues it and passes it down via doorcall to auditd for processing.
How do we determine what audit records to generate?
Part of the information stored in the session handle and the user's credential is an 'audit mask'. This mask is two
bitfields of audit classes - one for success and one for failure - that are configured to be audited. This is a combination of the 'default' audit mask, configured through auditconfig, and a per-user mask stored as the 'audit_flags' 'user_attr' (i.e. usermod -K audit_flags:<flags>). auditd passes the 'default' audit mask to the kernel, and libbsm() combines the two in the user's audit mask. When an auditable event occurs, it is first mapped to its corresponding class mask (based on /etc/security/audit_events.txt), and then if that class mask intersects with the appropriate audit mask, a record is generated from that event.
How do SACLs fit into this?
SACL events are, in essence, a process model event originating from the kernel, bypassing all of the syscall machinery. All of the underlying mechanisms for generating audit records are sufficiently generic that the syscall hooks are just a wrapper for it all. We can make sufficiently generic alternatives to some of the more syscall-specific functions (like asking the question 'should we generate SACL audit records for this user?') with minor modifications to existing code.
The current design for SACL-based auditing looks like this:
1. SMBD creates an auditing session handle. This retrieves the audit mask, uid, and session ID. It then passes this information up to the kernel, along with the rest of the authentication information.
2. SMBSRV writes this information into the auditing structure of the user's credential.
3. If auditing is configured, When a user tries to perform certain SMB2 operations (namely, open and rename), SMBSRV initializes the per-thread auditing structure with a value indicating to the underlying filesystem that it should create an auditing mask.
4. When ZFS performs an access check, it first determines whether SACL auditing is configured. If it is, it calls a version of zfs_zaccess_aces_check that also processes 'audit' and 'alarm' type acceses into success and failure masks, then writes those masks into the per-thread audit structure. It then disables further auditing, in case multiple access checks are performed, in order to prevent overwriting the audit mask.
5. When the filesystem call returns, SMBSRV compares the request's DesiredAccess to the appropriate audit mask, depending on whether or not the access was succesful. If the masks intersect, SMBSRV provides sufficient information to c2audit (via 'audit_sacl'), including by providing the credential used, so that an record of the access may be generated.
You may wonder why SMB is in charge of determining whether or not to generate on audit record. Access on Windows is, unfortunately, not determined solely by the security descriptor of an object. Accesses can additionally be granted by the security descriptor on the parent directory (namely, Delete granted by DeleteChild on the parent, or ReadAttributes granted by ListDirectory on the parent), or by specific grants of privilege, or by a combination of privilege and specific options in the Open request. The end result is that a single protocol-level access check can comprise of several filesystem-level access checks, and those access checks can be for a subset of the access actually granted. As such, while there may be ways to propogate sufficent knowledge to the underlying filesystem to allow it to generate records, SMBSRV is ultimately in the best position to know which accesses ought to be audited and for which requests and filesystem calls.

Audit Configuration Notes (from Matt)

audit config notes:

auditconfig -setflags sa #set user default preselection flags to audit SACLs
auditconfig -setflags -sa #set user default preselection flags to audit only failure SACLs
auditconfig -setflags +sa #set user default preselection flags to audit only success SACLs
auditconfig -setkmask sa #set non-attributes selection flags of machine to audit SACLs. This is necessary to audit non-IDMU domain users (i.e. those with ephemeral IDs.)
auditconfig -setnaflags sa #set non-attributable audit flags non-persistently.
svc:/system/auditd
use "audit" to control, and "auditconfig" to configure
audit -n #tell auditd to start a new file
audit -s #tell auditd to reload and validate audit configuration. Also starts auditd.
audit -t #tell auditd to shutdown.
audit -v #validate audit configuration.
auditstat #see audit statistics

============================
I wrote up these instructions while (trying) to prepare a demo for auditing; they're probably a good place to start for investigating how to expose these commands.
auditconfig -setflags sa #set user default preselection flags to audit SACLs
auditconfig -setnaflags sa #domain users that don't get mapped to non-ephemeral UIDs are audited based on naflags.

1. Note: One can set the active non-attributable audit flags temporarily via 'auditconfig -setkmask sa'. This will not record the update in smf.
   

   audit -v #should be "configuration ok" 
   audit -s #load configuration and enable auditing.
   

   
   Then, audit specific files by setting 'audit type ACLS:
   

   chmod A+everyone@:rwxpdDaARWcCos:fdSF:audit /export/junk/audit #audit all successful (S) and failed (F) accesses from everyone, and inherit to files and directories.
   chmod A+<type>:<id>:rwpdD:S:audit <file> #audit successful Read, Write, Append, and Delete accesses from <id>
   audit -n #close the current audit file, and open a new one
   auditreduce <options> # select some subset of audit records. see manpage for record selection options.
   auditreduce | praudit <opts> # print audit records selected by auditreduce. use '-r' for raw, '-s' for 'short form', and -l for 'one line per record'. -x can be combined with any of these to print in XML.
   audit -n; auditreduce -D <suffix-or-path> # Delete completed audit records, storing the selected records in a new file. Note that 'notterminated' files are ignored by this, unless specified explicilty on the command line.
   auditconfig -setkaudit <ipv4 or ipv6> <ip address>
   auditconfig -setplugin <name> <(in)active> <attributes>
   auditconfig -setpolicy <policy flag> #auditconfig -lspolicy to see descriptions of each.
   some useful commands. see auditconfig(1M) for more info.
   auditconfig -getplugin #see the installed plugins and their configurations
   auditconfig -getpolicy #kernel audit policy. auditconfig -lspolicy to see descriptions of each.
   - notable: 'group' includes group info in audit records, 'perzone' allows configuring auditd per-zone, 'seq' adds a sequence number to records, 'zonename' includes zone name in records, 'trail' adds a trailer token to records (like header)
   auditconfig -getqctrl #get write buffer size, hiwater mark, lowater mark, and delay interval.
   auditconfig -getstat #get audit stats (see auditstat(1M))
   

   
   NOTE: SMBD also offers the ability to audit SMB logons; simply select the 'lo' (login/logoff) audit class, as well as 'sa' (auditconfig -setflags sa,lo). One can then corrolate who logged in with who accessed what files. Note that, unless the default event mappings are changed, several other logon events may be generated as well. Local accounts can be given user-specific audit preselection masks by setting the 'audit_flags' keyword in the user_attr database (usermod -K audit_flags=<always-audit-flags>:<never-audit-flags> ...)

This code has been out in a couple NexentaStor releases.

Here's a version of that work that looks pretty much ready to go upstream.
https://github.com/Nexenta/illumos-nexenta/compare/915fa9178dcd903daad5b65b3cd5a4926ab54f7e..61e6d5f3e0e866bc655de57993e56b3cfdf80e6e

commit 61e6d5f3e0e866bc655de57993e56b3cfdf80e6e
Author: Matt Barden <mbarden@tintri.com>
Date:   Fri Feb 21 12:09:39 2020 -0500

    FIR-291 Port File Access Auditing feature from NexentaStor
    (NFS changes for NFS auditing)

    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@tintri.com>
    Reviewed by: Suresh Jayaraman <sjayaraman@tintri.com>
    Reviewed by: Aditya Agnihotri <aagnihotri@tintri.com>

    and fix merge

42    3    usr/src/cmd/fs.d/nfs/mountd/nfsauth.c
92    21    usr/src/cmd/fs.d/nfs/mountd/nfsauth_xdr.c
7    1    usr/src/lib/libsecdb/common/chkauthattr.c
3    1    usr/src/tools/quick/make-gss
1    0    usr/src/tools/quick/make-nfs
3    2    usr/src/uts/common/Makefile.files
6    2    usr/src/uts/common/fs/nfs/nfs4_acl.c
382    0    usr/src/uts/common/fs/nfs/nfs4_audit.c
277    28    usr/src/uts/common/fs/nfs/nfs4_srv.c
25    2    usr/src/uts/common/fs/nfs/nfs4_srv_attr.c
17    3    usr/src/uts/common/fs/nfs/nfs4_srv_readdir.c
454    288    usr/src/uts/common/fs/nfs/nfs_auth.c
92    21    usr/src/uts/common/fs/nfs/nfs_auth_xdr.c
2    3    usr/src/uts/common/fs/nfs/nfs_export.c
1    1    usr/src/uts/common/fs/nfs/nfs_server.c
33    7    usr/src/uts/common/nfs/auth.h
2    94    usr/src/uts/common/nfs/export.h
2    0    usr/src/uts/common/nfs/nfs.h
17    2    usr/src/uts/common/nfs/nfs4.h
254    0    usr/src/uts/common/nfs/nfsauth_cache.h

commit 056da2c40fbe25bcaa3a2665d0501dd0cb30b8f1
Author: Matt Barden <matt.barden@nexenta.com>
Date:   Fri Jan 12 19:16:25 2018 -0500

    FIR-291 Port File Access Auditing feature from NexentaStor
    (SMB changes for SMB auditing)

    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@tintri.com>
    Reviewed by: Suresh Jayaraman <sjayaraman@tintri.com>
    Reviewed by: Aditya Agnihotri <aagnihotri@tintri.com>

6    1    usr/src/cmd/auditreduce/auditrt.h
3    3    usr/src/cmd/auditreduce/option.c
2    2    usr/src/cmd/auditreduce/token.c
2    1    usr/src/cmd/mdb/intel/amd64/smbsrv/Makefile
2    1    usr/src/cmd/mdb/sparc/v9/smbsrv/Makefile
3    0    usr/src/cmd/smbsrv/smbd/smbd_logon.c
58    1    usr/src/common/smbsrv/smb_token_xdr.c
2    1    usr/src/lib/smbsrv/libfksmbsrv/Makefile.com
63    0    usr/src/lib/smbsrv/libfksmbsrv/common/fksmb_audit.c
3    3    usr/src/lib/smbsrv/libfksmbsrv/common/fksmb_cred.c
1    0    usr/src/uts/common/Makefile.files
198    0    usr/src/uts/common/fs/smbsrv/smb_audit.c
1    1    usr/src/uts/common/fs/smbsrv/smb_authenticate.c
36    0    usr/src/uts/common/fs/smbsrv/smb_cmn_rename.c
12    3    usr/src/uts/common/fs/smbsrv/smb_common_open.c
19    1    usr/src/uts/common/fs/smbsrv/smb_cred.c
40    0    usr/src/uts/common/fs/smbsrv/smb_fsops.c
135    0    usr/src/uts/common/fs/smbsrv/smb_node.c
2    0    usr/src/uts/common/fs/smbsrv/smb_vops.c
12    1    usr/src/uts/common/smbsrv/smb_kproto.h
10    1    usr/src/uts/common/smbsrv/smb_token.h

commit f4169d9891aa184cd7e3ed30a103878bf8ad1629
Author: Matt Barden <mbarden@tintri.com>
Date:   Mon May 18 16:27:30 2020 -0400

    FIR-291 Port File Access Auditing feature from NexentaStor
    (ZFS changes for NFS auditing)

    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@tintri.com>
    Reviewed by: Suresh Jayaraman <sjayaraman@tintri.com>
    Reviewed by: Aditya Agnihotri <aagnihotri@tintri.com>

151    14    usr/src/uts/common/fs/zfs/zfs_acl.c
2    1    usr/src/uts/common/fs/zfs/zfs_ioctl.c
11    2    usr/src/uts/common/fs/zfs/zfs_replay.c

commit a822726b9350bc5e2daeb990fe8f43e2633dff6a
Author: Matt Barden <mbarden@tintri.com>
Date:   Mon May 18 16:29:58 2020 -0400

    FIR-291 Port File Access Auditing feature from NexentaStor
    (ZFS changes for SMB auditing)

    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@tintri.com>
    Reviewed by: Suresh Jayaraman <sjayaraman@tintri.com>
    Reviewed by: Aditya Agnihotri <aagnihotri@tintri.com>

1    0    usr/src/tools/quick/make-zfs
2    1    usr/src/uts/common/fs/zfs/sys/zfs_znode.h
182    25    usr/src/uts/common/fs/zfs/zfs_acl.c

commit db8e3b8b6a824179f568c4aad1f10feab76832d3
Author: Matt Barden <mbarden@tintri.com>
Date:   Mon May 18 16:36:35 2020 -0400

    FIR-291 Port File Access Auditing feature from NexentaStor
    (Audit changes for SMB auditing)

    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@tintri.com>
    Reviewed by: Suresh Jayaraman <sjayaraman@tintri.com>
    Reviewed by: Aditya Agnihotri <aagnihotri@tintri.com>

3    1    usr/src/cmd/auditreduce/auditrd.h
2    0    usr/src/cmd/auditreduce/auditrt.h
4    0    usr/src/cmd/auditreduce/main.c
29    1    usr/src/cmd/auditreduce/option.c
14    2    usr/src/cmd/auditreduce/proc.c
35    0    usr/src/cmd/auditreduce/token.c
3    1    usr/src/cmd/praudit/Makefile
93    33    usr/src/cmd/praudit/format.c
4    0    usr/src/cmd/praudit/praudit.h
38    0    usr/src/cmd/praudit/token.c
8    0    usr/src/cmd/praudit/toktable.c
2    0    usr/src/cmd/praudit/toktable.h
26    0    usr/src/lib/auditd_plugins/syslog/systoken.c
4    0    usr/src/lib/auditd_plugins/syslog/systoken.h
2    1    usr/src/lib/libfakekernel/Makefile.com
19    0    usr/src/lib/libfakekernel/common/audit.c
81    0    usr/src/lib/libfakekernel/common/c2/audit.h
67    0    usr/src/lib/libfakekernel/common/c2/audit_kernel.h
2    1    usr/src/lib/libfakekernel/common/mapfile-vers
4    1    usr/src/lib/libidmap/common/idmap.h
87    1    usr/src/lib/libidmap/common/idmap_api.c
2    0    usr/src/lib/libidmap/common/mapfile-vers
12    1    usr/src/man/man1m/auditreduce.1m
3    1    usr/src/man/man4/audit_class.4
2    1    usr/src/pkg/manifests/system-header.mf
282    0    usr/src/tools/quick/make-audit
3    3    usr/src/uts/common/c2/Makefile
47    0    usr/src/uts/common/c2/audit.c
14    23    usr/src/uts/common/c2/audit.h
42    0    usr/src/uts/common/c2/audit_kernel.h
4    0    usr/src/uts/common/c2/audit_record.h
28    0    usr/src/uts/common/c2/audit_start.c
77    0    usr/src/uts/common/c2/audit_token.c
56    0    usr/src/uts/common/c2/audit_types.h
3    0    usr/src/uts/intel/ia32/ml/modstubs.s

commit 9568ca09e09569687828422c2582426f0fadff44
Author: Matt Barden <mbarden@tintri.com>
Date:   Mon May 18 16:46:03 2020 -0400

    FIR-291 Port File Access Auditing feature from NexentaStor
    (vnode header changes)

    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@tintri.com>
    Reviewed by: Suresh Jayaraman <sjayaraman@tintri.com>
    Reviewed by: Aditya Agnihotri <aagnihotri@tintri.com>

3    1    usr/src/lib/smbclnt/libfknsmb/common/sys/vnode.h
3    0    usr/src/lib/smbsrv/libfksmbsrv/common/sys/vnode.h
3    0    usr/src/uts/common/sys/vnode.h

commit 0afa32307fc8bf268847bf9e3918a63609931f17
Author: Matt Barden <mbarden@tintri.com>
Date:   Tue Jun 16 16:41:59 2020 -0400

    FIR-291 Port File Access Auditing feature from NexentaStor
    (smbd_krb5ssp_work and smbd_user_auth_logon need to do the same post-work)

    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@tintri.com>
    Reviewed by: Suresh Jayaraman <sjayaraman@tintri.com>
    Reviewed by: Aditya Agnihotri <aagnihotri@tintri.com>

2    1    usr/src/cmd/smbsrv/smbd/smbd.h
42    15    usr/src/cmd/smbsrv/smbd/smbd_krb5ssp.c
60    36    usr/src/cmd/smbsrv/smbd/smbd_logon.c

commit 7a9583d281b08b47c88969bd51a6d6b96a3915f3
Author: Matt Barden <mbarden@tintri.com>
Date:   Fri May 15 15:19:22 2020 -0400

    idmap_cache_add_sid2pid() doesn't initialize winname_ttl, which can cause segfaults
    a.k.a. NEX-22788 Praudit can segfault in libidmap
    Prerequisite to FIR-291

    Reviewed by: Joyce McIntosh <jmcintosh@tintri.com>
    Reviewed by: Evan Layton <elayton@tintri.com>
    Reviewed by: Gordon Ross <gordon.ross@tintri.com>

6    0    usr/src/lib/libidmap/common/idmap_cache.c

This project enhances the SMB server so that it can generate audit events, through the existing BSM audit infrastructure, based on any AUDIT- or ALARM-type ACEs present on any ZFS file/directory shared over the protocol.

It adds a new event 'AUE_SACL', which belongs to the 'sa' audit class, containing the UID, GID, remote client ip/port, requested access, accessed file path, and the user SID (if present in the cred) of the user that caused the event to be generated, with an indicator of whether the access was or was not successful.
The SMB server generates 'AUE_SACL' events in a similar manner to Windows event ID 4656; when a file is opened (including implicit opens like rename/create), the DesiredAccess in the request is compared against an 'audit mask' calculated by combining the bits set in any matching AUDIT or ALARM-type ACE on the file; if the open was successful, it compares against the 'success' mask; otherwise, it compares against the 'failure' mask.
This works for both global and non-global zones, and respects the 'per-zone' audit policy (records are sent to the correct zone, and the path in the record is relative to the zone where the record is stored).
'auditreduce' has a new '-o wsid' option that can be used to filter records by SID.
'praudit' will print access masks in 'verbose' ACL format. In 'short mode', it prints them in 'compact' ACL format. In raw mode, it prints them as a single hexadecimal value. (See chmod(1) for ACL formats.)
praudit will also attempt to map SIDs to names using libidmap outside of raw mode.

Because the SMB server supports 'implied accesses' - such as DELETE_CHILD implying DELETE, or LIST_DIRECTORY implying READ_ATTRIBUTES, or the 'MAXIMUM_ALLOWED' bit - the server is the ultimate arbiter of whether a given request causes a record to be generated. Thus, the accesses that appear in the record may be different than those that were passed to VOP_ACCESS(). The filesystem passes the calculated 'audit mask' up to the server, which the server then compares against its calculated 'DesiredAccess'. It is also determines whether an event represents a successful or failed access, as well as whether one needs to be generated at all. For example: a protocol error that occurs before the file is opened won't generate a record at all, and a 'successful' record might be generated even if a protocol error occurs after the file was successfully opened.

Limitations:
1. Only SMB2+ is supported. (SMB1 has too many 'implicit opens')
2. Other Windows event types, like 4663 ('use of access' events), aren't implemented.
3. Events for creating files and directories generate a record for the directory in which it was created, using ADD_SUBDIRECTORY/ADD_FILE access, instead of an 'open handle' record on the newly created file.
4. auditreduce's '-o wsid' option does not accept names that could be mapped to a SID; that can be done via the local 'idmap' command.