illumos gate

During code review the following issue was raised.

It appears other systems ignore the failure here (FreeBSD 
at least from a quick glance). I guess following the Linux
error makes the most sense? Did you consider how others
swallow the error at all?

Enumerate in the ticket what Linux and FreeBSD do here, where they differ and where they're the same

FreeBSD
        The system will attempt to avoid caching the data you
        read or write. If it cannot avoid caching the data, it will
        minimize the impact the data has on the cache.

Linux
        Try to minimize cache effects of the I/O to and from this file.
        error return:
        EINVAL The filesystem does not support the O_DIRECT flag.

        The Linux man page also documents a lot more information
        about O_DIRECT caveats and possible behavior. I'm not
        including all of that here, but a few highlights:
        - File I/O is done directly to/from user-space buffers.
        - The O_DIRECT flag may impose alignment restrictions.
        - FreeBSD 4.x introduced a flag of the same name, but 
          without alignment restrictions.
        - It is recommended that applications treat use of O_DIRECT as
          a performance option which is disabled by default.

Summary        
        The only important, application visible difference between
        FBSD and Linux is in the error handling when the underlying
        file system doesn't support O_DIRECT. 
        Linux returns EINVAL, FBSD does not return an error

        The currently proposed solution is doing this.

        The currently proposed solution is doing this.

Postgres behavior summary

#ifdef O_DIRECT
#define PG_O_DIRECT                             O_DIRECT
#else
#define PG_O_DIRECT                             0
#endif

The code then uses PG_O_DIRECT in many places as part of
the open(2) flags.

Thus, once the illumos headers have O_DIRECT, PG will start using it.

The PG code will fail out in most cases if open(2) returns an
error of almost every kind (there are a few places they specifically
handle ENOENT, but nothing else).

On illumos, this behavior would be fine for a DB on:
    zfs 
    ufs 
    nfs (although probably not used with PG)

On illumos, this behavior would fail for a DB on:
    pcfs
    tmpfs
Both of these are unlikely file systems to host a PG DB.

Thanks for putting this together, Jerry. I think this causes the Linux behavior to make sense. As in erroring with EINVAL if it's not supported. I think the big question for me is what happens to folks from a performance perspective if O_DIRECT is now being used all of a sudden by applications like postgres where having data cached in the arc was load bearing. I don't know how much of a risk that really is or not. I assume that some of the changes being proposed around how O_DIRECT works in ZFS basically wouldn't really be at issue here and that the system wouldn't default to some of those proposals?

Literally nothing happens in that case since ZFS on both illumos and OpenZFS currently does absolutely for O_DIRECT. I put the explanation for this in the comment when I added support to ZFS for the _FIODIRECTIO ioctl. I'll include that here for reference

                 * ZFS inherently provides the basic semantics for directio.
                 * This is the summary from the ZFS on Linux support for
                 * O_DIRECT, which is the common form of directio, and required
                 * no changes to ZFS.
                 *
                 * 1. Minimize cache effects of the I/O.
                 *
                 *    By design the ARC is already scan-resistant, which helps
                 *    mitigate the need for special O_DIRECT handling.
                 *
                 * 2. O_DIRECT _MAY_ impose restrictions on IO alignment and
                 *    length.
                 *
                 *    No additional alignment or length restrictions are
                 *    imposed by ZFS.
                 *
                 * 3. O_DIRECT _MAY_ perform unbuffered IO operations directly
                 *    between user memory and block device.
                 *
                 *    No unbuffered IO operations are currently supported. In
                 *    order to support features such as compression, encryption,
                 *    and checksumming a copy must be made to transform the
                 *    data.
                 *
                 * 4. O_DIRECT _MAY_ imply O_DSYNC (XFS).
                 *
                 *    O_DIRECT does not imply O_DSYNC for ZFS.
                 *
                 * 5. O_DIRECT _MAY_ disable file locking that serializes IO
                 *    operations.
                 *
                 *    All I/O in ZFS is locked for correctness and this locking
                 *    is not disabled by O_DIRECT.

OK, given all that and the fact that things on ZFS will probably always default to the current behavior, then it seems like this is pretty safe and there won't be too much in terms of adverse reactions. Thanks for taking a look.