illumos gate

POSIX 2008 (SUSv7) specifies the use of O_DIRECTORY:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html

The justification is here:

"In addition, the open() function refuses to open non-directories if the O_DIRECTORY flag is set. This avoids race conditions whereby a user might compromise the system by substituting a hard link to a sensitive file (e.g., a device or a FIFO) while a privileged application is running, where opening a file even for read access might have undesirable side-effects."

Apparently some software now depends on this, as reported to me by Sebastian Wiedenroth:

Some software (slibtool) recently started using O_DIRECTORY, which is how I noticed that it’s not yet available on illumos.
  Looking further I found that you had already implemented support for it in illumos-core, it just never made it into the gate:

Indeed I have implemented this in illumos-core.

The code I have for this today does not allow O_CREAT and O_DIRECTORY be supplied together. The version of the specification I recall reading about this was a big vague about this. The 2018 version linked above is also still vague, but seems to suggest that it should be possible to use the two together with O_RDWR or O_WRONLY (presumably to create a directory). The version of the code I am planning to supply at first lacks this capability, and simply reports EINVAL if the two are used together.

It may be reasonable to refactor this slightly to add support so that O_DIRECTORY|O_CREAT|O_WR behaves like mkdir. Ideally I would like to do that in a follow up task if needed.