illumos gate

The code was introduced by Git commit 7c478bd95313f5f23a4c, so I think the code is at least 9 years old. To start with a delay of 1s doesn’t seem to be reasonable in modern times.

We’ve encountered a side effect when dropping the initial delay from 1s to 1ms. The distribution of lock assignments gets much better in case the first process(P1) does some periodical file locking and unlocking. For P2, there is a better chance or a higher probability to catch the lock after it was freed by P1. With the current back-off behavior, P1 is able to catch the lock again and again while P2 would probably sleep for a long time.

1ms back off may be under the threshold for user land sleep times (clock is often 100Hz, i.e. 10msec). I'd recommend trying a 20msec delay. This is still far better than 1 sec (1/50th the time), but it is likely to give other participants a fair chance to get the lock.

The actual locking at the NFSv4 server is implemented in setlock() - see file nfs4_srv.c.

In a case the file is already locked, setlock() tries to lock the file four times in a row with delays 10ms, 20ms, and 40ms between retries. So in a case the file is locked for long time by someone else, the (denied) lock reply will arrive to the NFSv4 client in about 70ms.

Based on that, it is not very important for the NFSv4 server whether the initial delay at the NFSv4 client between lock retries is 1ms, 10ms, or even more. The illumos NFSv4 server already have some overload protection implemented.

OTOH, it also won't help much to have the initial delay at the client set to 1ms only. When considering the illumos NFSv4 implementation, the better initial delay would be tens of milliseconds, IMHO. It should be noted, that for other NFSv4 server implementations the best initial delay might differ.

Please note that it is always possible to construct a test case where any arbitrarily chosen initial delay timeout is the best. You apparently found a scenario where 1ms seems to be optimal, but it is easy to construct another test case where, for example, 100ms would behave better than 1ms.

Simon Klinkert wrote:

We’ve encountered a side effect when dropping the initial delay from 1s to 1ms. The distribution of lock assignments gets much better in case the first process(P1) does some periodical file locking and unlocking. For P2, there is a better chance or a higher probability to catch the lock after it was freed by P1. With the current back-off behavior, P1 is able to catch the lock again and again while P2 would probably sleep for a long time.

The above looks like #4842.

The server side has a (slow) retry loop for locking requests as well. Three retries are possible, starting with a delay of 10ms:

 6  73172                    setlock:entry
  6  20809                 fop_frlock:entry
  6  20810                fop_frlock:return ret=11
  6  20809                 fop_frlock:entry
  6  20810                fop_frlock:return ret=11
  1  20809                 fop_frlock:entry
  1  20810                fop_frlock:return ret=11
  2  20809                 fop_frlock:entry
  2  20810                fop_frlock:return ret=11
  2  20809                 fop_frlock:entry
  2  20810                fop_frlock:return ret=0
  2  73173                   setlock:return time=69ms

The nfs4 file locking feels slower than it should be on a system with hundreds of processes struggling with the same file. Even an unlock may need 120s(!).

Unfortunately, rfs4_maxlock_tries and rfs4_lock_delay are static variables and thus not tunable by mdb or /etc/system.