illumos gate

When 2 clients want to do a setattr on a file simultaneously, one might fail with EIO. The trace for this client looks like this:

48554 12:55:5.77231   c -> s   NFS C 4 (open        ) PUTFH FH=9BC2 OPEN file...
48555 12:55:5.86796   s -> c   NFS R 4 (open        ) NFS4_OK PUTFH NFS4_OK OPEN NFS4_OK ST=26C3:1 RF=PL DT=W DST=23A1:0 LB=SZ(0) GETFH NFS4_OK FH=7E8C GETATTR...
48556 12:55:5.86798   s -> c   NFS C CB4 (cb_recall) CBID=1073742022 CB_RECALL FH=7E8C ST=23A1:0 TR=F 
48557 12:55:5.86816   c -> s   NFS R CB4 (cb_recall) NFS4ERR_BAD_STATEID CB_RECALL NFS4ERR_BAD_STATEID 
48558 12:55:5.86817   c -> s   NFS C 4 (setattr     ) PUTFH FH=7E8C SETATTR ST=23A1:0 GETATTR 10011a b0a23a 
48559 12:55:5.86823   s -> c   NFS R 4 (setattr     ) NFS4ERR_BAD_STATEID PUTFH NFS4_OK SETATTR 0 0 
48560 12:55:5.87271   c -> s   NFS C 4 (delegreturn ) PUTFH FH=7E8C GETATTR 10011a b0a23a DELEGRETURN DST=23A1:0 
48561 12:55:5.87277   s -> c   NFS R 4 (delegreturn ) NFS4ERR_BAD_STATEID PUTFH NFS4_OK GETATTR NFS4_OK DELEGRETURN NFS4ERR_BAD_STATEID 

i.e. the server grants a delegation with the open and immediately recalls it afterwards. Although the recall arrives at the client after the delegation, the client doesn't know about the delegation yet, because it arrives on a different tcp connection and can internally overtake the delegation.
After discussion with Marcel Telka a possible solution could be that in case the client receives a recall for a delegation for a file for which it has an outstanding open request, it should delay the answer to the recall until it has received (and processed) the open reply.