illumos gate

Upon receiving a seemingly complete zfs snapshot from its leader, the mounted snapshot shows "phantom" file entries.

-rw-------  1 postgres postgres 16777216 Jul  1 16:05 000000010000000900000075
-rw-------  1 postgres postgres 16777216 Jul  1 16:05 000000010000000900000076
??????????  ? ?        ?               ?            ? 000000010000000900000077
??????????  ? ?        ?               ?            ? 000000010000000900000078
??????????  ? ?        ?               ?            ? 000000010000000900000079
??????????  ? ?        ?               ?            ? 00000001000000090000007A
??????????  ? ?        ?               ?            ? 00000001000000090000007B
??????????  ? ?        ?               ?            ? 00000001000000090000007C
??????????  ? ?        ?               ?            ? 00000001000000090000007D
-rw-------  1 postgres postgres 16777216 Jul  1 20:05 00000001000000090000007E
-rw-------  1 postgres postgres 16777216 Jul  1 21:05 00000001000000090000007F
-rw-------  1 postgres postgres 16777216 Jul  1 21:05 000000010000000900000080
-rw-------  1 postgres postgres 16777216 Jul  1 21:27 000000010000000900000081
-rw-------  1 postgres postgres 16777216 Jun 17 20:00 000000010000000900000082
-rw-------  1 postgres postgres 16777216 Jun 17 20:00 000000010000000900000083
-rw-------  1 postgres postgres 16777216 Jun 17 20:00 000000010000000900000084
-rw-------  1 postgres postgres 16777216 Jun 17 20:00 000000010000000900000085
-rw-------  1 postgres postgres 16777216 Jun 17 20:00 000000010000000900000086
-rw-------  1 postgres postgres 16777216 Jun 17 20:00 000000010000000900000087

The filesystem has corrupt dirents, pointing to nonexistent object IDs. The destination snapshot from which the filesystem was constructed is corrupt in exactly the same manner.

The source snapshot gets EBUSY when zdb attempts to open it. This strongly suggests that DS_FLAG_INCONSISTENT is set on it. This flag is set only when we are receiving the snapshot or destroying the dataset. In this case, the snapshot was taken on the source at a time when it was itself inconsistent; upon receiving it, the destination filesystem becomes similarly corrupt.

This is easily reproduced as follows:

1. zfs create pool/testfs
2a. zfs recv pool/testfs < somestream &
2b. zfs snapshot pool/testfs@snap
3. zfs send pool/testfs@snap > whatever

It's easiest to see if the stream being received has interesting contents, since that increases the window during which the snapshot will be noticeably corrupt. You can then zfs recv 'whatever' into a new dataset on the same or a different system and that filesystem will be corrupt.