pipe buffer size could be larger
The pipe buffer size on illumos systems is not especially large. At time of writing, the value of
FIFOHIWAT is 16KB. This value is used as the high water mark for the pipe, inducing blocking when exceeded. The value is used in the following chain of calls:
pipe() -> makepipe() -> kmem_cache_alloc(pipe_cache) -> pipe_constructor() -> fnode_constructor -> fn_hiwat = FIFOHIWAT
A buffer size of 16KB is quite small, and on modern systems represents a serious performance bottleneck. It is commonly received wisdom to use program like
pv in the middle of a pipeline; e.g., "
zfs send | mbuffer | ssh remotehost", or "
nc -l 10240 | pv | zfs recv".
Other operating systems have increased the default pipe buffer size over time. In Linux 2.6.11, the default size on x86 systems grew to 64KB (16 times the page size). As of Linux 2.6.35, the pipe buffer size can also be altered with
fcntl(F_SETPIPE_SZ) and set to an even higher value. A cursory inspection suggests that FreeBSD will also grow pipe buffer space as needed up to
BIG_PIPE_SIZE, set to 64KB.
It seems like it would be reasonable to set
FIFOHIWAT to 64KB at this stage. As part of making this change, we should produce a benchmark that demonstrates the improvement for a range of high water mark values.