Project

General

Profile

Feature #10426

pipe buffer size could be larger

Added by Joshua M. Clulow 10 months ago. Updated 10 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
kernel
Start date:
2019-02-19
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:

Description

The pipe buffer size on illumos systems is not especially large. At time of writing, the value of FIFOHIWAT is 16KB. This value is used as the high water mark for the pipe, inducing blocking when exceeded. The value is used in the following chain of calls:

pipe() -> makepipe() -> kmem_cache_alloc(pipe_cache) -> pipe_constructor() -> fnode_constructor -> fn_hiwat = FIFOHIWAT

A buffer size of 16KB is quite small, and on modern systems represents a serious performance bottleneck. It is commonly received wisdom to use program like mbuffer or pv in the middle of a pipeline; e.g., "zfs send | mbuffer | ssh remotehost", or "nc -l 10240 | pv | zfs recv".

Other operating systems have increased the default pipe buffer size over time. In Linux 2.6.11, the default size on x86 systems grew to 64KB (16 times the page size). As of Linux 2.6.35, the pipe buffer size can also be altered with fcntl(F_SETPIPE_SZ) and set to an even higher value. A cursory inspection suggests that FreeBSD will also grow pipe buffer space as needed up to BIG_PIPE_SIZE, set to 64KB.

It seems like it would be reasonable to set FIFOHIWAT to 64KB at this stage. As part of making this change, we should produce a benchmark that demonstrates the improvement for a range of high water mark values.

History

#1

Updated by Joshua M. Clulow 10 months ago

  • Description updated (diff)

Also available in: Atom PDF