Bug #13436
open
EPOLLET should wake up for each pipe write, even without reads
0%
Description
Some software makes use of a self pipe to wake up an event loop; i.e., a pipe where the read end is in the epoll set and the write end is available to other threads in order to inject wakeups.
On Linux systems, an edge-triggered wait on a pipe will fire for every write to the pipe, even if that pipe has not been read after poll. Some software in the wild has been observed to make use of this behaviour, draining the pipe only once the pipe buffer is full or nearing full.
On illumos systems, today, the epoll wakeup appears to occur only on a transition from an empty pipe buffer to a pipe buffer with at least one byte. The pipe must be drained completely before it will trigger again
A test, test_pipe_et, has been added to our epoll test suite: https://github.com/illumos/epoll-test-suite/blob/master/functional/test_pipe_et.c
This passes on Linux but fails on illumos:
$ ./test_pipe_et test_pipe_et 0 TPASS test_pipe_et 1 TPASS test_pipe_et 2 TFAIL: 0 != 1
Updated by Michael Zeller 3 months ago
bahamas10 noticed that this behavior causes a rustup install to fail on lx (omnios/smartos). He distilled it down to the following https://gist.github.com/bahamas10/a773315ebbbdba445a11bd62206b0480
Gist contents posted below:
Cargo.toml
[package]
name = "tokio-test"
version = "0.1.0"
edition = "2021"
[dependencies]
tokio = { version = "1.17.0", features = ["full"] }
main.rs
use tokio::fs;
use tokio::runtime::Builder;
fn main() {
let runtime = Builder::new_current_thread()
.enable_all() // THIS IS THE LINE THAT CAUSES THE BREAKAGE (this is default from the `tokio::main` macro)
.build()
.unwrap();
runtime.block_on(async {
let want_data = b"data\n";
let fname = "foo.txt";
fs::write(fname, want_data).await.unwrap();
let got_data = fs::read(fname).await.unwrap();
println!("want_data = {:?}", want_data);
println!("got_data = {:?}", got_data);
assert_eq!(got_data, want_data);
});
}
This sample code works in illumos but not under lx for whatever reason.
Updated by Michael Zeller 3 months ago
It appears the mio crate actually has a work around for illumos
https://github.com/tokio-rs/mio/blob/e077a23a4d7fd548da1381889ae3eaa5341512e7/src/sys/unix/waker.rs#L140-L143
139 pub fn wake(&self) -> io::Result<()> {
140 // The epoll emulation on some illumos systems currently requires
141 // the pipe buffer to be completely empty for an edge-triggered
142 // wakeup on the pipe read side.
143 #[cfg(target_os = "illumos")]
144 self.empty();