Project

General

Profile

Actions

Bug #13436

open

EPOLLET should wake up for each pipe write, even without reads

Added by Joshua M. Clulow over 1 year ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
kernel
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Some software makes use of a self pipe to wake up an event loop; i.e., a pipe where the read end is in the epoll set and the write end is available to other threads in order to inject wakeups.

On Linux systems, an edge-triggered wait on a pipe will fire for every write to the pipe, even if that pipe has not been read after poll. Some software in the wild has been observed to make use of this behaviour, draining the pipe only once the pipe buffer is full or nearing full.

On illumos systems, today, the epoll wakeup appears to occur only on a transition from an empty pipe buffer to a pipe buffer with at least one byte. The pipe must be drained completely before it will trigger again

A test, test_pipe_et, has been added to our epoll test suite: https://github.com/illumos/epoll-test-suite/blob/master/functional/test_pipe_et.c

This passes on Linux but fails on illumos:

$ ./test_pipe_et
test_pipe_et    0       TPASS
test_pipe_et    1       TPASS
test_pipe_et    2       TFAIL: 0 != 1
Actions #1

Updated by Michael Zeller 3 months ago

bahamas10 noticed that this behavior causes a rustup install to fail on lx (omnios/smartos). He distilled it down to the following https://gist.github.com/bahamas10/a773315ebbbdba445a11bd62206b0480

Gist contents posted below:

Cargo.toml

[package]
name = "tokio-test" 
version = "0.1.0" 
edition = "2021" 

[dependencies]
tokio = { version = "1.17.0", features = ["full"] }

main.rs

use tokio::fs;
use tokio::runtime::Builder;

fn main() {
    let runtime = Builder::new_current_thread()
        .enable_all() // THIS IS THE LINE THAT CAUSES THE BREAKAGE (this is default from the `tokio::main` macro)
        .build()
        .unwrap();

    runtime.block_on(async {
        let want_data = b"data\n";
        let fname = "foo.txt";

        fs::write(fname, want_data).await.unwrap();
        let got_data = fs::read(fname).await.unwrap();

        println!("want_data = {:?}", want_data);
        println!("got_data = {:?}", got_data);
        assert_eq!(got_data, want_data);
    });
}

This sample code works in illumos but not under lx for whatever reason.

Actions #2

Updated by Michael Zeller 3 months ago

It appears the mio crate actually has a work around for illumos
https://github.com/tokio-rs/mio/blob/e077a23a4d7fd548da1381889ae3eaa5341512e7/src/sys/unix/waker.rs#L140-L143

  139         pub fn wake(&self) -> io::Result<()> {
  140             // The epoll emulation on some illumos systems currently requires
  141             // the pipe buffer to be completely empty for an edge-triggered
  142             // wakeup on the pipe read side.
  143             #[cfg(target_os = "illumos")]
  144             self.empty();
Actions

Also available in: Atom PDF