illumos gate

I am seeing some strange behaviour on a couple of large machines (Dell PowerEdge R7525, 2xAMD EPYC 7543 32-Core and 512GB RAM) that I can't get a handle on. I don't see this on smaller hosts.

The machines have a number lx zones that are allocated 64 cores worth of CPU and 64 GB of RAM. These are used as compile engines, running up to 64 jobs.

What I see is very high host CPU usage (measured from cpu_nsec_idle), close to 100% with no measured CPU activity from the zones. This manifests as a stall in the zone, compiles take a very long time and the console (both zone and host) is unresponsive then things recover.

I ran a high load test on one of these machines and an older Dual E5-2687W 12-core CPU machine. Both had an lx zone with the cpu_cap and ram equal to the number of cores on the machine. In the zone, I run clang-tidy over a large code base with one job per core. The test script runs "nice" so it shouldn't make the system unresponsive.

While the tests were running, this dtrace script form Dan McDonald was run to see what was hogging the CPUs:
dtrace n 'profile-97 /curthread>t_name == NULL/ { @hogs[execname, "noname"] = count(); } profile-97 /curthread->t_name != NULL/ { @hogs[execname, stringof(curthread->t_name) ] = count(); } tick-10s {printa(@hogs);clear(@hog;}'

On the Xeon box, the machine is happy, other stuff just works and my kstat based logging shows the machine fully busy with the zone using 80% of the CPU for the duration of the test.
A typical sample is:

Tue Jun 21 07:49:28 2022
clang-tidy noname 1021094
qemu-system-x86_ noname 152650
bhyve vcpu 0 37661
bhyve vcpu 1 18564
bhyve vcpu 3 14050
bhyve vcpu 2 7540

Which is what I would expect to see for this test.

On the Big dual-socket EPYC 7543 machine, the system is unresponsive, I see 100% global zone CPU, I can't ssh on while the test is running and a typical sample is:

Tue Jun 21 21:08:01 2022
clang-tidy noname 52913
sched noname 43670
pageout noname 7805
bhyve vcpu 6 1593
bhyve vcpu 2 1336
bhyve vcpu 0 1285
zpool-zones noname 1239

when starting the test and

Tue Jun 21 21:10:51 2022
clang-tidy noname 55693
node noname 34680
svc.startd noname 6157
init noname 3530
svc.configd noname 3529
sched noname 1655
nscd revalidate 1309

once the test gets going.

The AMD machine has 176G of RAM allocated to bhyve machines and the ARC size remains constant at around 256G throughout.