As part of calibrating the APIC tick rate, we use the Intel 8254 Programmable Interval Timer (PIT). This ancient hardware still appears, often emulated, in most physical and virtual x86 systems.

In the past, Google have fixed bug 130531009 in the PIT emulation they provide in the Google Compute Engine (GCE) hypervisor. After that bug fix, the situation is much improved, but an apparent quirk in the emulation still remains.

In apic_calibrate_impl(), we disable interrupts and loop waiting for the PIT counter to hit a particular point in its periodic cycle. On a current GCE instance, we see the PIT value decrease on the first few reads, and then come to rest at 0. Subsequent reads in the first loop get stuck on this zero value. Looking at the implementation of the function, we're pretty much assuming that anybody who has used the PIT before us has returned it to an appropriate state for this operation. With other PIT implementations, our efforts to do this appear to be good enough -- but in GCE, it seems the PIT is stuck after counting down once for whatever reason.

Using kmdb, I was able to demonstrate (through the use of ::in and ::out) that resetting the PIT to a periodic mode (i.e., mode 2 or 3) at the top of the function and loading a counter value was enough to "unstick" it. Rick McNeal from Nexenta reports that adding a call to microfind() at the top of the function also unsticks the PIT, which makes sense; the last thing microfind() does is reprogram the PIT into mode 3 (square wave) and load a value, in attempt to leave things as we expect we might have found them.

Adding the PIT program and load sequence at the top of apic_calibrate_impl() has the advantage of being documented PIT behaviour, and behaviour that works in the other places we use the PIT today. This should dramatically reduce the risk of making such a change.