Project

General

Profile

Bug #13039

HPET on AWS EC2 may be Hazard-Prone Error Trigger

Added by Joshua M. Clulow 7 months ago. Updated 6 months ago.

Status:
Closed
Priority:
Normal
Category:
kernel
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Expert
Tags:
Gerrit CR:

Description

Amazon provide a wide variety of virtual machine configurations in their cloud. Many of them advertise themselves as hardware virtualised Xen guests. These instances provide a number of Xen-specific paravirtualised facilities, such as a clock and timer source. In addition, some number of classical facilities are advertised and emulated, though the emulation is not always perfect.

In at least m4.16xlarge instance size, though notably not in the slightly smaller m4.10xlarge instance size, programming the HPET will immediately reset the guest instance. This also happens on current Ubuntu Linux, which I confirmed with the samples/timers/hpet_example.c program that ships with the Linux kernel. This program will open and engage the HPET from a user mode process (via /dev/hpet) on that system. In a 16x guest, it would reset immediately; in a 10x guest it would correctly program the timer and report various firings.

Because HPET support is of somewhat variable quality on AWS, we should refuse to program the HPET there. Amazon is a big user of Xen, and it is not hard to assume that this defect may appear in other Xen distributions as well; certainly we have had other unresolved issues on Xen platforms in the last few years (e.g., #8053, #8079, #9533, #5230, etc). We can gate on the HW_XEN_HVM bit in the return of get_hwenv() as was done for #9533 / #8079

Refusing to program the HPET will have the most impact on systems which do not offer the Always-Running APIC Timer (ARAT) CPU feature; on such systems, the LAPIC timer does not run in deep C-states and thus we need the HPET to wake CPUs up from that state. If the HPET is also unavailable, cpu_deep_cstates_supported() will return false and we will not use deep C-states at all. While this is somewhat less than ideal, it is also the way stock SmartOS systems have been shipping (for better or worse) for a number of years, as SmartOS includes a idle_cpu_no_deep_c override in /etc/system which has the same effect.

In a follow-up issue we could do the work to support the Xen timer source, which would enable us to use that as a proxy timer instead of the HPET. In the interim, this unabashed workaround will allow us to boot in more AWS instance types.

#1

Updated by Joshua M. Clulow 7 months ago

To check whether ARAT, HPET, or the deep C-state override are active, you can run:

mdb -ke 'cpu_cstate_hpet/D; cpu_cstate_arat/D; idle_cpu_no_deep_c/D'
#2

Updated by Joshua M. Clulow 7 months ago

With the HPET programming disabled, I can boot an m4.16xlarge instance:

root@ip-172-31-15-44:~# uname -a
SunOS ip-172-31-15-44 5.11 fixaws-aws/hpet-0-gd94076685e i86pc i386 i86pc
root@ip-172-31-15-44:~# mdb -ke 'cpu_cstate_hpet/D; cpu_cstate_arat/D; idle_cpu_no_deep_c/D'
cpu_cstate_hpet:
cpu_cstate_hpet:0
cpu_cstate_arat:
cpu_cstate_arat:0
idle_cpu_no_deep_c:
idle_cpu_no_deep_c:             0
root@ip-172-31-15-44:~# curl -s http://169.254.169.254/latest/meta-data/instance-type ; echo
m4.16xlarge
#3

Updated by Electric Monk 7 months ago

  • Gerrit CR set to 851
#4

Updated by Joshua M. Clulow 6 months ago

Testing Notes

I built the changes, and then installed them to a disk image that can be booted in Amazon EC2. In EC2, I then created a series of VMs:

  • instance type m4.10xlarge
    • stock OpenIndiana boots (expected); HPET was programmed because the ARAT CPU feature is not present
    • RTI build for 13039; boots OK, HPET not programmed due to the updated behaviour
  • instance type m4.16xlarge
    • stock OpenIndiana does not boot (expected); enters reboot loop
    • RTI build for 13039; boots OK, HPET not programmed due to the updated behaviour

In addition, I booted these bits on an Intel NUC that supports C-states and currently programs the HPET successfully even though it doesn't need it:

Physical hardware HPET programming information before reboot:

$ pfexec mdb -ke hpet_info::print
{
    gen_cap = {
        counter_clk_period = 0x27bc86b
        vendor_id = 0x8086
        leg_route_cap = 0x1
        res1 = 0
        count_size_cap = 0x1
        num_tim_cap = 0x8
        rev_id = 0x1
    }
    gen_config = {
        leg_rt_cnf = 0
        enable_cnf = 0x1
    }
    gen_intrpt_stat = 0
    main_counter_value = 0x2712f0d1
    logical_address = 0xfffffe2c94347000
    timer_n_config = 0xfffffe2ce05291c0
    num_timers = 0
    allocated_timers = 0x4
    cstate_timer = {
        timer = 0x2
        intr = 0xb
    }
    hpet_main_counter_reads = [ 0x2712f063, 0x2712f0d1 ]
    tsc = [ 0x10a66dba24, 0x10a66decf8, 0x10a66e1b2a ]
    period = 0x27bc86b
}

Programming information after reboot:

newcastle # pfexec mdb -ke hpet_info::print
{
    gen_cap = {
        counter_clk_period = 0x27bc86b
        vendor_id = 0x8086
        leg_route_cap = 0x1
        res1 = 0
        count_size_cap = 0x1
        num_tim_cap = 0x8
        rev_id = 0x1
    }
    gen_config = {
        leg_rt_cnf = 0
        enable_cnf = 0x1
    }
    gen_intrpt_stat = 0
    main_counter_value = 0x2ca22bf4
    logical_address = 0xfffffe2c94347000
    timer_n_config = 0xfffffe2ce05091c0
    num_timers = 0
    allocated_timers = 0x4
    cstate_timer = {
        timer = 0x2
        intr = 0xb
    }
    hpet_main_counter_reads = [ 0x2ca22b86, 0x2ca22bf4 ]
    tsc = [ 0x17826f9d7a293c, 0x17826f9d7a5c2e, 0x17826f9d7a8a78 ]
    period = 0x27bc86b
}
#5

Updated by Electric Monk 6 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 92f7b1e23c49ac9522feadd9b58cb103a05a509b

commit  92f7b1e23c49ac9522feadd9b58cb103a05a509b
Author: Joshua M. Clulow <josh@sysmgr.org>
Date:   2020-09-12T23:52:31.000Z

    13039 HPET on AWS EC2 may be Hazard-Prone Error Trigger
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Also available in: Atom PDF