Project

General

Profile

Actions

Bug #13961

closed

Add HPET as a TSC calibration source

Added by Jason King 3 months ago. Updated 29 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
kernel
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

With #13354 nearing completion, to support platforms that don't have a TSC (such as the latest generation Intel NUCs), we should support using the HPET to calibrate the TSC.

The HPET is present on most (if not all) modern x86 systems, and the specification requires a minimum frequency of 10Mhz -- roughly 10x the frequency of the PIT. As part of this change, the HPET will be used (when present) in preference to the PIT. Any system without a PIT will fallback to using the PIT a today.

Additionally, the HPET code has to move out of the PSMs so it can be used during startup. Currently the HPET initialization enabled the HPET as well as enables interrupts. Because the APIC isn't initialized until the PSM is loaded, the HPET initialization is split into two pieces.

The first portion executes during the early boot and enables enough of the HPET that it can begin counting down (when programmed). The remainder of the HPET initialization is performed (including interrupts) at the point where HPET initialization occurs today.


Related issues

Related to illumos gate - Bug #13354: illumos should calibrate the TSC earlier in the boot process.ClosedJason King

Actions
Actions #1

Updated by Jason King 3 months ago

  • Related to Bug #13354: illumos should calibrate the TSC earlier in the boot process. added
Actions #2

Updated by Jason King 3 months ago

  • Description updated (diff)
Actions #3

Updated by Jason King about 2 months ago

To test this, I first booted a Xeon-D server w/ this change. I verified that the calibration source was the HPET and that it had correctly determined the TSC frequency (2.1Ghz). I also compared the HPET vs the TSC measurements:

HPET: 2100836143
PIT: 2099960238

A difference of < 0.04% (or ~875KHz -- less than 1 tick of the TSC).

I also repeated the same test on a Bhyve VM. I also verified it calibrated using the HPET (from mdb -k):

> *tsc_calibration_source::print tsc_calibrate_t
{
    tscc_source = 0xfffffffffb997848 "HPET" 
    tscc_preference = 0x32
    tscc_calibrate = tsc_calibrate_hpet
}

HPET: 2101565301
PIT: 2098786469

The Bhyve VM shows a larger different, though still small -- 0.13% or ~2.8Mhz. Given that the HPET and PIT are both emulated, this still seems reasonable.

Finally, I build a USB image of SmartOS w/ this change and booted it on a 10th gen Intel NUC (which was the impetus for #13354 as well as this issue). Prior to this change, this system would not boot SmartOS due to the lack of a functioning PIT (with a bit of digging, I was able to find some literature from Intel that confirmed the lack of a PIT was deliberate). With this change, the system was able to boot successfully.

Actions #4

Updated by Joshua M. Clulow about 2 months ago

How do these values (for the hardware and for bhybe) compare with HPET calibration results on other platforms; e.g., Linux?

Actions #5

Updated by Jason King about 2 months ago

I haven't been able to find a way to force Linux to use the HPET for TSC calibration -- at least in the Ubuntu VM, it's always preferring the ACPI PM timer over the HPET. You can tell the Linux kernel to use the HPET in lieu of the TSC, but I haven't found anything obvious (yet) to force it to use the HPET over the PMTIMER for calibrating the TSC (if anyone has ideas to try, I can give them a shot).

However, the values returned do match the known actual CPU frequencies (within experimental error -- since we're doing a measurement vs. them being assigned from some authoritative source, I would expect some amount of variance), and where we do have a PIT, match closely to the values we would have used previously (within 1 PIT tick on raw iron, and about 2 PIT ticks on a VM) -- i.e. if Linux or any other OS were getting values other than the actual CPU frequency, that would be rather surprising.

Actions #6

Updated by Jason King about 1 month ago

  • Description updated (diff)

Hadfl was kind enough to verify the HPET info on a physical system running OmniOS before/after the change:

Before:

Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci zfs sata sd ip hook neti sockfs arp usba xhci mm stmf stmf_sbd lofs ipc ptm ]
> hpet_info::print hpet_info_t
{
    gen_cap = {
        counter_clk_period = 0x429b17f
        vendor_id = 0x8086
        leg_route_cap = 0x1
        res1 = 0
        count_size_cap = 0x1
        num_tim_cap = 0x8
        rev_id = 0x1
    }
    gen_config = {
        leg_rt_cnf = 0
        enable_cnf = 0x1
    }
    gen_intrpt_stat = 0
    main_counter_value = 0x29a5498b
    logical_address = 0xfffffe2ca7522000
    timer_n_config = 0xfffffe2cf3949940
    num_timers = 0
    allocated_timers = 0x4
    cstate_timer = {
        timer = 0x2
        intr = 0xb
    }
    hpet_main_counter_reads = [ 0x29a54983, 0x29a5498b ]
    tsc = [ 0x28c3ade8ca, 0x28c3adf456, 0x28c3adfbe2 ]
    period = 0x429b17f
}
> hpet_info::print hpet_info_t timer_n_config | ::print hpet_TN_conf_cap_t
{
    int_route_cap = 0xf00000
    res1 = 0
    fsb_int_del_cap = 0x1
    fsb_int_en_cnf = 0
    int_route_cnf = 0
    mode32_cnf = 0
    res2 = 0
    val_set_cnf = 0
    size_cap = 0x1
    per_int_cap = 0x1
    type_cnf = 0
    int_enb_cnf = 0
    int_type_cnf = 0
    res3 = 0
}
> hpet_state::print struct hpet_state
{
    proxy_installed = 0 (0)
    cpr = 0 (0)
    cpu_deep_idle = 0x1 (B_TRUE)
    uni_cstate = 0x1 (B_TRUE)
}
> hpet::print hpet_t
{
    supported = 0x3
    install_proxy = hpet_install_proxy
    callback = hpet_callback
    use_hpet_timer = hpet_use_hpet_timer
    use_lapic_timer = hpet_use_lapic_timer
}

With this change:

Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci zfs sata sd ip hook neti sockfs arp usba xhci stmf stmf_sbd mm lofs ipc ptm ]
> hpet_info::print hpet_info_t
{
    gen_cap = {
        counter_clk_period = 0x429b17f
        vendor_id = 0x8086
        leg_route_cap = 0x1
        res1 = 0
        count_size_cap = 0x1
        num_tim_cap = 0x8
        rev_id = 0x1
    }
    gen_config = {
        leg_rt_cnf = 0
        enable_cnf = 0x1
    }
    gen_intrpt_stat = 0
    main_counter_value = 0x29196a75
    logical_address = 0xfffffe2ca7400000
    timer_n_config = 0xfffffe2ceb500e00
    num_timers = 0
    allocated_timers = 0x4
    cstate_timer = {
        timer = 0x2
        intr = 0xb
    }
    hpet_main_counter_reads = [ 0x29193205, 0x2919320d ]
    tsc = [ 0x2841d03a30, 0x2841d045bc, 0x2841d04d44 ]
    period = 0x429b17f
}
> hpet_info::print hpet_info_t timer_n_config | ::print hpet_TN_conf_cap_t
{
    int_route_cap = 0xf00000
    res1 = 0
    fsb_int_del_cap = 0x1
    fsb_int_en_cnf = 0
    int_route_cnf = 0
    mode32_cnf = 0
    res2 = 0
    val_set_cnf = 0
    size_cap = 0x1
    per_int_cap = 0x1
    type_cnf = 0
    int_enb_cnf = 0
    int_type_cnf = 0
    res3 = 0
}
> hpet_state::print struct hpet_state
{
    proxy_installed = 0 (0)
    cpr = 0 (0)
    cpu_deep_idle = 0x1 (B_TRUE)
    uni_cstate = 0x1 (B_TRUE)
}
> hpet::print hpet_t
{
    supported = 0x3
    install_proxy = hpet_install_proxy
    callback = hpet_callback
    use_hpet_timer = hpet_use_hpet_timer
    use_lapic_timer = hpet_use_lapic_timer
}

The differences seen between the two seem reasonable -- the virtual address used to map the HPET is going to vary depending on the behavior of system, and the counter values are going to vary based on when they're read (and we read the HPET earlier with this change).

Actions #7

Updated by Electric Monk 29 days ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 21bcbe6e4903d8521ec66863bf0c21d9ed378cff

commit  21bcbe6e4903d8521ec66863bf0c21d9ed378cff
Author: Jason King <jason.brian.king@gmail.com>
Date:   2021-09-20T18:03:04.000Z

    13961 Add HPET as a TSC calibration source
    Reviewed by: Andy Fiddaman <andy@omnios.org>
    Reviewed by: Toomas Soome <tsoome@me.com>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF