Project

General

Profile

Feature #10277

Properly detect SMT on AMD

Added by Robert Mustacchi 11 months ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2019-01-24
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

With the advent of AMD's Zen microarchitecture, AMD now has a fully formed support for SMT. We need to properly detect this and note it while handling various parts of the CPU topology work.

As a part of this, this overhauls a lot of the different topology related cpuid functions which have been done at different points in time and are used in different ways. Making this more consistent and hopefully more understandable in the process.

A notable side effect of this is that we will have reduced certain classes of warnings that popped up on AMD systems:

2019-01-03T18:45:12.582427+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 16 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.582542+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 18 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.582702+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 20 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.582824+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 22 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.582966+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 24 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.583082+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 26 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.583291+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 28 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.583485+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 30 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.583649+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 32 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.583858+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 34 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.584074+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 36 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.584203+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 38 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.584372+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 40 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.584500+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 42 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.584620+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 44 strand 0 (cmi_hdl_create returned NULL)#012
2019-01-03T18:45:12.584739+00:00 odyssey unix: [ID 230305 kern.warning] WARNING: There will be no MCA support on chip 0 core 46 strand 0 (cmi_hdl_create returned NULL)#012
...

This was caused because we don't properly determine the topology information of all of the AMD systems in the same way as we do for Intel. This change takes care of this as well as making sure that all of the cache information needed for scheduling is updated.


Related issues

Related to illumos gate - Feature #10263: Update cpuid detection for new EPYC Socket formatClosed2019-01-18

Actions
Related to illumos gate - Bug #10514: 10277 broke older Intel CPUsClosed2019-03-08

Actions

History

#1

Updated by Robert Mustacchi 11 months ago

To test these changes, I put together a suite of five different commands
that I was looking at. These are:

  • Looking at differences in isainfo -x
  • Looking at differences in psrinfo -vp
  • Looking at differences in pginfo -v
  • Looking at differences in kstat -p cpu_info
  • Looking at differences in mdb -ke '::walk cpu | ::print cpu_t cpu_m.mcpu_cpi[]'

The following table describs the systems tested and the differences that
have occurred. With this scheme, one could always see trivial changes in
the mdb and kstat output respectively. The kstat output changes were
because of the following:

  • Differences in the ACPI state at time of gathering which was because
    of whether or not the system was idle.
  • Differences in the snaptime and crtime of the kstat which was because
    of normal differences in the kstats.
  • Occasionally a difference in current CPU frequency.

The data in mdb was different in the following ways for normal cases:

  • Two of the members of the struct cpuid_info were renamed.
  • The address of dynamically allocated strings changed from boot to boot.
  • For AMD CPUs, the way that a CPU revision ID was encoded had changed
    due to '10263 Update cpuid detection for new EPYC Socket format'.

The data in isainfo occasionally changed, but it was always because the
testing was done on either side of the change 'OS-7167 Need support for
new EPYC ISA extensions'.

Systems that only have differences which are in those cases, are not
called out as differences in the table below:

CPU Notes
Haswell Client 1s -
Skylake Server 2s Gold 6132 -
AMD EPYC 2s There were substantial changes here, which wase point of this overhaul. The output of pginfo -v now correctly reflects the CPU's actual topology. Specifically this means that we're correctly noting what parts of the last level cache are shared, the hyperthreading relationships, etc.
AMD Athlon(tm) II Neo N36L Dual-Core Processor (family 10h) -
AMD Turion(tm) II Neo N40L Dual-Core Processor (family 10h) -
AMD Opteron(tm) X3216 APU (family 15h) Socket now correctly detected
AMD GX-412TC (family 16h) Socket, revision, and last level cache now correctly detected
Six-Core AMD Opteron(tm) Processor 8431 (family 10h) -
AMD Athlon(tm) II X4 620 Processor We noted something very interesting here, the core and clogid didn't match. This is because one was derived from the CPU ID and the other from the APIC ID, and conincidentally this time, they were not equivalent. This is allowed.

The next set of systems were all virtualized:

Hypervisor CPU VCPUs Notes
VMware Fusion 2 Modified host (Client Ivy Bridge) The chipid labeling had been removed; however, the core labeling was much more accurate now
illumos KVM qemu64 1 -
illumos KVM qemu64 4 -
illumos KVM Modified host (Hawell Client 1s) 1 -
illumos KVM Modified host (Hawell Client 1s) 4 -
illumos bhyve Modified host (Haswell Client 1s) 1 cpi_chipid enumeration had changed and all chipids are set to -1
illumos bhyve Modified host (Haswell Client 1s) 4 cpi_chipid enumeration had changed and all chipids are set to -1
Linux KVM Skylake, IBRS 2 cpi_chipid enumeration had changed and all chipids are set to -1
Linux KVM EPYC, IBPB 16 -
Virtual Box Modified host (Kaby Lake) 2 -

It is notable that the cpi_chipid has changed in the virtualization case
to be more explicitly -1. This ultimately seems fine as it is part of us
being honest that we're virtualized. It didn't seem like other parts of
the system really cared about this.

In terms of other testing that was performed, on the AMD EPYC system in
question, it appeared that the full MCA architecture was now correctly
functioning as we were able to have the MCA framework attach to all CPUs
and unlike when we got it wrong, there were no spurious or strange
events on the system. I compared /proc/cpuinfo in an lx branded zone. With
these changes we now report both the number of siblings and the physical id.
We don't report additional features as the work to map them has not been done.

#2

Updated by Robert Mustacchi 11 months ago

  • Related to Bug #10275: Kernel attempts to apply old erratum to AMD Threadripper 1950X added
#3

Updated by Robert Mustacchi 11 months ago

  • Related to deleted (Bug #10275: Kernel attempts to apply old erratum to AMD Threadripper 1950X)
#4

Updated by Robert Mustacchi 11 months ago

  • Related to Feature #10263: Update cpuid detection for new EPYC Socket format added
#5

Updated by Electric Monk 11 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 0ce813ffad7b20e510b276457d7615d97083b139

commit  0ce813ffad7b20e510b276457d7615d97083b139
Author: Robert Mustacchi <rm@joyent.com>
Date:   2019-01-30T04:47:29.000Z

    10277 Properly detect SMT on AMD
    Reviewed by: John Levon <john.levon@joyent.com>
    Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

#6

Updated by Marcel Telka 9 months ago

  • Related to Bug #10514: 10277 broke older Intel CPUs added

Also available in: Atom PDF