Project

General

Profile

Actions

Bug #16413

closed

Post-barrier Return Stack Buffer (PBRSB) fixes can be detected in HW

Added by Dan McDonald about 1 month ago. Updated 17 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
kernel
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

While preparing #16397, I noticed that in this public Intel doc:

https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/post-barrier-return-stack-buffer-predictions.html

that there is an indicator like RFDS has for CPUs that no longer have the PBRSB problem: "PBRSB_NO". This bug's fix will make sure x86_rsb_stuff() remains an instant-return in those cases.

ALSO the same document says: "The VM exit behavior does not apply to non-eIBRS processors." One might wish to have the vm_exit path not stuff the RSB on pre-eIBRS processors (while still maintaining stuffing in other cases). If we opt to do that we may, once again, have a second entry point, if not an whole implementation, for `x86_rsb_stuff_vmexit()`. On the other hand nobody has complained about HVM performance on such processors (Skylake and earlier), so the KISS principle may be best served by only acting on PBRSB_NO and maintaining the single entry point x86_rsb_stuff().

This issue is considered a Bug, because it could've been addressed as part of #14902 .


Related issues

Related to illumos gate - Bug #14902: Have Intel vm_exit paths guard against Post-Barrier RSB PredictionsClosedDan McDonald

Actions
Actions #1

Updated by Dan McDonald about 1 month ago

  • Related to Bug #14902: Have Intel vm_exit paths guard against Post-Barrier RSB Predictions added
Actions #2

Updated by Dan McDonald about 1 month ago

  • Description updated (diff)
Actions #3

Updated by Dan McDonald about 1 month ago

  • Difficulty changed from Hard to Medium
Actions #4

Updated by Dan McDonald 30 days ago

ALSO the same document says: "The VM exit behavior does not apply to non-eIBRS processors." One might wish to have the vm_exit path not stuff the RSB on pre-eIBRS processors (while still maintaining stuffing in other cases). If we opt to do that we may, once again, have a second entry point, if not an whole implementation, for `x86_rsb_stuff_vmexit()`. On the other hand nobody has complained about HVM performance on such processors (Skylake and earlier), so the KISS principle may be best served by only acting on PBRSB_NO and maintaining the single entry point x86_rsb_stuff().

Turns out if you honor the language, and use lack-of-eIBRS as an indicator, it's not too difficult to have VMEXIT not stuff the RSB, but still have context switches do the stuffing. There's code (as two commits so no Gerrit for now) at https://kebe.com/~danmcd/webrevs/16413/ and on Gerrit that does just that, by using dual entry points to x86_rsb_stuff() per my earlier paragraph.

Actions #5

Updated by Electric Monk 27 days ago

  • Gerrit CR set to 3384
Actions #6

Updated by Dan McDonald 27 days ago

Testing notes:

This single command line:

(echo x86_rsb_stuff_vmexit/4I ; echo x86_rsb_stuff/3I ; echo spec_uarch_flush/P ; echo x86_spectrev2_mitigation::print ) | \
     mdb -k ; mdb -ke "::x86_featureset" | egrep "pbrsb|rfds" 

Will print indicators of how the machine behaves after this patch:

x86_rsb_stuff_vmexit: RET if no need for VMEXIT RSB stuffing, NOP if there is a need.

x86_rsb_stuff: RET if no need for context-switching RSB stuffing, NOP if there is a need.

(IMPORTANT: There are no known cases today where VMEXIT stuffing is needed but context-switching is not. This is VERIFYed in the code. If I'm wrong, x86_rsb_stuff* functions will need to be revisited slightly.)

spec_uarch_flush: Holdover from 16397,. where it is enabled on certain low-power modern Intel CPUs (e.g. E-core on Alder Lake).

The default SPECTREv2 mitigation is also printed, as well as what CPU indicators/enumerations for RFDS and PBRSB.

HASWELL E (has no eIBRS, so needs no VMEXIT RSB flushing but needs context-switch RSB flushing, too old for RFDS or PBRSB):

[root@curly (kebecloud) ~]# (echo x86_rsb_stuff_vmexit/4I ; echo x86_rsb_stuff/3I ; echo spec_uarch_flush/P ; echo x86_spectrev2_mitigation::print ) | mdb -k ; mdb -ke "::x86_featureset" | egrep "pbrsb|rfds" 
x86_rsb_stuff_vmexit:
x86_rsb_stuff_vmexit:           fffffffffb886e30 505790c3: ret    
fffffffffb886e31 bf505790: nop    
fffffffffb886e32 10bf5057: pushq  %rdi
fffffffffb886e33 10bf50: pushq  %rax
x86_rsb_stuff:
x86_rsb_stuff:  fffffffffb886e31 bf505790: nop    
fffffffffb886e32 10bf5057: pushq  %rdi
fffffffffb886e33 10bf50: pushq  %rax
spec_uarch_flush:
spec_uarch_flush:               spec_uarch_flush_msr
0 (X86_SPECTREV2_RETPOLINE)
[root@curly (kebecloud) ~]# 

TIGER LAKE (has eIBRS, but is vulnerable to PBRSB. Immune from RFDS):

[root@nuc ~]# (echo x86_rsb_stuff_vmexit/4I ; echo x86_rsb_stuff/3I ; echo spec_uarch_flush/P ; echo x86_spectrev2_mitigation::print ) | mdb -k ; mdb -ke "::x86_featureset" | egrep "pbrsb|rfds" 
x86_rsb_stuff_vmexit:
x86_rsb_stuff_vmexit:           fffffffffb886e30 50579090: nop    
fffffffffb886e31 bf505790: nop    
fffffffffb886e32 10bf5057: pushq  %rdi
fffffffffb886e33 10bf50: pushq  %rax
x86_rsb_stuff:
x86_rsb_stuff:  fffffffffb886e31 bf505790: nop    
fffffffffb886e32 10bf5057: pushq  %rdi
fffffffffb886e33 10bf50: pushq  %rax
spec_uarch_flush:
spec_uarch_flush:               spec_uarch_flush_noop
1 (X86_SPECTREV2_ENHANCED_IBRS)
rfds_no
[root@nuc ~]# 

ALDER LAKE N (E-core) (has eIBRS, vulnerable to RFDS, NOT vulnerable to PBRSB):

[root@alder-lake ~]# (echo x86_rsb_stuff_vmexit/4I ; echo x86_rsb_stuff/3I ; echo spec_uarch_flush/P ; echo x86_spectrev2_mitigation::print ) | mdb -k ; mdb -ke "::x86_featureset" | egrep "pbrsb|rfds" 
x86_rsb_stuff_vmexit:
x86_rsb_stuff_vmexit:           fffffffffb886e30 5057c3c3: ret    
fffffffffb886e31 bf5057c3: ret    
fffffffffb886e32 10bf5057: pushq  %rdi
fffffffffb886e33 10bf50: pushq  %rax
x86_rsb_stuff:
x86_rsb_stuff:  fffffffffb886e31 bf5057c3: ret    
fffffffffb886e32 10bf5057: pushq  %rdi
fffffffffb886e33 10bf50: pushq  %rax
spec_uarch_flush:
spec_uarch_flush:               x86_md_clear       
1 (X86_SPECTREV2_ENHANCED_IBRS)
rfds_clear
pbrsb_no
[root@alder-lake ~]# 

The HASWELL E machine runs BHYVE VMs in production. The ROCKET LAKE and ALDER LAKE N will be running small BHYVE VMs for testing of a different sort, so coverage here is a pleasant side-effect, and will be forthcoming.

Actions #7

Updated by Dan McDonald 27 days ago

Dan McDonald wrote in #note-6:

SNIP!

The HASWELL E machine runs BHYVE VMs in production. The TIGER LAKE and ALDER LAKE N will be running small BHYVE VMs for testing of a different sort, so coverage here is a pleasant side-effect, and will be forthcoming.

Both of the non-production machines are now running BHYVE VMs:

kebe(~)[0]% ssh root@nuc "psrinfo -vp | tail -3 ; vmadm list" 
(root@nuc) Password: 
  The core has 2 virtual processors (3 7)
    x86 (GenuineIntel 806C1 family 6 model 140 step 1 clock 2420 MHz)
      11th Gen Intel(r) Core(tm) i5-1135G7 @ 2.40GHz
UUID                                  TYPE  RAM      STATE             ALIAS
170e57f0-4ffe-4fad-af11-c235d04ee16d  BHYV  8192     running           ubuntu-22
d00f8241-409b-4b1c-8de6-546d4c16410f  OS    16384    running           nuc-build
kebe(~)[0]% ssh root@alder-lake "psrinfo -vp | tail -3 ; vmadm list" 
(root@alder-lake) Password: 
The physical processor has 4 virtual processors (0-3)
  x86 (GenuineIntel B06E0 family 6 model 190 step 0 clock 807 MHz)
    Intel(r) N100
UUID                                  TYPE  RAM      STATE             ALIAS
22b9b253-1b21-4835-b66b-8063121efec7  BHYV  4096     running           debian12
kebe(~)[0]% 

Actions #8

Updated by Electric Monk 17 days ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit a6e309ba2366a72510e1b1cfbcad5ea2c531f49d

commit  a6e309ba2366a72510e1b1cfbcad5ea2c531f49d
Author: Dan McDonald <danmcd@mnx.io>
Date:   2024-04-04T19:28:17.000Z

    16413 Post-barrier Return Stack Buffer (PBRSB) fixes can be detected in HW
    Reviewed by: Robert Mustacchi <rm@fingolfin.org>
    Reviewed by: Richard Lowe <richlowe@richlowe.net>
    Approved by: Gordon Ross <Gordon.W.Ross@gmail.com>

Actions

Also available in: Atom PDF