Project

General

Profile

Actions

Feature #13689

closed

Want AWS ENA driver

Added by Ryan Zezeski 8 months ago. Updated 15 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
networking
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Hard
Tags:
Gerrit CR:

Description

Late last summer I wrote a preliminary driver for the AWS Nitro ENA network device (Elastic Network Adapter). Support for this virtual device allows illumos-based distros to run on the "newer" AWS Nitro instances, which offer much beefier configurations than the older Xen-based instances. I have a working driver with good throughput (even without doing the typical performance tricks required for higher rate NICs), but I'm still working on some polish before I put it up for CR. I just wanted to add this placeholder in to let others know this work is upcoming.


Related issues

Related to illumos gate - Bug #14193: PCI config space I/O port mechanisms need to check offsetsClosedRobert Mustacchi

Actions
Actions #1

Updated by Electric Monk about 2 months ago

  • Gerrit CR set to 1763
Actions #2

Updated by Ryan Zezeski about 2 months ago

  • Subject changed from want AWS ENA driver to Want AWS ENA driver
Actions #3

Updated by Ryan Zezeski 20 days ago

Testing Notes

  • Verified this driver sends/receives traffic on Nitro/ENA instances t3.small (up to 5 Gbits/s) as well as m5n.4xlarge (up to 25 Gbits/s).
  • Verified kstats work.
    oot@ip-172-31-12-216:~# kstat -p ena | grep packets
    ena:0:device_basic:rx_packets   6488542
    ena:0:device_basic:tx_packets   6076778
    ena:0:mac:ipackets      6488542
    ena:0:mac:ipackets64    6488542
    ena:0:mac:opackets      6076778
    ena:0:mac:opackets64    6076778
    ena:0:rxq_0:packets     6488542
    ena:0:rxq_1:packets     0
    ena:0:rxq_2:packets     0
    ena:0:rxq_3:packets     0
    ena:0:rxq_4:packets     0
    ena:0:rxq_5:packets     0
    ena:0:rxq_6:packets     0
    ena:0:txq_0:packets     821296
    ena:0:txq_1:packets     789069
    ena:0:txq_2:packets     793039
    ena:0:txq_3:packets     335756
    ena:0:txq_4:packets     628169
    ena:0:txq_5:packets     1678310
    ena:0:txq_6:packets     1031139
    
  • Verified dladm works.
    root@ip-172-31-12-216:~# dladm show-link                           
    LINK        CLASS     MTU    STATE    BRIDGE     OVER               
    ena0        phys      9216   up       --         -- 
    
    root@ip-172-31-12-216:~# dladm show-linkprop ena0                   
    LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE                                                                            
    ena0         speed           r-   0              0              --  
    ena0         autopush        rw   --             --             --      
    ena0         zone            rw   --             --             --                                                                                  
    ena0         duplex          r-   unknown        unknown        half,full  
    ena0         state           r-   up             up             up,down                                                                             
    ena0         adv_autoneg_cap rw   0              0              1,0                                                                                 
    ena0         mtu             rw   9216           1500           9216 
    ena0         flowctrl        --   --             no             no,tx,rx,bi 
    ena0         adv_fec_cap     r-   --             r-             none,auto,rs,base-r 
    ena0         en_fec_cap      --   --             --             none,auto,rs,base-r 
    ena0         adv_100gfdx_cap r-   0              0              1,0                                                                                 
    ena0         en_100gfdx_cap  rw   0              0              1,0 
    ena0         adv_50gfdx_cap  r-   --             0              1,0 
    ena0         en_50gfdx_cap   --   --             0              1,0 
    ena0         adv_40gfdx_cap  r-   0              0              1,0 
    ena0         en_40gfdx_cap   rw   0              0              1,0 
    ena0         adv_25gfdx_cap  r-   0              0              1,0 
    ena0         adv_10gfdx_cap  r-   0              0              1,0 
    ena0         en_10gfdx_cap   rw   0              0              1,0 
    ena0         adv_5000fdx_cap r-   0              0              1,0 
    ena0         en_5000fdx_cap  rw   0              0              1,0 
    ena0         adv_2500fdx_cap r-   0              0              1,0 
    ena0         en_2500fdx_cap  rw   0              0              1,0 
    ena0         adv_1000fdx_cap r-   1              0              1,0 
    ena0         en_1000fdx_cap  rw   1              0              1,0 
    ena0         adv_1000hdx_cap r-   --             0              1,0 
    ena0         en_1000hdx_cap  --   --             0              1,0 
    ena0         adv_100fdx_cap  r-   --             0              1,0 
    ena0         en_100fdx_cap   --   --             0              1,0 
    ena0         adv_100hdx_cap  r-   --             0              1,0 
    ena0         en_100hdx_cap   --   --             0              1,0 
    ena0         adv_10fdx_cap   r-   --             0              1,0 
    ena0         en_10fdx_cap    --   --             0              1,0 
    ena0         adv_10hdx_cap   r-   --             0              1,0 
    ena0         en_10hdx_cap    --   --             0              1,0 
    ena0         maxbw           rw   --             --             -- 
    ena0         cpus            rw   --             --             -- 
    ena0         cpus-effective  r-   0-15           --             -- 
    ena0         pool            rw   --             --             -- 
    ena0         pool-effective  r-   --             --             -- 
    ena0         priority        rw   high           high           low,medium,high 
    ena0         tagmode         rw   vlanonly       vlanonly       normal,vlanonly 
    ena0         forward         rw   1              1              1,0 
    ena0         default_tag     rw   1              1              -- 
    ena0         learn_limit     rw   1000           1000           -- 
    ena0         learn_decay     rw   200            200            -- 
    ena0         stp             rw   1              1              1,0 
    ena0         stp_priority    rw   128            128            -- 
    ena0         stp_cost        rw   auto           auto           -- 
    ena0         stp_edge        rw   1              1              1,0 
    ena0         stp_p2p         rw   auto           auto           true,false,auto 
    ena0         stp_mcheck      rw   0              0              1,0 
    LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
    ena0         protection      rw   --             --             mac-nospoof,
                                                                    restricted,
                                                                    ip-nospoof,
                                                                    dhcp-nospoof 
    ena0         allowed-ips     rw   --             --             -- 
    ena0         allowed-dhcp-cids rw --             --             -- 
    ena0         rxrings         rw   --             --             -- 
    ena0         rxrings-effective r- 7              --             -- 
    ena0         txrings         rw   --             --             sw,hw,<1-6> 
    ena0         txrings-effective r- 7              --             -- 
    ena0         txrings-available r- 6              --             -- 
    ena0         rxrings-available r- 0              --             -- 
    ena0         rxhwclnt-available r- 0             --             -- 
    ena0         txhwclnt-available r- 6             --             -- 
    
    root@ip-172-31-12-216:~# dladm show-phys
    LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
    ena0         Ethernet             up         0      unknown   ena0
    
    root@ip-172-31-12-216:~# dladm show-phys -H ena0
    LINK         RINGTYPE RINGS                CLIENTS
    ena0         RX       0-6                  <default,mcast>,ena0 
    ena0         TX       0-6                  <default>,ena0 
    

Perf

t3.small (1 thread)

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  4.61 GBytes  3.96 Gbits/sec                  sender
[  4]   0.00-10.00  sec  4.61 GBytes  3.96 Gbits/sec                  receiver

m5n.4xlarge (1 thread, Linux -> illumos)

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.53 GBytes  7.33 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  8.53 GBytes  7.33 Gbits/sec                  receiver

m5n.4xlarge (4 threads, Linux -> illumos)

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  7.07 GBytes  6.08 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  7.07 GBytes  6.08 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  7.30 GBytes  6.27 Gbits/sec    0             sender
[  6]   0.00-10.00  sec  7.30 GBytes  6.27 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  7.03 GBytes  6.04 Gbits/sec    0             sender
[  8]   0.00-10.00  sec  7.03 GBytes  6.04 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  7.37 GBytes  6.33 Gbits/sec    0             sender
[ 10]   0.00-10.00  sec  7.37 GBytes  6.33 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.8 GBytes  24.7 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  28.8 GBytes  24.7 Gbits/sec                  receiver

m5n.4xlarge (1 thread, illumos -> Linux)

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  10.9 GBytes  9.32 Gbits/sec                  sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.32 Gbits/sec                  receiver

m5n.4xlarge (4 threads, illumos -> Linux)

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.00 GBytes  2.58 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.00 GBytes  2.58 Gbits/sec                  receiver
[  6]   0.00-10.00  sec  2.95 GBytes  2.53 Gbits/sec                  sender
[  6]   0.00-10.00  sec  2.95 GBytes  2.53 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  2.91 GBytes  2.50 Gbits/sec                  sender
[  8]   0.00-10.00  sec  2.91 GBytes  2.50 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  2.87 GBytes  2.47 Gbits/sec                  sender
[ 10]   0.00-10.00  sec  2.87 GBytes  2.47 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  11.7 GBytes  10.1 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  11.7 GBytes  10.1 Gbits/sec                  receiver

Actions #4

Updated by Electric Monk 15 days ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 6f443ebc1fb4fec01d6e8fa8ca4648182ed215bb

commit  6f443ebc1fb4fec01d6e8fa8ca4648182ed215bb
Author: Ryan Zezeski <ryan@zinascii.com>
Date:   2021-11-23T18:18:50.000Z

    13689 Want AWS ENA driver
    Reviewed by: Robert Mustacchi <rm@fingolfin.org>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions #5

Updated by Ryan Zezeski 15 days ago

  • Related to Bug #14193: PCI config space I/O port mechanisms need to check offsets added
Actions

Also available in: Atom PDF