Project

General

Profile

Actions

Feature #12677

closed

simnet has bogus mi_tx_cksum_flags

Added by Patrick Mooney about 2 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
kernel
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
bhyve
Gerrit CR:

Description

Upstreaming OS-7904 for bhyve:

While working on another issue I noticed that mac_tx() was calling mac_hw_emul() when it shouldn't. Using the IP forwarding test suite and some dtrace I was able to determine why.

# NET_TESTS=/opt/net-tests /opt/net-tests/tests/forwarding/ip_forwarding -nfluv cea523d9-5e7e-4043-bbe8-fb920c3f8997 e9a97d65-b1e5-e77c-d573-f85ee5e0aa7c e9d88af9-352e-6419-ba42-88a25f2e967f

# dtrace -x dynvarsize=4m -qn 'mac_tx:entry { self->t = 1; this->mcip = (mac_client_impl_t *)arg0; self->mip = this->mcip->mci_mip; self->mci_name = stringof(this->mcip->mci_name); } mac_hw_emul:entry /self->t/ { this->mp = *args[0]; printf("emul [%s %s 0x%x (mp: 0x%x)] 0x%x %a\n", self->mci_name, self->mip->mi_name, self->mip->mi_tx_cksum_flags, this->mp->b_datap->db_struioun.cksum.flags, args[3], caller); stack(); } mac_tx_send:entry,mac_provider_tx:entry,mac_tx:return /self->t/ { self->t = 0; self->mci_name = 0; self->mip = 0; }' -o /var/tmp/emul.out

The resulting output includes entries like the following.

emul [ipft_server0 simnet1060 0x0 (mp: 0x5)] 0x3 mac`mac_tx+0x120

              mac`mac_tx+0x120
              dld`str_mdata_fastpath_put+0x53
              ip`ip_xmit+0x82d
              ip`ire_send_wire_v4+0x401
              ip`conn_ip_output+0x190
              ip`tcp_input_listener+0x12f1
              ip`squeue_drain+0x1c8
              ip`squeue_worker+0xbf
              unix`thread_start+0x8

emul [ipft_client_r0 simnet1060 0x0 (mp: 0x108)] 0x0 mac`mac_tx+0x120

              mac`mac_tx+0x120
              dld`str_mdata_fastpath_put+0x53
              ip`ip_xmit+0x82d
              ip`ip_forward_xmit_v4+0x116
              ip`ire_recv_forward_v4+0x5bd
              ip`ill_input_short_v4+0x4ee
              ip`ip_input_common_v4+0x3a7
              ip`ip_input+0x2b
              mac`mac_rx_soft_ring_process+0x19a
              mac`mac_rx_srs_proto_fanout+0x29a
              mac`mac_rx_srs_drain+0x363
              mac`mac_srs_worker+0x326
              unix`thread_start+0x8

Notice the hex value after the simnet name (in this case the name of the mac_impl_t) -- it's zero. This means the mi_tx_cksum_flags value is zero. That's the wrong value considering that I ran the full HW checksum + LSO test. The value should be 0x15. What does mdb show?

> ::walk mac_client_impl_cache | ::grep '*(.+118) != 0' | ::printf "0x%p %s [%s 0x%x] %s 0x%x\n" mac_client_impl_t . mci_name mci_upper_mip->mi_name mci_upper_mip->mi_tx_cksum_flags mci_mip->mi_name mci_mip->mi_tx_cksum_flags
0xfffffe5a776a31c0 ipft_server0 [vnic1064 0x15] simnet1060 0x0
0xfffffe5a776a3e30 ipft_client_r0 [vnic1062 0x15] simnet1060 0x0
0xfffffe594b276c80 ipft_client0 [vnic1061 0x15] simnet1059 0x0
0xfffffe594b2778f0 ipft_server_r0 [vnic1063 0x15] simnet1060 0x0

So the upper mip (the VNIC) has the correct value (0x15) but the lower mip (simnet) does not. Currently, mi_tx_cksum_flags is set at mac_register. But the HW offloads are enabled on simnet as private properties, AFTER it has already registered.

This doesn't affect real hardware (luckily) as the HW offloads are determined at time of registration. But it is still important to fix as simnet is being used to emulate real hardware and test things like IP forwarding. With this current behavior we are always emulating HW checksums, thus not accurately emulating the real thing, and potentially not catching bugs.

For testing I repeated the network test mentioned in the original report and used mdb to verify the simnet's mi_tx_cksum_flags were updated.

> ::walk mac_client_impl_cache | ::grep '*(.+118) != 0' | ::printf "0x%p %s [%s 0x%x] %s 0x%x\n" mac_client_impl_t . mci_name mci_upper_mip->mi_name mci_upper_mip->mi_tx_cksum_flags mci_mip->mi_name mci_mip->mi_tx_cksum_flags ! grep simnet
0xfffffe5a5f900e30 ipft_server_r0 [vnic1075 0x15] simnet1072 0x15
0xfffffe59410b0c78 ipft_client_r0 [vnic1074 0x15] simnet1072 0x15
0xfffffe5a5f8fdc70 ipft_server0 [vnic1076 0x15] simnet1072 0x15
0xfffffe5a5f8ff550 ipft_client0 [vnic1073 0x15] simnet1071 0x15

I also ran the entire network test suite to confirm no regressions.

# /opt/net-tests/bin/nettest 
Test: /opt/net-tests/tests/forwarding/ip_fwd_no_cksum (run as root) [00:31] [PASS]
Test: /opt/net-tests/tests/forwarding/ip_fwd_partial_cksum (run as root) [00:31] [PASS]
Test: /opt/net-tests/tests/forwarding/ip_fwd_full_cksum (run as root) [00:32] [PASS]
Test: /opt/net-tests/tests/forwarding/ip_fwd_partial_cksum_lso (run as root) [00:32] [PASS]
Test: /opt/net-tests/tests/forwarding/ip_fwd_full_cksum_lso (run as root) [00:32] [PASS]

Results Summary
PASS       5

Running Time:   00:02:40
Percent passed: 100.0%

Related issues

Related to illumos gate - Feature #12678: mac_tx() is too eager to emulate hardware offloadsClosedRyan Zezeski

Actions
Actions #1

Updated by Patrick Mooney about 2 years ago

  • Related to Feature #12678: mac_tx() is too eager to emulate hardware offloads added
Actions #2

Updated by Electric Monk about 2 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit c61a1653a4d73dbc950dac7d96350fd6cb517486

commit  c61a1653a4d73dbc950dac7d96350fd6cb517486
Author: Ryan Zezeski <rpz@joyent.com>
Date:   2020-05-18T18:37:51.000Z

    12676 want better offloads for vnics
    12677 simnet has bogus mi_tx_cksum_flags
    12678 mac_tx() is too eager to emulate hardware offloads
    Portions contributed by: Patrick Mooney <patrick.mooney@joyent.com>
    Portions contributed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Patrick Mooney <pmooney@oxide.computer>
    Reviewed by: Andy Fiddaman <andy@omniosce.org>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF