illumos gate

1. Why is the limit reached?
2. The system does not need to panic.

The limit is reached because mlxcx has a default of 64 rings per group, so after 2 links in the aggr MAX_RINGS_PER_GROUP is reached. That default is too high and will be addressed in another bug report.

The code in aggr_grp_create() needs to be fixed to remove the VERIFY's.
This snippet:

        /*
         * Attach each port if necessary.
         */
        for (port = grp->lg_ports; port != NULL; port = port->lp_next) {
                /*
                 * Create the pseudo ring for each HW ring of the
                 * underlying port. Note that this is done after the
                 * aggr registers its MAC.
                 */
                VERIFY3S(aggr_add_pseudo_tx_group(port, &grp->lg_tx_group),
                    ==, 0);

                for (i = 0; i < grp->lg_rx_group_count; i++) {
                        VERIFY3S(aggr_add_pseudo_rx_group(port,
                            &grp->lg_rx_groups[i]), ==, 0);
                }

                if (aggr_port_notify_link(grp, port))
                        link_state_changed = B_TRUE;

                /*
                 * Initialize the callback functions for this port.
                 */
                aggr_port_init_callbacks(port);
        }

I tested by creating an aggr with too many mlxcx links and (before fixing #13208), dladm now fails:

# dladm create-aggr -t  -l mlxcx0 -l mlxcx1 -l mlxcx2 aggr0
dladm: create operation failed: I/O error

I also hard code an error to simulate the failure at what was the 2nd VERIFY to confirm that failure path worked.

Creating aggr with 2 mlxcx was succesful, and aggr with more than 2 of other NIC varieties was successful