Project

General

Profile

Actions

Bug #14581

closed

ipadm could not create address after delete-addr for addrconf

Added by Ryan Zezeski 5 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
networking
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Issue 13316 probably has the same underlying cause as this issue, but they look different enough to the user that I think it's helpful to file this one as well to make searching easier.

Summary

A new IPv6 addrconf address cannot be created for an interface that previously already had an addrconf address but it has been deleted. That's a mouthful, the steps below make it more clear what's going on.

Work around

You have two options:

  1. Delete the underlying interface (via ipadm delete-if), which will reset the ipmgmtd state.
  2. Restart the ip-interface-management service.

Steps to reproduce

1. Create an addrconf address on an interface. In this case I'm using a simnet device as the underlying device should have no bearing on this bug.

rpz@kalm:~$ dladm show-link ryan0
LINK        CLASS     MTU    STATE    BRIDGE     OVER
ryan0       simnet    1500   up       --         --

rpz@kalm:~$ pfexec ipadm create-addr -t -T addrconf ryan0/v6

rpz@kalm:~$ ipadm show-addr ryan0/v6
ADDROBJ           TYPE     STATE        ADDR
ryan0/v6          addrconf ok           fe80::7822:91ff:fed3:9634/10

2. Delete the address just created.

rpz@kalm:~$ pfexec ipadm delete-addr ryan0/v6

rpz@kalm:~$ ipadm show-addr ryan0/v6
ipadm: Address object not found

3. Attempt to recreate the address.

rpz@kalm:~$ pfexec ipadm create-addr -t -T addrconf ryan0/v6
ipadm: Could not create address: Address object already exists

Additional debugging context

1. The state of ipmgmtd before adding the addrconf address.

rpz@kalm:~$ pfexec mdb -p $(pgrep ipmgmtd)
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libnvpair.so.1 libuutil.so.1 ]

> aobjmap::print -t ipmgmt_aobjmap_list_t aobjmap_head | ::list ipmgmt_aobjmap_t am_next | ::printf "0x%p %s\n" ipmgmt_aobjmap_t . am_aobjname
0x80cf948 e1000g0/v4
0x80cfac8 e1000g0/v6
0x80cfc48 lo0/v6
0x80cfdc8 lo0/v4

2. Create the address and view the state again.

rpz@kalm:~$ pfexec ipadm create-addr -t -T addrconf ryan0/v6

rpz@kalm:~$ ipadm show-addr ryan0/v6
ADDROBJ           TYPE     STATE        ADDR
ryan0/v6          addrconf ok           fe80::7822:91ff:fed3:9634/10

rpz@kalm:~$ pfexec mdb -p $(pgrep ipmgmtd)
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libnvpair.so.1 libuutil.so.1 ]

> aobjmap::print -t ipmgmt_aobjmap_list_t aobjmap_head | ::list ipmgmt_aobjmap_t am_next | ::printf "0x%p %s\n" ipmgmt_aobjmap_t . am_aobjname
0x80cf648 ryan0/v6
0x80cf7c8 ryan0/v6
0x80cf948 e1000g0/v4
0x80cfac8 e1000g0/v6
0x80cfc48 lo0/v6
0x80cfdc8 lo0/v4

3. Interesting, there are two ryan0/v6 entries. What are their values?

> 0x80cf648::print -t ipmgmt_aobjmap_t
ipmgmt_aobjmap_t {
    struct ipmgmt_aobjmap_s *am_next = 0x80cf7c8
    char [64] am_aobjname = [ "ryan0/v6" ]
    char [32] am_ifname = [ "ryan0" ]
    int32_t am_lnum = 0
    sa_family_t am_family = 0x1a
    ipadm_addr_type_t am_atype = 2 (IPADM_ADDR_IPV6_ADDRCONF)
    uint32_t am_nextnum = 0
    uint32_t am_flags = 0x1
    ipmgmt_addr_type_cache_u am_atype_cache = {
        struct  ipmgmt_ipv6_cache_s = {
            boolean_t ipmgmt_linklocal = 0x1 (B_TRUE)
            struct sockaddr_in6 ipmgmt_ifid = {
                sa_family_t sin6_family = 0
                in_port_t sin6_port = 0
                uint32_t sin6_flowinfo = 0
                struct in6_addr sin6_addr = ::
                uint32_t sin6_scope_id = 0
                uint32_t __sin6_src_id = 0
            }
        }
        struct  ipmgmt_dhcp_cache_s = {
            char [256] ipmgmt_reqhost = [ '\001', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', ... ]
        }
    }
}

> 0x80cf7c8::print -t ipmgmt_aobjmap_t
ipmgmt_aobjmap_t {
    struct ipmgmt_aobjmap_s *am_next = 0x80cf948
    char [64] am_aobjname = [ "ryan0/v6" ]
    char [32] am_ifname = [ "ryan0" ]
    int32_t am_lnum = 0
    sa_family_t am_family = 0x1a
    ipadm_addr_type_t am_atype = 2 (IPADM_ADDR_IPV6_ADDRCONF)
    uint32_t am_nextnum = 0
    uint32_t am_flags = 0x1
    ipmgmt_addr_type_cache_u am_atype_cache = {
        struct  ipmgmt_ipv6_cache_s = {
            boolean_t ipmgmt_linklocal = 0 (0)
            struct sockaddr_in6 ipmgmt_ifid = {
                sa_family_t sin6_family = 0
                in_port_t sin6_port = 0
                uint32_t sin6_flowinfo = 0
                struct in6_addr sin6_addr = ::
                uint32_t sin6_scope_id = 0
                uint32_t __sin6_src_id = 0
            }
        }
        struct  ipmgmt_dhcp_cache_s = {
            char [256] ipmgmt_reqhost = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', ... ]
        }
    }
}

4. One is marked ipmgmt_linklocal and the other is not. What happens when I run delete-addr?

rpz@kalm:~$ pfexec ipadm delete-addr ryan0/v6

rpz@kalm:~$ ipadm show-addr ryan0/v6
ipadm: Address object not found

rpz@kalm:~$ pfexec mdb -p $(pgrep ipmgmtd)
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libnvpair.so.1 libuutil.so.1 ]
> aobjmap::print -t ipmgmt_aobjmap_list_t aobjmap_head | ::list ipmgmt_aobjmap_t am_next | ::printf "0x%p %s\n" ipmgmt_aobjmap_t . am_aobjname
0x80cf7c8 ryan0/v6
0x80cf948 e1000g0/v4
0x80cfac8 e1000g0/v6
0x80cfc48 lo0/v6
0x80cfdc8 lo0/v4

> 0x80cf7c8::print -t ipmgmt_aobjmap_t
ipmgmt_aobjmap_t {
    struct ipmgmt_aobjmap_s *am_next = 0x80cf948
    char [64] am_aobjname = [ "ryan0/v6" ]
    char [32] am_ifname = [ "ryan0" ]
    int32_t am_lnum = 0
    sa_family_t am_family = 0x1a
    ipadm_addr_type_t am_atype = 2 (IPADM_ADDR_IPV6_ADDRCONF)
    uint32_t am_nextnum = 0
    uint32_t am_flags = 0x1
    ipmgmt_addr_type_cache_u am_atype_cache = {
        struct  ipmgmt_ipv6_cache_s = {
            boolean_t ipmgmt_linklocal = 0 (0)
            struct sockaddr_in6 ipmgmt_ifid = {
                sa_family_t sin6_family = 0
                in_port_t sin6_port = 0
                uint32_t sin6_flowinfo = 0
                struct in6_addr sin6_addr = ::
                uint32_t sin6_scope_id = 0
                uint32_t __sin6_src_id = 0
            }
        }
        struct  ipmgmt_dhcp_cache_s = {
            char [256] ipmgmt_reqhost = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', ... ]
        }
    }
}

5. The instance NOT marked linklocal is still there. This entry is found when trying to create-addr again, causing ipadm to claim it already exists; confirmed by this truss output.

rpz@kalm:~$ pfexec truss -t \!all -u libipadm::ipadm_create_addr,i_ipadm_lookupadd_addrobj,i_ipadm_create_ipv6addrs ipadm create-addr -t -T addrconf ryan0/v6
/1@1:   -> libipadm:ipadm_create_addr(0x8f79e18, 0x8f79ea0, 0x3002, 0xfec1194e)
/1@1:     -> libipadm:i_ipadm_lookupadd_addrobj(0x8f79e18, 0x8f79ea0, 0x3002, 0xfef5192f)
/1@1:     <- libipadm:i_ipadm_lookupadd_addrobj() = 11
/1@1:   <- libipadm:ipadm_create_addr() = 11
ipadm: Could not create address: Address object already exists

6. Looking into the ipadm code reveals something interesting. The ipadm return code of 11 matches up with IPADM_IF_EXISTS. This indicates that it's complaining that the interface already exist. Which, of course it does, because we only deleted the address object. So perhaps the bug is the fact that it bails if the interface already exist. Why wouldn't it just confirm it already exists and go on to creating the address object? Especially given that for most cases there will also be an IPv4 address object on this interface, and there is no way to selectively delete either just the v4 or v6 portion of the interface (via delete-if); users cannot delete this interface if they rely on it for v4 comms.

    IPADM_IF_EXISTS,    /* Interface already exists */

Related issues

Related to illumos gate - Bug #13316: ipmgmtd inconsistent with kernel on failureNew

Actions
Actions #1

Updated by Ryan Zezeski 5 months ago

  • Related to Bug #13316: ipmgmtd inconsistent with kernel on failure added
Actions #2

Updated by Ryan Goodfellow 3 months ago

I believe I've found the issue here.

When adding an object to the global aobjmap there is special handling of link-local addresses. This handling requires that stub nodes have ipmgmt_am_linklocal set to true. But the code that handles creating stub/placeholder nodes does not set ipmgmt_am_linklocal. This means that ipmgmt_aobjmap_op will create a new node instead of updating the existing one. Then when the address object is deleted we have two, but only one gets deleted leading to the behavior observed in this issue.

I have a patch that appears to be working that I'll submit shortly.

Actions #3

Updated by Electric Monk 3 months ago

  • Gerrit CR set to 2158
Actions #4

Updated by Ryan Goodfellow 3 months ago

Testing Notes:

Demonstrating the Bug

On a current illumos system do the following.

Create a simnet device with an addrconf address.

$ pfexec dladm create-simnet sim0
$ pfexec ipadm create-addr -t -T addrconf sim0/v6

The debugging technique rpz noted in the initial report shows that two entries were created consistent with the root cause identified in my previous note.

$ pfexec mdb -p $(pgrep ipmgmtd)
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libnvpair.so.1 libuutil.so.1 ]
> aobjmap::print -t ipmgmt_aobjmap_list_t aobjmap_head | ::list ipmgmt_aobjmap_t am_next | ::printf "0x%p %s\n" ipmgmt_aobjmap_t . am_aobjname
0x81d37c8 sim0/v6
0x81d3948 sim0/v6
0x81d3ac8 igb0/dhcp
0x81d3c48 lo0/v6
0x81d3dc8 lo0/v4

Delete the address

$ pfexec ipadm delete-addr sim0/v6

ipadm shows that it appears to be gone

$ ipadm
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
igb0/dhcp         dhcp     ok           10.47.0.24/24
lo0/v6            static   ok           ::1/128

Then try to create the address again

$ pfexec ipadm create-addr -t -T addrconf sim0/v6
ipadm: Could not create address: Address object already exists

And we also see the additional address hanging around in mdb.

$ pfexec mdb -p $(pgrep ipmgmtd)
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libnvpair.so.1 libuutil.so.1 ]
> aobjmap::print -t ipmgmt_aobjmap_list_t aobjmap_head | ::list ipmgmt_aobjmap_t am_next | ::printf "0x%p %s\n" ipmgmt_aobjmap_t . am_aobjname
0x81d3948 sim0/v6
0x81d3ac8 igb0/dhcp
0x81d3c48 lo0/v6
0x81d3dc8 lo0/v4

Demonstrating the Fix

On a patched illumos system following the same procedure.

Create a simnet device with an addrconf address.

$ pfexec dladm create-simnet sim0
$ pfexec ipadm create-addr -t -T addrconf sim0/v6

Now mdb only shows one address for sim0/v6

# mdb -p $(pgrep ipmgmtd)
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libnvpair.so.1 libuutil.so.1 ]
> aobjmap::print -t ipmgmt_aobjmap_list_t aobjmap_head | ::list ipmgmt_aobjmap_t am_next | ::printf "0x%p %s\n" ipmgmt_aobjmap_t . am_aobjname
0x80df948 sim0/v6
0x80dfac8 vioif0/v4
0x80dfc48 lo0/v6
0x80dfdc8 lo0/v4

Delete the address

$ pfexec ipadm delete-addr sim0/v6

ipadm shows that it appears to be gone

# ipadm
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
vioif0/v4         dhcp     ok           10.47.0.151/24
lo0/v6            static   ok           ::1/128

mdb shows no addresses for sim0/v6

# mdb -p $(pgrep ipmgmtd)
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libnvpair.so.1 libuutil.so.1 ]
> aobjmap::print -t ipmgmt_aobjmap_list_t aobjmap_head | ::list ipmgmt_aobjmap_t am_next | ::printf "0x%p %s\n" ipmgmt_aobjmap_t . am_aobjname
0x80dfac8 vioif0/v4
0x80dfc48 lo0/v6
0x80dfdc8 lo0/v4

and finally we can create the address again without issue

# pfexec ipadm create-addr -t -T addrconf sim0/v6
# ipadm
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
vioif0/v4         dhcp     ok           10.47.0.151/24
lo0/v6            static   ok           ::1/128
sim0/v6           addrconf ok           fe80::8ce2:4bff:fe32:99c3/10
Actions #5

Updated by Electric Monk 3 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 662993c9b2fd7e3e0a5c2e535902c2360206095a

commit  662993c9b2fd7e3e0a5c2e535902c2360206095a
Author: Ryan Goodfellow <ryan.goodfellow@oxide.computer>
Date:   2022-05-25T21:04:54.000Z

    14581 ipadm could not create address after delete-addr for addrconf
    Reviewed by: C Fraire <cfraire@me.com>
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Reviewed by: Ryan Zezeski <ryan@oxide.computer>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF