illumos gate

Believe it or not, this is functioning as designed -- but it's
certainly confusing if you don't know what's going on. And dladm's
interface is lacking the necessary information to understand what is
happening.

The dlmgmt daemon, which does the real work on behalf of dladm, has
the notion of zone-namespaced links -- i.e., a link's fully qualified
name is its zone identifier and vanity name (the vanity name is what
you see in dladm). When you boot the zone the link is "loaned" to the
zone and dlmgmtd changes the link's zoneid field to the id of the zone
that owns it. Then, when you create the second instance of the same
name dlmgmtd searches the GZ namespace for the link name and sees no
matches -- so it creates another. At this point you have two links
with the same name but assigned to different zones. However, dladm
doesn't really communicate this by say printing a ZONE column when
running show-link. There is a zone linkprop but if you show-linkprop I
think it only shows the link in the GZ. You can show-linkprop in the
NGZ but it doesn't seem to have the rights to view the zone property.

Basically, it feels like some of this work was still in an unfinished
state from the time illumos forked from OpenSolaris. There has been
some additional work done in SmartOS to better support namespaced
links but I'm not sure if a) it's complete yet or b) if it's the right
way to do it (I don't mean that in any negative way, simply that
there's probably a few ways to get this job done).

Anyways, I've been working on dladm and dlmgmtd in SmartOS lately. It
would be nice to make this less confusing in illumos-gate so everyone
can benefit. But I'm not sure what the first step is. I'll see if I
can get some ideas on my side. In the meantime, I hope this
explanation helped.

For further comprehension here is a dtrace script I cooked up to
demonstrate exactly what is happening. The first run is when the zone
is stopped, so the vnic already exists in the GZ. The second run is
when the zone is running, so the vnic exists in the NGZ, not the GZ
(pid 47 is dlmgmtd).

root@midgar:/root# dtrace -Fqn 'pid$target::dlmgmt_createid:entry { self->t = 1; } pid$target::dlmgmt_*:entry,pid$target::cmp_link*:entry /self->t/ { printf("\n"); } pid$target::dlmgmt_*:return,pid$target::cmp_link*:return /self->t/ { printf("%d (%p)\n", arg1, arg1); } pid$target::dlmgmt_createid:return /self->t/ { self->t = 0; }' -p 47
CPU FUNCTION
  0  -> dlmgmt_createid
  0    -> dlmgmt_table_lock
  0      -> dlmgmt_table_readwritelock
  0      <- dlmgmt_table_readwritelock        0 (0)
  0    <- dlmgmt_table_lock                   0 (0)
  0    -> dlmgmt_checkprivs
  0    <- dlmgmt_checkprivs                   0 (0)
  0    -> dlmgmt_create_common
  0      -> cmp_link_by_zname
  0        -> cmp_link_by_name
  0        <- cmp_link_by_name                1 (1)
  0      <- cmp_link_by_zname                 1 (1)
  0      -> cmp_link_by_zname
  0        -> cmp_link_by_name
  0        <- cmp_link_by_name                0 (0)
  0      <- cmp_link_by_zname                 0 (0)
  0    <- dlmgmt_create_common                17 (11)
  0    -> dlmgmt_table_unlock
  0    <- dlmgmt_table_unlock                 0 (0)
  0  <- dlmgmt_createid                       0 (0)
^C

root@midgar:/root# dtrace -Fqn 'pid$target::dlmgmt_createid:entry { self->t = 1; } pid$target::dlmgmt_*:entry,pid$target::cmp_link*:entry /self->t/ { printf("\n"); } pid$target::dlmgmt_*:return,pid$target::cmp_link*:return /self->t/ { printf("%d (%p)\n", arg1, arg1); } pid$target::dlmgmt_createid:return /self->t/ { self->t = 0; }' -p 47
CPU FUNCTION
  1  -> dlmgmt_createid
  1    -> dlmgmt_table_lock
  1      -> dlmgmt_table_readwritelock
  1      <- dlmgmt_table_readwritelock        0 (0)
  1    <- dlmgmt_table_lock                   0 (0)
  1    -> dlmgmt_checkprivs
  1    <- dlmgmt_checkprivs                   0 (0)
  1    -> dlmgmt_create_common
  1      -> cmp_link_by_zname
  1        -> cmp_link_by_name
  1        <- cmp_link_by_name                1 (1)
  1      <- cmp_link_by_zname                 1 (1)
  1      -> cmp_link_by_zname
  1      <- cmp_link_by_zname                 4294967295 (ffffffff)
  1      -> cmp_link_by_id
  1      <- cmp_link_by_id                    1 (1)
  1      -> cmp_link_by_id
  1      <- cmp_link_by_id                    1 (1)
  1      -> dlmgmt_advance
  1        -> dlmgmt_advance_linkid
  1        <- dlmgmt_advance_linkid           0 (0)
  1        -> dlmgmt_advance_ppa
  1        <- dlmgmt_advance_ppa              8 (8)
  1      <- dlmgmt_advance                    8 (8)
  1    <- dlmgmt_create_common                0 (0)
  1    -> dlmgmt_write_db_entry
  1      -> dlmgmt_db_update
  1        -> dlmgmt_db_req_alloc
  1        <- dlmgmt_db_req_alloc             134663480 (806cd38)
  1        -> dlmgmt_process_db_req
  1          -> dlmgmt_process_db_onereq
  1            -> dlmgmt_zfopen
  1              -> dlmgmt_zfop
  1                -> dlmgmt_zopen_cb
  1                <- dlmgmt_zopen_cb         0 (0)
  1              <- dlmgmt_zfop               0 (0)
  1            <- dlmgmt_zfopen               134657968 (806b7b0)
  1            -> dlmgmt_zfopen
  1              -> dlmgmt_zfop
  1                -> dlmgmt_zopen_cb
  1                <- dlmgmt_zopen_cb         0 (0)
  1              <- dlmgmt_zfop               0 (0)
  1            <- dlmgmt_zfopen               134657984 (806b7c0)
  1            -> cmp_link_by_id
  1            <- cmp_link_by_id              1 (1)
  1            -> cmp_link_by_id
  1            <- cmp_link_by_id              1 (1)
  1            -> cmp_link_by_id
  1            <- cmp_link_by_id              0 (0)
  1            -> dlmgmt_zrename
  1              -> dlmgmt_zfop
  1                -> dlmgmt_zrename_cb
  1                <- dlmgmt_zrename_cb       0 (0)
  1              <- dlmgmt_zfop               0 (0)
  1            <- dlmgmt_zrename              0 (0)
  1          <- dlmgmt_process_db_onereq      0 (0)
  1        <- dlmgmt_process_db_req           0 (0)
  1      <- dlmgmt_db_update                  0 (0)
  1    <- dlmgmt_write_db_entry               0 (0)
  1    -> dlmgmt_table_unlock
  1    <- dlmgmt_table_unlock                 0 (0)
  1  <- dlmgmt_createid                       4 (4)
^C