illumos gate

I applied the patch and build a new SmartOS PI as mentioned IRC, I am now getting segfaults in nscd and getent.

Stack from getent core:

Loading modules: [ libc.so.1 libproc.so.1 libnvpair.so.1 libavl.so.1 libuutil.so.1 ld.so.1 ]
> $c
libc.so.1`realfree+0x40(80b1738)
libc.so.1`cleanfree+0x5b(0)
libc.so.1`_malloc_unlocked+0xdf(7)
libc.so.1`malloc+0x3e(7)
libc.so.1`strdup+0x2e(80b17d8)
libsldap.so.1`__ns_ldap_dn2uid+0xac(80b4910, 8047738, 0, 804773c)
nss_ldap.so.1`getmembers_DN+0x8e(80477a8, 80477ac, 8088af0)
nss_ldap.so.1`_nss_ldap_group2str+0x28c(806b148, 8047bc8)
nss_ldap.so.1`_nss_ldap_lookup+0x6a(806b148, 8047bc8, fed6d2f6, 8047a40, 0, fed6c458)
nss_ldap.so.1`getbynam+0xc7(806b148, 8047bc8)
libc.so.1`nss_search+0x1bf(fef4a514, fee4bd40, 4, 8047bc8)
libc.so.1`getgrnam_r+0x89(8047e2d, 8066724, 8066734, 4000)
libc.so.1`getgrnam+0x49(8047e2d)
dogetgr+0xad(8047d24, 80543bd, 8047d1c, fef4a000, 8047d1c, 8065000)
main+0x9c(8047cbc, fef57528, 8047cf8, 80528a7, 3, 8047d1c)
_start_crt+0x96(3, 8047d1c, fefcff74, 0, 0, 0)
_start+0x1a(3, 8047e20, 8047e27, 8047e2d, 0, 8047e34)

Stack from nscd core

Loading modules: [ libumem.so.1 libc.so.1 libuutil.so.1 libavl.so.1 libnvpair.so.1 ld.so.1 ]
> $c
libc.so.1`_lwp_kill+0x15(4, 6, 0, 1, feaa2000, fea80dfd)
libc.so.1`raise+0x2b(6)
libumem.so.1`umem_do_abort+0x53()
libumem.so.1`__umem_assert_failed(fea80dfd, fea81077, 80e0178)
libumem.so.1`process_free+0x74(80e0178, 1, 0)
libumem.so.1`umem_malloc_free+0x1a(80e0178)
libsldap.so.1`__ns_ldap_freeEntry+0x64(80d6028)
libsldap.so.1`__ns_ldap_freeResult+0x46(8121518)
nss_ldap.so.1`_nss_ldap_group2str+0xdf(81214f8, fe11bac8)
nss_ldap.so.1`_nss_ldap_lookup+0x6a(81214f8, fe11bac8, feccd2f6, fe11b888, 0, feccc458)
nss_ldap.so.1`getbynam+0xc7(81214f8, fe11bac8)
nss_search+0xa86(0, 8068aa2, 4, fe11bac8)
nss_psearch+0xcc(fe11bca0, 80000)
lookup_int+0x791(fe19fcb8, 0)
nsc_lookup+0x15(fe19fcb8, 0)
lookup+0x10a(fe19fd58, a8)
switcher+0x151(deadbeed, fe19fd58, a8, 0, 0)
libc.so.1`__door_return+0x4b()

For testing, I setup an LDAP server and loaded some dummy data including 3 groups, 1 with no secondary members, 1 with a secondary member via the memberUID (by username) attribute, 1 with a secondary member via the member (by ldap DN) attribute -- a user name 'test' was added as a secondary member to the latter two groups.

Without the changes the groups with secondary members didn't show up (due to #12236).
With the changes (including the fix for #12236), the secondary groups showed up, and displayed the secondary members correctly (getent group as well as id -a test).

In addition, since the find_domainname() was refactored to use a common 'lookup by dn' function as part of this fix, I added an /etc/hosts style entry as part of the LDAP data, and then ran getent hosts to verify the entry appeared. In addition, I was snooping on the LDAP server, and viewed the search request used -- the filter (nisdomain=*) is only used by the find_domainname function, and that was observed being used for the request -- so find_domainname was called and was returning correct results.

Jorge was also able to test the final version sent for review and confirmed it fixed his issues.

In addition, Gordon Ross was kind enough to test the fix against AD and verify it did not introduce any regressions.