Actions
Bug #14730
open
When only site DC reboots, chaos ensues with LDAP and SMB
Status:
New
Priority:
Normal
Assignee:
-
Category:
smb - SMB server and client
Start date:
Due date:
% Done:
0%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
Description
Once a month at one of our sites a DC reboots for routine updates. During this time, the ldap/client service has issues reaching the DC as does the smb server. Eventually, some diagnostic messages are produced before it gives up and never tries again. Because of this, domain based credentials can't be used to login to the system. Sometimes a local login will work but usually it gets so hung up that it can't even open a new connection on the local console via IPMI, let alone over SSH.
This might be two separate issues. The not being able to establish a new SSH session (were able to get a login prompt but after entering local credentials, it just hung) seems like an odd symptom but I do believe it to be related to this, as it seems to occur quite frequently during the DC reboots. Here's a screenshot taken during this fiasco. Obviously, scheduling a cron job to restart smbd and ldap/client around this updating window daily would probably fix it but a less hacky solution would probably be be better suited.
Files
Updated by Jorge Schrauwen 22 days ago
This might be related to https://www.illumos.org/issues/13326, with plain LDAP on my setup everything also hangs if the ldap servers are rebooted.
Updated by Gordon Ross 22 days ago
Can you please attach the last parts of these files:
/var/svc/log/system-idmap*.log
/var/svc/log/network-smb-server*.log
Updated by Adam Stylinski 22 days ago
Gordon Ross wrote in #note-2:
Can you please attach the last parts of these files:
/var/svc/log/system-idmap*.log
@ Mon Jun 6 05:05:58 2022
created thread ID 41 - 37 threads currently active
@ Mon Jun 6 05:06:43 2022
created thread ID 42 - 38 threads currently active
@ Mon Jun 6 05:07:28 2022
created thread ID 43 - 39 threads currently active
@ Mon Jun 6 05:08:13 2022
created thread ID 44 - 40 threads currently active
@ Mon Jun 6 05:08:58 2022
thread creation refused - 40 threads currently active
@ Mon Jun 6 09:26:21 2022
Terminating.
scf_pg_update() failed: connection to repository broken
Fatal errors while reading SMF properties
[ Jun 6 09:28:51 Executing start method ("/usr/lib/idmapd"). ]
@ Mon Jun 6 09:28:51 2022
Global Catalog servers not configured/discoverable
Domain controller servers not configured/discoverable
created thread ID 3 - 1 threads currently active
[ Jun 6 09:28:51 Method "start" exited with status 0. ]
@ Mon Jun 6 09:29:55 2022
created thread ID 5 - 2 threads currently active
@ Mon Jun 6 09:31:40 2022
/var/svc/log/network-smb-server*.log
With usernames redacted:
smbd: smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: smb_idmap_batch_getmappings: RPC error
authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failedsmbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failedsmbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failedsmbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failedsmbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failedsmbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failedsmbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failedsmbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failedsmbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: smbd.err: smb_idmap_batch_getmappings: RPC error
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failedsmbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: smb_idmap_batch_getmappings: RPC error
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: smbd.err: smb_idmap_batch_getmappings: RPC error
authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failedsmbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: smbd.err: smb_idmap_batch_getmappings: RPC error
authsvc_work: send failedsmbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failedsmbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: idmap_get_mappings: RPC error
smbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: idmap_get_mappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failedsmbd.err: INTERNAL\someuser: idmap failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: authsvc_work: send failed
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: idmap_get_mappings: RPC error
authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd: authsvc_work: send failedsmbd.err: INTERNAL\someuser: idmap failed
smbd.err: idmap_get_mappings: RPC error
smbd: authsvc_work: send failed
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: smb_idmap_batch_getmappings: RPC error
smbd.err: INTERNAL\someuser: idmap failed
smbd: smbd.err: INTERNAL\someuser: idmap failed
authsvc_work: send failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
smbd: authsvc_work: send failed
@ Mon Jun 6 09:26:22 2022
smbd.err: SMF initialization problem: %s
: handle not bound
smbd: service terminated
[ Jun 6 09:28:31 Enabled. ]
[ Jun 6 09:29:45 Executing start method ("/usr/lib/smbsrv/smbd start"). ]
smbd: smbd starting, pid 912
smbd: NetBIOS services disabled
smbd: service initialized
[ Jun 6 09:29:45 Method "start" exited with status 0. ]
smbd_dc_monitor: online
smbd_localtime_monitor: online
@ Mon Jun 6 09:29:56 2022
smbd.info: smbd_dc_update: internal.etegent.com: located et-day-dc01.internal.etegent.com
@ Mon Jun 6 14:43:29 2022
smbd.err: Can't get SID for ID=0 type=1, status=-9977
smbd.err: Can't get SID for ID=0 type=0, status=-9977
[ Jun 12 05:00:00 Stopping because service restarting. ]
[ Jun 12 05:00:00 Executing stop method (:kill). ]
Updated by Adam Stylinski about 15 hours ago
Gordon,
We just ran into this issue on another system after updating it from a revision from all the back in 2019. I know that's not a very good bisectable range but...it was a recent issue introduced if that helps at all. The issue about smb hanging in the OI bug tracker seems to suggest you may have a solution in the works or at least a smoking gun?
Actions