It looks like this problem is in the IPv6 part of Illumos, not Red Hat.
I configured a test server that has both IPv4 and IPv6 to have SMB and was able to access a test share from it on windows.
I then removed the A record for that host from DNS and tried to access the same share from a different host (so it wouldn't have DNS cached or any existing mounts) and got
"Windows cannot access \\hvfs1\foo
Check the spelling of the name. Otherwise, there might be a problem with your network. ..."
The test server, like all my file servers, is running OmniOS 151038.
SunOS hvfs1 5.11 omnios-r151038-91d640aafa i86pc i386 i86pc
Here's a redacted snoop run on the server host:
; sudo snoop -d aggr0 port 445
Using device aggr0 (promiscuous mode)
[CLIENT - REDACTED] -> [SERVER - REDACTED] SMB C port=64719
[SERVER - REDACTED] -> [CLIENT - REDACTED] SMB R port=64719
[CLIENT - REDACTED] -> [SERVER - REDACTED] SMB C port=64719
[SERVER - REDACTED] -> [CLIENT - REDACTED] SMB R port=64719
[CLIENT - REDACTED] -> [SERVER - REDACTED] SMB C port=64719
[SERVER - REDACTED] -> [CLIENT - REDACTED] SMB R port=64719
[CLIENT - REDACTED] -> [SERVER - REDACTED] SMB C port=64719
[SERVER - REDACTED] -> [CLIENT - REDACTED] SMB R port=64719
[CLIENT - REDACTED] -> [SERVER - REDACTED] SMB C port=64719
[SERVER - REDACTED] -> [CLIENT - REDACTED] SMB R port=64719
nomad