Kernelbased SMB should support Apple Timemachine
I have tried TimeMachine on OmniOS 151032
First step: svcadm enable dns/multicast
then I started a script.sh with the following content
dns-sd -R hostname _adisk._tcp. local 2 dk0=adVN=test,adVF=0x82 &
dns-sd -R hostname _device-info._tcp. local 445 model=Xserve &
dns-sd -R hostname _smb._tcp. local. 445 &
Result (beside the nice Xserve icon) on OSX Catalina:
The SMB share "test" is shown and selectable as a TimeMachine target.
When I tried to connect I got an: "The selected network backup volume does not support the needed functions"