illumos gate

Описание

On a heavily loaded system we've seen many cases where we get a message that's effectively:

svcadm: svc:/SERVICE/:default is misconfigured (lacks "restarter" property group).

This is usually done in a cycle of:

$ svccfg import *.xml
$ svcadm enable -s *.xml

However, the latter usually errors and ends up emitting that message.

The race condition here is rather complicated. By default, most services / instances do not specify the restarter property group in their manifest. This is by design. Any time this is not specified, we end up defaulting to svc.startd as our restarter. This is what we generally want.

Before the service fully starts up, it will have this property group tacked on. This is being done in the bowels of configd via one of the two following functions:

instance_get_or_add_pg
    wait_fmri_enabled

This happens as a side effect of every state transition for every instance. The means that this happens in smf are long and convoluted and will be documented as a part of all this work. Basically, svccfg is changing state and svc.startd is asynchronously getting poll information with what has changed and acting on it.

svcadm with -s calls into a path which is waiting for every fmri. The first thing this does is request the restarter property group. Because svccfg returns before that property group has been set we call into svcadm enable -s. However, this ends up causing us to try and request the restarter property which may or may not exist yet depending on startd.

Because we are doing a lot of changes during startup and only one thread in startd is polling configd it makes sense that we're seeing this race occur.

The proper fix here is that svccfg import cannot return until it sees that there is a valid restarter property group for each of instances that it imports.