Project

General

Profile

Actions

Bug #5896

closed

svccfg import returns before service can be used by svcadm

Added by Robert Mustacchi about 7 years ago. Updated about 7 years ago.

Status:
Closed
Priority:
Normal
Category:
smf
Start date:
2015-05-01
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

On a heavily loaded system we've seen many cases where we get a message that's effectively:

svcadm: svc:/SERVICE/:default is misconfigured (lacks "restarter" property group).

This is usually done in a cycle of:

$ svccfg import *.xml
$ svcadm enable -s *.xml

However, the latter usually errors and ends up emitting that message.

The race condition here is rather complicated. By default, most services / instances do not specify the restarter property group in their manifest. This is by design. Any time this is not specified, we end up defaulting to svc.startd as our restarter. This is what we generally want.

Before the service fully starts up, it will have this property group tacked on. This is being done in the bowels of configd via one of the two following functions:

  • instance_get_or_add_pg
  • wait_fmri_enabled

This happens as a side effect of every state transition for every instance. The means that this happens in smf are long and convoluted and will be documented as a part of all this work. Basically, svccfg is changing state and svc.startd is asynchronously getting poll information with what has changed and acting on it.

svcadm with -s calls into a path which is waiting for every fmri. The first thing this does is request the restarter property group. Because svccfg returns before that property group has been set we call into svcadm enable -s. However, this ends up causing us to try and request the restarter property which may or may not exist yet depending on startd.

Because we are doing a lot of changes during startup and only one thread in startd is polling configd it makes sense that we're seeing this race occur.

The proper fix here is that svccfg import cannot return until it sees that there is a valid restarter property group for each of instances that it imports.

Actions #1

Updated by Electric Monk about 7 years ago

  • Status changed from New to Closed

git commit 2ba6d2b94a398caab9e751c277f0acbd1cc22c77

commit  2ba6d2b94a398caab9e751c277f0acbd1cc22c77
Author: Robert Mustacchi <rm@joyent.com>
Date:   2015-05-15T16:19:40.000Z

    5896 svccfg import returns before service can be used by svcadm
    5897 improve comments for svc.startd
    Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Actions

Also available in: Atom PDF