Failure modes of incorrect Type= in systemd units

Proper supervision of daemons requires knowledge which cannot be inferred automatically. Some aspects of service behaviour have to be described by administrator. Widely used upstart system manager has 3 type-like job properties: expect daemon, fork and stop. Their description starts with a warning:

This stanza is extremely important: read this section carefully!

The warning is no less true with systemd's Type=.

Wrong type specification impacts service monitoring, failure detection and dependencies handling. There are six service types in systemd. Three basic ones (simple, forking, oneshot) and three more sophisticated (dbus, idle, notify).

Failure to set proper type may not bite you immediately. Service may run for minute or two and then get killed mysteriously. Services depending on wrongly specified one won't be started, even if it appears to run fine; but systemd won't consider the service ready and will not start dependent units.

Type highlights

• simple - default type; daemon has to stay in foreground. Ready: as soon as the binary is executed (which may be too soon, service may not be ready to serve requests yet).
• forking - daemon must fork. Ready: after first process exits. If PIDFile= is defined, readiness is delayed until PID is written to the pidfile.
• oneshot - suitable for running scripts; systemd waits until process finishes. Ready: when main process exits.
• dbus - for D-Bus services. Ready: when specified BusName= is acquired.
• idle - like “simple” , but with delayed execution.
• notify - most flexible and robust type; by defining simple communication protocol between daemon and systemd, precise information about state can be provided. Requires simple patching, for scripts you can use systemd-notify helper. Ready: when service says so.

Basic type determination

Suitable type can be guessed correctly in 75% cases by just running the daemon binary. If it stays in foreground, connected to the terminal, the type is “simple”. If it forks into the background, the type is “forking” (suprisingly!)

 # /usr/sbin/daemon
Copyright 2014 Foo Baz Bar Corp.
Serving Requests…

→ Type=simple

# /usr/sbin/otherdaemon
#

→ Type=forking

If you need to run a program/script which does its jobs and exits (does not stay running), use oneshot. Please note: if you can choose how the daemon behaves, it generally better to go with "forking". This makes systemd declare "ready" after daemon is actually able to accept connections.

Failure modes

Most of the failures shown below happen after TimeoutStartSec=. By default it’s set to 90 seconds, so lets stick with this value.

		Incorrectly selected Type=
		simple	forking	oneshot	notify
Correct type	simple	✓	killed after 90 s1	stays in “activating” state forever2	killed after 90 s3
	forking	killed almost immediately4	✓	stays in “activating” state forever	killed after 90 s
	oneshot	becomes “failed” after executing5	killed after 90 s	✓	killed after 90 s
	notify	probably works fine	killed after 90 s	stays in “activating” state forever	✓

Notes:

1. systemd expects process to fork and parent to exit. It waits until timeout.
2. it never gets to the “ready” state, so no units depending on it will start; start timeout is ignored for “oneshot” units.
3. systemd will wait for ready notification. Until timeout.
4. ”main” process exits
5. daemons should not exit, but “oneshot” units should

I hope that explains why sometimes systemd kills your service after a minute. Of course you should read man systemd.service, it contains much more details.