Systemd does a good job at monitoring and restarting services. It also keeps track of "failed" services, which you can easily see in systemctl output.
root@damnation:~# rc-status -a Runlevel: shutdown killprocs [ stopped ] savecache [ stopped ] mount-ro [ stopped ] Runlevel: single Runlevel: nonetwork local [ started ] Runlevel: cryptinit rsyslog [ started ] ip6tables [ started ] ... twistd [ started ] local [ started ] Runlevel: sysinit dmesg [ started ] udev [ started ] devfs [ started ] Runlevel: boot hwclock [ started ] lvm [ started ] ... wdd [ started ] keymaps [ started ] Runlevel: default rsyslog [ started ] ip6tables [ started ] ... twistd [ started ] local [ started ] Dynamic Runlevel: hotplugged Dynamic Runlevel: needed sysfs [ started ] rpc.pipefs [ started ] ... rpcbind [ started ] rpc.idmapd [ started ] Dynamic Runlevel: manual
So, meet the systemd-dashboard tool:
root@damnation:~# systemd-dashboard -h usage: systemd-dashboard [-h] [-s] [-u] [-n] Tool to compare the set of enabled systemd services against currently running ones. If started without parameters, it'll just show all the enabled services that should be running (Type != oneshot) yet for some reason they aren't. optional arguments: -h, --help show this help message and exit -s, --status Show status report on found services. -u, --unknown Show enabled but unknown (not loaded) services. -n, --not-enabled Show list of services that are running but are not enabled directly.
Simple invocation will show what's not running while it should be:
root@damnation:~# systemd-dashboard smartd.service systemd-readahead-replay.service apache.service
Adding "-s" flag will show what happened there in more detail (by the grace of "systemctl status" command):
root@damnation:~# systemd-dashboard -s smartd.service - smartd Loaded: loaded (/lib64/systemd/system/smartd.service) Active: failed since Sun, 27 Feb 2011 11:44:05 +0500; 2s ago Process: 16322 ExecStart=/usr/sbin/smartd --no-fork --capabilities (code=killed, signal=KILL) CGroup: name=systemd:/system/smartd.service systemd-readahead-replay.service - Replay Read-Ahead Data Loaded: loaded (/lib64/systemd/system/systemd-readahead-replay.service) Active: inactive (dead) CGroup: name=systemd:/system/systemd-readahead-replay.service apache.service - apache2 Loaded: loaded (/lib64/systemd/system/apache.service) Active: inactive (dead) since Sun, 27 Feb 2011 11:42:34 +0500; 51s ago Process: 16281 ExecStop=/usr/bin/apachectl -k stop (code=exited, status=0/SUCCESS) Main PID: 5664 (code=exited, status=0/SUCCESS) CGroup: name=systemd:/system/apache.service
Would you've noticed that readahead fails on a remote machine because the kernel is missing fanotify and the service apparently thinks "it's okay not to start" in this case? What about smartd you've killed a while ago and forgot to restart?
And you can check if you forgot to enable something with "-n" flag, which will show all the running stuff that was not explicitly enabled.
Code is under a hundred lines of python with the only dep of dbus-python package. You can grab the initial (probably not updated much, although it's probably finished as it is) version from here or a maintained version from fgtk repo (yes, there's an anonymous login form to pass).
If someone will also find the thing useful, I'd appreciate if you'll raise awareness to the issue within systemd project - I'd rather like to see such functionality in the main package, not hacked-up on ad-hoc basis around it.
Update (+20d): issue was noticed and will probably be addressed in systemd. Yay!