Feb 27, 2011

Dashboard for enabled services in systemd

Systemd does a good job at monitoring and restarting services. It also keeps track of "failed" services, which you can easily see in systemctl output.

Problem for me is that services that should be running at the machine don't always "fail".
I can stop them and forget to start again, .service file can be broken (like, reload may actually terminate the service), they can be accidentally or deliberately killed or just exit with 0 code due to some internal event, just because they think that's okay to stop now.
Most often such "accidents" seem to happen on boot - some services just perform sanity checks, see that some required path or socket is missing and exit, sometimes with code 0.
As a good sysadmin, you take a peek at systemctl, see no failures there and think "ok, successful reboot, everything is started", and well, it's not, and systemd doesn't reveal that fact.
What's needed here is kinda "dashboard" of what is enabled and thus should be running with clear indication if something is not. Best implementation of such thing I've seen in openrc init system, which comes with baselayout-2 on Gentoo Linux ("unstable" or "~" branch atm, but guess it'll be stabilized one day):
root@damnation:~# rc-status -a
Runlevel: shutdown
  killprocs  [ stopped ]
  savecache  [ stopped ]
  mount-ro   [ stopped ]
Runlevel: single
Runlevel: nonetwork
  local      [ started ]
Runlevel: cryptinit
  rsyslog    [ started ]
  ip6tables  [ started ]
...
  twistd     [ started ]
  local      [ started ]
Runlevel: sysinit
  dmesg      [ started ]
  udev       [ started ]
  devfs      [ started ]
Runlevel: boot
  hwclock    [ started ]
  lvm        [ started ]
...
  wdd        [ started ]
  keymaps    [ started ]
Runlevel: default
  rsyslog    [ started ]
  ip6tables  [ started ]
...
  twistd     [ started ]
  local      [ started ]
Dynamic Runlevel: hotplugged
Dynamic Runlevel: needed
  sysfs      [ started ]
  rpc.pipefs [ started ]
...
  rpcbind    [ started ]
  rpc.idmapd [ started ]
Dynamic Runlevel: manual
Just "grep -v started" and you see everything that's "stopped", "crashed", etc.
I tried to raise issue on systemd-devel, but looks like I'm the only one who cares about it, so I went ahead to write my own tool for the job.
Implementation uses extensive dbus interface provided by systemd to get a set of all the .service units loaded by systemd, then gets "enabled" stuff from symlinks on a filesystem. Latter are easily located in places /{etc,lib}/systemd/system/*/*.service and systemd doesn't seem to keep track of these, using them only at boot-time.
Having some experience using rc-status tool from openrc I also fixed the main annoyance it has - there's no point to show "started" services, ever! I always cared about "not enabled" or "not started" only, and shitload of "started" crap it dumps is just annoying, and has to always be grepped-out.

So, meet the systemd-dashboard tool:

root@damnation:~# systemd-dashboard -h
usage: systemd-dashboard [-h] [-s] [-u] [-n]

Tool to compare the set of enabled systemd services against currently running
ones. If started without parameters, it'll just show all the enabled services
that should be running (Type != oneshot) yet for some reason they aren't.

optional arguments:
 -h, --help  show this help message and exit
 -s, --status Show status report on found services.
 -u, --unknown Show enabled but unknown (not loaded) services.
 -n, --not-enabled Show list of services that are running but are not
   enabled directly.

Simple invocation will show what's not running while it should be:

root@damnation:~# systemd-dashboard
smartd.service
systemd-readahead-replay.service
apache.service

Adding "-s" flag will show what happened there in more detail (by the grace of "systemctl status" command):

root@damnation:~# systemd-dashboard -s

smartd.service - smartd
  Loaded: loaded (/lib64/systemd/system/smartd.service)
  Active: failed since Sun, 27 Feb 2011 11:44:05 +0500; 2s ago
  Process: 16322 ExecStart=/usr/sbin/smartd --no-fork --capabilities (code=killed, signal=KILL)
  CGroup: name=systemd:/system/smartd.service

systemd-readahead-replay.service - Replay Read-Ahead Data
  Loaded: loaded (/lib64/systemd/system/systemd-readahead-replay.service)
  Active: inactive (dead)
  CGroup: name=systemd:/system/systemd-readahead-replay.service

apache.service - apache2
  Loaded: loaded (/lib64/systemd/system/apache.service)
  Active: inactive (dead) since Sun, 27 Feb 2011 11:42:34 +0500; 51s ago
  Process: 16281 ExecStop=/usr/bin/apachectl -k stop (code=exited, status=0/SUCCESS)
  Main PID: 5664 (code=exited, status=0/SUCCESS)
  CGroup: name=systemd:/system/apache.service

Would you've noticed that readahead fails on a remote machine because the kernel is missing fanotify and the service apparently thinks "it's okay not to start" in this case? What about smartd you've killed a while ago and forgot to restart?

And you can check if you forgot to enable something with "-n" flag, which will show all the running stuff that was not explicitly enabled.

Code is under a hundred lines of python with the only dep of dbus-python package. You can grab the initial (probably not updated much, although it's probably finished as it is) version from here or a maintained version from fgtk repo (yes, there's an anonymous login form to pass).

If someone will also find the thing useful, I'd appreciate if you'll raise awareness to the issue within systemd project - I'd rather like to see such functionality in the main package, not hacked-up on ad-hoc basis around it.

Update (+20d): issue was noticed and will probably be addressed in systemd. Yay!

Feb 26, 2011

cgroups initialization, libcgroup and my ad-hoc replacement for it

Linux control groups (cgroups) rock.
If you've never used them at all, you bloody well should.
"git gc --aggressive" of a linux kernel tree killing you disk and cpu?
Background compilation makes desktop apps sluggish? Apps step on each others' toes? Disk activity totally kills performance?

I've lived with all of the above on the desktop in the (not so distant) past and cgroups just make all this shit go away - even forkbombs and super-multithreaded i/o can just be isolated in their own cgroup (from which there's no way to escape, not with any amount of forks) and scheduled/throttled (cpu hard-throttling - w/o cpusets - seem to be coming soon as well) as necessary.

Some problems with process classification and management of these limits seem to exist though.

Systemd does a great job of classification of everything outside of user session (i.e. all the daemons) - any rc/cgroup can be specified right in the unit files or set by default via system.conf.
And it also makes all this stuff neat and tidy because cgroup support there is not even optional - it's basic mechanism on which systemd is built, used to isolate all the processes belonging to one daemon or the other in place of hopefully-gone-for-good crappy and unreliable pidfiles. No renegade processes, leftovers, pids mixups... etc, ever.
Bad news however is that all the cool stuff it can do ends right there.
Classification is nice, but there's little point in it from resource-limiting perspective without setting the actual limits, and systemd doesn't do that (recent thread on systemd-devel).
Besides, no classification for user-started processes means that desktop users are totally on their own, since all resource consumption there branches off the user's fingertips. And desktop is where responsiveness actually matters for me (as in "me the regular user"), so clearly something is needed to create cgroups, set limits there and classify processes to be put into these cgroups.
libcgroup project looks like the remedy at first, but since I started using it about a year ago, it's been nothing but the source of pain.
First task that stands before it is to actually create cgroups' tree, mount all the resource controller pseudo-filesystems and set the appropriate limits there.
libcgroup project has cgconfigparser for that, which is probably the most brain-dead tool one can come up with. Configuration is stone-age pain in the ass, making you clench your teeth, fuck all the DRY principles and write N*100 line crap for even the simpliest tasks with as much punctuation as possible to cram in w/o making eyes water.
Then, that cool toy parses the config, giving no indication where you messed it up but the dreaded message like "failed to parse file". Maybe it's not harder to get right by hand than XML, but at least XML-processing tools give some useful feedback.
Syntax aside, tool still sucks hard when it comes to apply all the stuff there - it either does every mount/mkdir/write w/o error or just gives you the same "something failed, go figure" indication. Something being already mounted counts as failure as well, so it doesn't play along with anything, including systemd.
Worse yet, when it inevitably fails, it starts a "rollback" sequence, unmounting and removing all the shit it was supposed to mount/create.
After killing all the configuration you could've had, it will fail anyway. strace will show you why, of course, but if that's the feedback mechanism the developers had in mind...
Surely, classification tools there can't be any worse than that? Wrong, they certainly are.
Maybe C-API is where the project shines, but I have no reason to believe that, and luckily I don't seem to have any need to find out.

Luckily, cgroups can be controlled via regular filesystem calls, and thoroughly documented (in Documentation/cgroups).

Anyways, my humble needs (for the purposes of this post) are:

  • isolate compilation processes, usually performed by "cave" client of paludis package mangler (exherbo) and occasionally shell-invoked make in a kernel tree, from all the other processes;
  • insulate specific "desktop" processes like firefox and occasional java-based crap from the rest of the system as well;
  • create all these hierarchies in a freezer and have a convenient stop-switch for these groups.

So, how would I initially approach it with libcgroup? Ok, here's the cgconfig.conf:

### Mounts

mount {
  cpu = /sys/fs/cgroup/cpu;
  blkio = /sys/fs/cgroup/blkio;
  freezer = /sys/fs/cgroup/freezer;
}

### Hierarchical RCs

group tagged/cave {
  perm {
    task {
      uid = root;
      gid = paludisbuild;
    }
    admin {
      uid = root;
      gid = root;
    }
  }

  cpu {
    cpu.shares = 100;
  }
  freezer {
  }
}

group tagged/desktop/roam {
  perm {
    task {
      uid = root;
      gid = users;
    }
    admin {
      uid = root;
      gid = root;
    }
  }

  cpu {
    cpu.shares = 300;
  }
  freezer {
  }
}

group tagged/desktop/java {
  perm {
    task {
      uid = root;
      gid = users;
    }
    admin {
      uid = root;
      gid = root;
    }
  }

  cpu {
    cpu.shares = 100;
  }
  freezer {
  }
}

### Non-hierarchical RCs (blkio)

group tagged.cave {
  perm {
    task {
      uid = root;
      gid = users;
    }
    admin {
      uid = root;
      gid = root;
    }
  }

  blkio {
    blkio.weight = 100;
  }
}

group tagged.desktop.roam {
  perm {
    task {
      uid = root;
      gid = users;
    }
    admin {
      uid = root;
      gid = root;
    }
  }

  blkio {
    blkio.weight = 300;
  }
}

group tagged.desktop.java {
  perm {
    task {
      uid = root;
      gid = users;
    }
    admin {
      uid = root;
      gid = root;
    }
  }

  blkio {
    blkio.weight = 100;
  }
}
Yep, it's huge, ugly and stupid.
Oh, and you have to do some chmods afterwards (more wrapping!) to make the "group ..." lines actually matter.

So, what do I want it to look like? This:

path: /sys/fs/cgroup

defaults:
  _tasks: root:wheel:664
  _admin: root:wheel:644
  freezer:

groups:

  base:
    _default: true
    cpu.shares: 1000
    blkio.weight: 1000

  tagged:
    cave:
      _tasks: root:paludisbuild
      _admin: root:paludisbuild
      cpu.shares: 100
      blkio.weight: 100

    desktop:
      roam:
        _tasks: root:users
        cpu.shares: 300
        blkio.weight: 300
      java:
        _tasks: root:users
        cpu.shares: 100
        blkio.weight: 100

It's parseable and readable YAML, not some parenthesis-semicolon nightmare of a C junkie (you may think that because of these spaces don't matter there btw... well, think again!).

After writing that config-I-like-to-see, I just spent a few hours to write a script to apply all the rules there while providing all the debugging facilities I can think of and wiped my system clean of libcgroup, it's that simple.
Didn't had to touch the parser again or debug it either (especially with - god forbid - strace), everything just worked as expected, so I thought I'd dump it here jic.

Configuration file above (YAML) consists of three basic definition blocks:

"path" to where cgroups should be initialized.
Names for the created and mounted rc's are taken right from "groups" and "defaults" sections.
Yes, that doesn't allow mounting "blkio" resource controller to "cpu" directory, guess I'll go back to using libcgroup when I'd want to do that... right after seeing the psychiatrist to have my head examined... if they'd let me go back to society afterwards, that is.
"groups" with actual tree of group parameter definitions.
Two special nodes here - "_tasks" and "_admin" - may contain (otherwise the stuff from "defaults" is used) ownership/modes for all cgroup knob-files ("_admin") and "tasks" file ("_tasks"), these can be specified as "user[:group[:mode]]" (with brackets indicating optional definition, of course) with non-specified optional parts taken from the "defaults" section.
Limits (or any other settings for any kernel-provided knobs there, for that matter) can either be defined on per-rc-dict basis, like this:
roam:
  _tasks: root:users
  cpu:
    shares: 300
  blkio:
    weight: 300
    throttle.write_bps_device: 253:9 1000000

Or just with one line per rc knob, like this:

roam:
  _tasks: root:users
  cpu.shares: 300
  blkio.weight: 300
  blkio.throttle.write_bps_device: 253:9 1000000
Empty dicts (like "freezer" in "defaults") will just create cgroup in a named rc, but won't touch any knobs there.
And the "_default" parameter indicates that every pid/tid, listed in a root "tasks" file of resource controllers, specified in this cgroup, should belong to it. That is, act like default cgroup for any tasks, not classified into any other cgroup.

"defaults" section mirrors the structure of any leaf cgroup. RCs/parameters here will be used for created cgroups, unless overidden in "groups" section.

Script to process this stuff (cgconf) can be run with --debug to dump a shitload of info about every step it takes (and why it does that), plus with --dry-run flag to just dump all the actions w/o actually doing anything.
cgconf can be launched as many times as needed to get the job done - it won't unmount anything (what for? out of fear of data loss on a pseudo-fs?), will just create/mount missing stuff, adjust defined permissions and set defined limits without touching anything else, thus it will work alongside with everything that can also be using these hierarchies - systemd, libcgroup, ulatencyd, whatever... just set what you need to adjust in .yaml and it wll be there after run, no side effects.
cgconf.yaml (.yaml, generally speaking) file can be put alongside cgconf or passed via the -c parameter.
Anyway, -h or --help is there, in case of any further questions.

That handles the limits and initial (default cgroup for all tasks) classification part, but then chosen tasks also need to be assigned to a dedicated cgroups.

libcgroup has pam_cgroup module and cgred daemon, neither of which can sensibly (re)classify anything within a user session, plus cgexec and cgclassify wrappers to basically do "echo $$ >/.../some_cg/tasks && exec $1" or just "echo" respectively.
These are dumb simple, nothing done there to make them any easier than echo, so even using libcgroup I had to wrap these.

Since I knew exactly which (few) apps should be confined to which groups, I just wrote a simple wrapper scripts for each, putting these in a separate dir, in the head of PATH. Example:

#!/usr/local/bin/cgrc -s desktop/roam/usr/bin/firefox
cgrc script here is a dead-simple wrapper to parse cgroup parameter, putting itself into corresponding cgroup within every rc where it exists, making special conversion in case not-yet-hierarchical (there's a patchset for that though: http://lkml.org/lkml/2010/8/30/30) blkio, exec'ing the specified binary with all the passed arguments afterwards.
All the parameters after cgroup (or "-g ", for the sake of clarity) go to the specified binary. "-s" option indicates that script is used in shebang, so it'll read command from the file specified in argv after that and pass all the further arguments to it.
Otherwise cgrc script can be used as "cgrc -g /usr/bin/firefox " or "cgrc. /usr/bin/firefox ", so it's actually painless and effortless to use this right from the interactive shell. Amen for the crappy libcgroup tools.
Another special use-case for cgroups I've found useful on many occasions is a "freezer" thing - no matter how many processes compilation (or whatever other cgroup-confined stuff) forks, they can be instantly and painlessly stopped and resumed afterwards.
cgfreeze dozen-liner script addresses this need in my case - "cgfreeze cave" will stop "cave" cgroup, "cgfreeze -u cave" resume, and "cgfreeze -c cave" will just show it's current status, see -h there for details. No pgrep, kill -STOP or ^Z involved.

Guess I'll direct the next poor soul struggling with libcgroup here, instead of wasting time explaining how to work around that crap and facing the inevitable question "what else is there?" *sigh*.

All the mentioned scripts can be found here.

Nov 05, 2010

From Baselayout to Systemd setup on Exherbo

It's been more than a week since I've migrated from sysvinit and gentoo'ish baselayout scripts to systemd with it's units, and aside from few initial todos it's been surprisingly easy.
Nice guide for migration (which actually tipped me into trying systemd) can be found here, in this post I'd rather summarize my experiences.
Most distributions seem to take "the legacy" way of migration, starting all the old initscripts from systemd just as sysinit did before that.
It makes some sense, since all the actions necessary to start the service are already written there, but most of them are no longer necessary with systemd - you don't need pidfiles, daemonization, killing code, LSB headers and most checks for other stuff... which kinda leaves nothing at all for 95% of software I've encountered!
I haven't really tried to adapt fedora or debian init for systemd (since my setup runs exherbo), so I may be missing some crucial points here, but it looks like even in these systems initscripts, written in simple unaugmented *sh, are unnecessary evil, each one doing the same thing in it's own crappy way.
With exherbo (or gentoo, for that matter), which has a bit more advanced init system, it's even harder to find some sense in keeping these scripts. Baselayout allows some cool stuff beyond simple LSB headers, but does so in it's own way, typical initscript here looks like this:
#!/sbin/runscript
depend() {
    use logger
    need clock hostname
    provide cron
}
start() {
    ebegin "Starting ${SVCNAME}"
    start-stop-daemon --start --pidfile ${FCRON_PIDFILE}\
      --exec /usr/sbin/fcron -- -c ${FCRON_CONF}
    eend $?
}
stop() {
    ebegin "Stopping ${SVCNAME}"
    start-stop-daemon --stop --pidfile ${FCRON_PIDFILE}
    eend $?
}

...with $SVCNAME taken from the script name and other vars from complimentary "/etc/conf.d/someservice" file (with sensible defaults in initscript itself).

Such script already allows nice and logged output (with e* commands) and clearly-defined, relatively uncluttered sections for startup and shutdown. You don't have to parse commandline arguments (although it's perfectly possible), since baselayout scripts will do that, and every daemon is accounted for via "start-stop-daemon" wrapper - it has a few simple ways to check their status via passed --pidfile or --exec lines, plus it handles forking (if necessary), IO redirection, dropping privileges and stuff like that.

All these feats lead to much more consistent init and control over services' state:

root@damnation:~# rc-status -a
Runlevel: shutdown
  killprocs        [ stopped ]
  savecache        [ stopped ]
  mount-ro         [ stopped ]
Runlevel: single
Runlevel: nonetwork
  local            [ started ]
Runlevel: cryptinit
  rsyslog          [ started ]
  ip6tables        [ started ]
...
  twistd           [ started ]
  local            [ started ]
Runlevel: sysinit
  dmesg            [ started ]
  udev             [ started ]
  devfs            [ started ]
Runlevel: boot
  hwclock          [ started ]
  lvm              [ started ]
...
  wdd              [ started ]
  keymaps          [ started ]
Runlevel: default
  rsyslog          [ started ]
  ip6tables        [ started ]
...
  twistd           [ started ]
  local            [ started ]
Dynamic Runlevel: hotplugged
Dynamic Runlevel: needed
  sysfs            [ started ]
  rpc.pipefs       [ started ]
...
  rpcbind          [ started ]
  rpc.idmapd       [ started ]
Dynamic Runlevel: manual
One nice colored list of everything that should be running, is running, failed to start, crashed and whatever. One look and you know if unscheduled reboot has any surprises for you. Weird that such long-lived and supported distros as debian and fedora make these simple tasks so much harder (chkconfig --list? You can keep it! ;)
Furthermore, it provides as many custom and named runlevels as you want, as a way to flip the state of the whole system with a painless one-liner.

Now, systemd provides all of these features, in a cleaner nicer form and much more, but that makes migration from one to the other actually harder.

Systemd is developed/tested mainly on and for fedora, so abscence of LSB headers in these scripts is a problem (no dependency information), and presence of other headers (which start another scripts w/o systemd help or permission) is even more serious problem.
start-stop-daemon interference is also redundant and actually harmful and so is e* (and other special bl-commands and wrappers), and they won't work w/o baselayout framework.

Thus, it makes sense for systemd on exherbo to be totally independent of baselayout and it's scripts, and having a separate package option to install systemd and baselayout-specific init stuff:

root@sacrilege:~# cave show -f acpid
* sys-power/acpid
   ::arbor   2.0.6-r2* {:0}
   ::installed  2.0.6-r2 {:0}
   sys-power/acpid-2.0.6-r2:0::installed
   Description
acpid is designed to notify user-space programs of ACPI events. It will
will attempt to connect to the Linux kernel via the input layer and
netlink. When an ACPI event is received from one of these sources, acpid
will examine a list of rules, and execute the rules that match the event.
   Homepage  http://tedfelix.com/linux/acpid-netlink.html
   Summary  A configurable ACPI policy daemon for Linux
   From repositories arbor
   Installed time Thu Oct 21 23:11:55 YEKST 2010
   Installed using paludis-0.55.0-git-0.54.2-44-g203a470
   Licences  GPL-2
   Options  (-baselayout) (systemd) build_options: -trace

  sys-power/acpid-2.0.6-r2:0::arbor
   Homepage  http://tedfelix.com/linux/acpid-netlink.html
   Summary  A configurable ACPI policy daemon for Linux
   Description
acpid is designed to notify user-space programs of ACPI events. It will
will attempt to connect to the Linux kernel via the input layer and
netlink. When an ACPI event is received from one of these sources, acpid
will examine a list of rules, and execute the rules that match the event.
   Options  -baselayout systemd
     build_options: -recommended_tests split strip jobs -trace -preserve_work
   Overridden Masks
     Supported platforms ~amd64 ~x86

So, basically, the migration to systemd consists of enabling the option and flipping the "eclectic init" switch:

root@sacrilege:~# eclectic init list
Available providers for init:
 [1] systemd *
 [2] sysvinit
Of course, in reality things are little more complicated, and breaking init is quite undesirable prospect, so I took advantage of virtualization capabilities of cpu on my new laptop and made a complete virtual replica of the system.
Things got a bit more complicated since dm-crypt/lvm setup I've described before, but overally creating such a vm is trivial:
  • A dedicated lv for whole setup.
  • luksFormat it, so it'd represent an encrypted "raw" partition.
  • pvcreate / vgcreate / lvcreate / mkfs on top of it, identical (although much smaller) to original system.
  • A script to mount all these and rsync the "original" system to this replica, with a few post-sync hooks to make some vm-specific changes - different vg name, no extra devices for media content, simplier passwords.
I have this script here, list of "exclusions" for rsync is actually taken from backup scripts, since it's designed to omit various heavy and non-critical paths like caches, spools and debugging info, plus there's not much point syncing most /home contents. All in all, whole setup is about 2-3G and rsync makes a fast job of updating it.
vm (qemu-kvm) startup is right there in the script and uses exactly the same kernel/initrd as the host machine, although I skip encryption part (via kernel cmdline) for faster bootup.

And the first launch gave quite a mixed result: systemd fired a bunch of basic stuff at once, then hanged for about a minute before presenting a getty. After login, it turned out that none of the filesystems in /etc/fstab got mounted.

Systemd handles mounts in quite a clever (and fully documented) way - from each device in fstab it creates a "XXX.device" unit, "fsck@XXX.service", and either "XXX.mount" or "XXX.automount" from mountpoints (depending on optional "comment=" mount opts). All the autogenerated "XXX.mount" units without explicit "noauto" option will get started on boot.
And they do get started, hence that hang. Each .mount, naturally, depends on corresponding .device unit (with fsck in between), and these are considered started when udev issues an event.
In my case, even after exherbo-specific lvm2.service, which does vgscan and vgchange -ay stuff, these events are never generated, so .device units hang for 60 seconds and systemd marks them as "failed" as well as dependent .mount units.
It looks like my local problem, since I actually activate and use these in initrd, so I just worked around it by adding "ExecStart=-/sbin/udevadm trigger --subsystem-match=block --sysname-match=dm-*" line to lvm2.service. That generated the event in parallel to still-waiting .device units, so they got started, then fsck, then just mounted.

While this may look a bit like a problem, it's quite surprising how transparent and easy-to-debug whole process is, regardless of it's massively-parallel nature - all the information is available via "systemctl" and it's show/status commands, all the services are organized (and monitored) in systemd-cgls tree, and can be easily debugged with systemd monitoring and console/dmesg-logging features:

root@sacrilege:~# systemd-cgls
├ 2 [kthreadd]
├ 3 [ksoftirqd/0]
├ 6 [migration/0]
├ 7 [migration/1]
├ 9 [ksoftirqd/1]
├ 10 [kworker/0:1]
...
├ 2688 [kworker/0:2]
├ 2700 [kworker/u:0]
├ 2728 [kworker/u:2]
├ 2729 [kworker/u:4]
├ user
│ └ fraggod
│ └ no-session
│ ├ 1444 /bin/sh /usr/bin/startx
│ ├ 1462 xinit /home/fraggod/.xinitrc -- /etc/X11/xinit/xserverrc :0 -auth /home/fraggod/.serveraut...
...
│ ├ 2407 ssh root@anathema -Y
│ └ 2751 systemd-cgls
└ systemd-1
 ├ 1 /sbin/init
 ├ var-src.mount
 ├ var-tmp.mount
 ├ ipsec.service
 │ ├ 1059 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --nocrsend no --str...
 │ ├ 1060 logger -s -p daemon.error -t ipsec__plutorun
 │ ├ 1061 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --nocrsend no --str...
 │ ├ 1062 /bin/sh /usr/lib/ipsec/_plutoload --wait no --post
 │ ├ 1064 /usr/libexec/ipsec/pluto --nofork --secretsfile /etc/ipsec.secrets --ipsecdir /etc/ipsec.d -...
 │ ├ 1069 pluto helper # 0
 │ ├ 1070 pluto helper # 1
 │ ├ 1071 pluto helper # 2
 │ └ 1223 _pluto_adns
 ├ sys-kernel-debug.mount
 ├ var-cache-fscache.mount
 ├ net@.service
 ├ rpcidmapd.service
 │ └ 899 /usr/sbin/rpc.idmapd -f
 ├ rpcstatd.service
 │ └ 892 /sbin/rpc.statd -F
 ├ rpcbind.service
 │ └ 890 /sbin/rpcbind -d
 ├ wpa_supplicant.service
 │ └ 889 /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -u -Dwext -iwlan0
 ├ cachefilesd.service
 │ └ 883 /sbin/cachefilesd -n
 ├ dbus.service
 │ └ 784 /usr/bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation
 ├ acpid.service
 │ └ 775 /usr/sbin/acpid -f
 ├ openct.service
 │ └ 786 /usr/sbin/ifdhandler -H -p etoken64 usb /dev/bus/usb/002/003
 ├ ntpd.service
 │ └ 772 /usr/sbin/ntpd -u ntp:ntp -n -g -p /var/run/ntpd.pid
 ├ bluetooth.service
 │ ├ 771 /usr/sbin/bluetoothd -n
 │ └ 1469 [khidpd_046db008]
 ├ syslog.service
 │ └ 768 /usr/sbin/rsyslogd -n -c5 -6
 ├ getty@.service
 │ ├ tty1
 │ │ └ 1451 /sbin/agetty 38400 tty1
 │ ├ tty3
 │ │ └ 766 /sbin/agetty 38400 tty3
 │ ├ tty6
 │ │ └ 765 /sbin/agetty 38400 tty6
 │ ├ tty5
 │ │ └ 763 /sbin/agetty 38400 tty5
 │ ├ tty4
 │ │ └ 762 /sbin/agetty 38400 tty4
 │ └ tty2
 │ └ 761 /sbin/agetty 38400 tty2
 ├ postfix.service
 │ ├ 872 /usr/lib/postfix/master
 │ ├ 877 qmgr -l -t fifo -u
 │ └ 2631 pickup -l -t fifo -u
 ├ fcron.service
 │ └ 755 /usr/sbin/fcron -f
 ├ var-cache.mount
 ├ var-run.mount
 ├ var-lock.mount
 ├ var-db-paludis.mount
 ├ home-fraggod-.spring.mount
 ├ etc-core.mount
 ├ var.mount
 ├ home.mount
 ├ boot.mount
 ├ fsck@.service
 ├ dev-mapper-prime\x2dswap.swap
 ├ dev-mqueue.mount
 ├ dev-hugepages.mount
 ├ udev.service
 │ ├ 240 /sbin/udevd
 │ ├ 639 /sbin/udevd
 │ └ 640 /sbin/udevd
 ├ systemd-logger.service
 │ └ 228 //lib/systemd/systemd-logger
 └ tmp.mount
root@sacrilege:~# systemctl status ipsec.service
ipsec.service - IPSec (openswan)
  Loaded: loaded (/etc/systemd/system/ipsec.service)
  Active: active (running) since Fri, 05 Nov 2010 15:16:54 +0500; 2h 16min ago
  Process: 981 (/usr/sbin/ipsec setup start, code=exited, status=0/SUCCESS)
  Process: 974 (/bin/sleep 10, code=exited, status=0/SUCCESS)
  CGroup: name=systemd:/systemd-1/ipsec.service
   ├ 1059 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --noc...
   ├ 1060 logger -s -p daemon.error -t ipsec__plutorun
   ├ 1061 /bin/sh /usr/lib/ipsec/_plutorun --debug --uniqueids yes --force_busy no --noc...
   ├ 1062 /bin/sh /usr/lib/ipsec/_plutoload --wait no --post
   ├ 1064 /usr/libexec/ipsec/pluto --nofork --secretsfile /etc/ipsec.secrets --ipsecdir ...
   ├ 1069 pluto helper # 0
   ├ 1070 pluto helper # 1
   ├ 1071 pluto helper # 2
   └ 1223 _pluto_adns

It's not just hacking at some opaque *sh hacks (like debian init or even interactive-mode baselayout) and takes so little effort to the point that it's really enjoyable process.

But making it mount and start all the default (and available) stuff is not the end of it, because there are plenty of services not yet adapted to systemd.
I actually expected some (relatively) hard work here, because there are quite a few initscripts in /etc/init.d, even on a desktop machine, but once again, I was in for a nice surprise, since systemd just makes all the work go away. All you need to do is to decide on the ordering (or copy it from baselayout scripts) and put an appropriate "Type=" and "ExecStart=" lines in .service file. That's all there is, really.
After that, of course, complete bootup-shutdown test on a vm is in order, and everything "just works" as it is supposed to.
Bootup on a real hardware is exactly the same as vm, no surprises here. "udevadm trigger" seem to be necessary as well, proving validity of vm model.

Systemd boot time is way faster than sysvinit, as it is supposed to, although I don't really care, since reboot is seldom necessary here.

As a summary, I'd recommend everyone to give systemd a try, or at least get familiar with it's rationale and features (plus this three-part blog series: one, two, three). My units aren't perfect (and I'll probably update network-related stuff to use ConnMan), but if you're lazy, grab them here. Also, here is a repo with units for archlinux, which I loosely used as a reference point along with /lib/systemd contents.

← Previous Page 2 of 2
Member of The Internet Defense League