I want to apologise for not having given systemd a chance before. Like many, I found it an unnecessary change. But the moment I started using it, I loved it.

I don't really know what got into me, actually. I do normally like innovation. For instance, how could I have discovered the superb Notion window manager without giving it a chance, all the while knowing that xmonad was by and large good enough? Maybe I never considered systemd just because I always liked the simple idea of init scripts for starting services and couldn't imagine that something as complex as systemd could be any good. Because systemd is complex. It is sophisticated. But through all that sophistication it's also very helpful. It really tries very hard to be. And if there's anything I like in software, is when it's helpful.

1 So What Happened?

I was done writing autoumount, a program that automatically unmounts filesystems after a period of inactivity. I had intended from the beginning to run it as a service, which is why I went through the trouble of optionally turning it into a daemon thanks to the excellent python-daemon module. Likewise, I carefully set up the logging module to be able to easily collect information if something was working as unexpected. And then came the part I didn't like – actually running it as a service by writing an init script.

2 Init Scripts

If I do like the design choice of plain init scripts for their simplicity, I can't say I like writing them very much. Perhaps it's because I don't do it as often as I should. And because of this, perhaps it's also because I feel the way one should write one changes every time I get to it. That's possibly also because I may change Linux distribution a little too often.

In the beginning, init scripts were really just plain simple shell scripts which, when dropped into the bespoke /etc/rc*.d directory, would start in the corresponding runlevel. Give them a meaningful name prefix with S for start and K for kill followed by a number, and init would know by following the lexicographical order how to work out in which sequence to start or kill them. Simple. Elegant. What everybody was encouraged to do right from the beginning, was of course to further harness the potential of the UNIX filesystems and use symbolic links wherever they could. Write your scripts into /etc/init.d, create symbolic links into /etc/rc*.d and change the names to tell init whether to start or kill the service, and in which order.

That was pretty much all everyone needed to define dependencies. But only insofar as scripts were run serially and not in parallel. And as boot times were becoming long and increasingly more people started using Linux not only on servers which are seldom restarted but also on desktops which were much more often so, it became awfully tempting to think of a way to run init scripts in parallel. There were different implementations. As I came from a world of Red Hat-based distributions and settled for a while on Debian, LSBInitScripts was the one I first came across.

While the idea of starting services in parallel and the approach LSB – Linux Standard Base – took to define dependencies was laudable, this might have been when I started finding the job of writing init scripts off-putting. Looking at other ones for inspiration, I got hit in the face by that big INIT INFO block preceding the logic of the script which defined the dependencies, start and stop orders. For instance, a canonical init script I once cooked up to set up some firewalling looked like:

### BEGIN INIT INFO
# Provides:         firewall
# Required-Start:
# Required-Stop:
# Default-Start:    2
# Default-Stop:
### END INIT INFO
iptables -F
iptables -P INPUT DROP
iptables -P FORWARD DROP

I'd intentionally kept it that simple because I couldn't be bothered about dependencies and tried to stick to the original, simple init scripts. It's iptables, after all, it's close enough to the kernel that it didn't really matter to me when to start it up. I was annoyed enough as it was that the commented block took more lines than the actual logic. If you can call it that way.

When I moved to Ubuntu, Upstart greeted me. Again, when I had to write init scripts I felt a little lost. Especially because you're not really supposed to write init scripts anymore per se, but configuration files. In fairness, I didn't even notice, because Upstart had the good taste to support traditional init scripts too.

More or less at the same time, I also worked with Red Hat-based distributions again. Writing init scripts there was yet another story. They still used traditional init scripts. Not so much bothered about dependency-based boots, I noticed the prominence of functions coming from /etc/rc.d/init.d/functions. They made an attempt at harmonising how init scripts look and provided convenience routines which I certainly welcomed. But seeing this as yet another style of writing init scripts, all this did was to deepen my confusion as I was wondering if I hadn't been supposed to use such function libraries in init scripts I'd written before.

3 Writing services with systemd

Utterly confused, I did notice that all this was converging towards systemd. Red Hat introduced it, Debian had started using it, as did Ubuntu, along with a variety of other major or not-so-major distributions. Many complained about it, but this seemed to become the way to go anyway. So as I was about to turn autoumount into a service, I thought the time was ripe to give it a whirl.

It was love at first sight.

I'd already prepared myself to write an unnecessarily long and complex init script. All I ended up with was writing the /etc/systemd/system/autoumount.service file:

[Service]
ExecStart=/usr/bin/autoumount /media/foo 3600 'pumount'

[Install]
WantedBy=multi-user.target

Those 4 lines, followed by a run of systemctl enable autoumount and systemctl start autoumount was literally all it took. But my fascination didn't stop there. As I wanted to make sure it was running fine, I ran systemctl status autoumount, which I was expecting would give me a terse autoumount is up. But no, all sorts of information ranging from whether the service would be started at boot, when it last started, how long it's been running, which processes, using how much memory and CPU and even specific log messages were displayed. Now, that's what I mean by helpful. And then, two things occurred to me:

1. If systemd records log messages in a journal specific to my service, do I really need to bother setting up logging to write to a specific file, as I did with the excellent Python logging module? I removed all traces of it and simply printed messages to stdout and stderr instead to marvel that systemd recorded them indeed into the autoumount journal. No need to worry about logging anymore.

   Better still, I'd once or twice struggled to collect each line printed in stack traces to log unhandled exceptions. Since systemd collects both stdout and stderr, any such incident would be logged too and readable from systemd status, even with a time stamp before each line. So helpful.

2. If systemd treats everything as individual units and is – of course – capable of starting and running services in parallel, did I really need to implement a daemon? No, as there is no need to start them in the background. In fact, it's probably even best if you don't, because without further systemd configuration of your service, a daemon will sort of slip off systemd's hands, exit and be reported as inactive (dead) by systemd status – which it will be.

When setting up tools which are to run in the background, I often debated whether to turn them into daemons run as services or as cron jobs. I often found the cron job approach appealing because if a process crashes, it will get another chance at running instead of staying down indefinitely. This of course was a quick trick which spared me the trouble of planning failure handling. One colleague once suggested to try Supervisord which, among its many features, is capable of auto-restarting failing processes. No need for any of these clutches anymore with systemd which comes naturally with solutions to these concerns in mind. For instance, unsure of my autoumount service in its early days, I added the Restart option:

[Service]
ExecStart=/usr/bin/autoumount /media/foo 3600 'pumount'
Restart=on-failure

There's a variety of other conditions under which systemd may decide to restart a service. As you can expect, there is also an option to control the burst in which a service repeatedly attempts to restart (StartLimitIntervalSec), an option to wait for a certain time before attempting to restart it (RestartSec) and countless more to fulfil your wildest needs.

If there are ways to gracefully recover from exiting processes, there are also ways to define conditions in which to kill them. I've been using the ClipIt clipboard manager for many years and it's so good it changed my life. However, the version I'm currently using appears to be suffering from a memory leak. Since I only restart my laptop once in a blue moon – and that's usually only when I accidentally let the battery deplete – ClipIt does end up using a ludicrous amount. As you can expect, systemd works well with cgroups and could use them to kick out a process claiming too much memory. But until I work out how that's done, I simply run it as a service like this one, to kill it every 12 hours of runtime and promptly restart it:

[Service]
ExecStart=/usr/bin/clipit
Environment=DISPLAY=:0
RuntimeMaxSec=12h
Restart=on-failure

Also note the Environment option setting the DISPLAY and which is necessary here to be able to start X applications.

4 User-Wide Services

ClipIt as a service? Ah yes, but what I failed to mention is that systemd makes it trivial to configure user-wide services as well as system-wide ones. Whether you like it or not, logging in will cause a /lib/systemd/systemd --user process to be started and you can use it to manage services and a variety of other exciting things as yourself, without any particular privileges. In fact, there's more and more programs which I used to start in the background from ~/.xinitrc with the & control operator which I now start with systemd. It gives me more control (e.g. in keeping ClipIt reasonable), it enables me to make sure processes run as services aren't started multiple times and I'm sure there's many other benefits in doing this which I'll discover along the way.

The service files which I would write to /etc/systemd/system, I write to ~/.config/systemd/user the same way I would for system-wide services, except that the target would usually be default.target instead of multi-user.target. Otherwise, just use systemctl as you usually would, just by adding the --user option.

5 Timers Instead of Cron Jobs

Services aren't the only things systemd is good at managing. In particular, I stumbled upon so-called timers which can be used instead of cron jobs. Not surprisingly, it comes with a wealth of features, some of which missing from cron but which Anacron does offer, e.g. running jobs at intervals during uptime. I had tried a number of cron-like solutions until I lost patience and wrote my own micron. But systemd does everything I need. Thanks to its timers, I can single-handedly schedule jobs which I had on my desktop that stayed on at all time on my laptop which is half the time sleeping.

A timer runs a service and, as such, setting up a timer entails configuring the timer itself on the one hand, and configuring the service it runs on the other hand. So you end up writing both a .timer and a .service file. For instance, a job that runs kinit -R often enough to renew my Kerberos credentials has its timer configured in kinit.timer:

[Timer]
OnCalendar=08,20:00
Persistent=true

[Install]
WantedBy=timers.target

It is to run at 8 o'clock every morning, 8 o'clock every evening and be a bit persistent about it in that it should run immediately if the previous time it ran was before one of the scheduled times, e.g. while my laptop was sleeping. Also note that a timer expects a target. The service needs to have a corresponding name, kinit.service, here:

[Service]
ExecStart=/usr/bin/kinit -R
Restart=on-failure
RestartSec=30m

Note that there is no need to specify a target here, in that it's not at service I wish to start e.g. at boot. Also note how I wielded again some of the useful options systemd has to offer: should kinit -R fail, e.g. I've got no network, try again in 30 minutes.

The way time and intervals are specified in systemd is convenient. You can omit parts of time, e.g. 08,20:00 here means 8 o'clock in the morning and evening everyday. You can define ranges or lists of times, e.g. 08,20:00. You can use units, e.g. 30m means 30 minutes. You can define timers by specifying calendar times such as I've done here by aiming for 8 o'clock in the morning and evening. Or I could have specified an interval relative to the last time the timer was activated. Or relative to the last time the service was activated. Or even relative to when the system last booted. The reason I didn't, is that Persistent is effective only with OnCalendar timers.

6 Outlook

It's surprising how long I'd been hearing about systemd and never bothered to give it a try. I haven't got to even starting to look into it seriously, that it already trivially solved several of the problems I had. If I didn't like running my software as services with init before – now I enjoy it so much with systemd there's no stopping me anymore. There's several interesting aspects I can't wait to learn about next, pertaining to managing mounted filesystems, working with the journal, making temporary files. One page I quite liked reading to get started was the systemd page on ArchWiki.

7 References

• systemd official website
• systemd on ArchWiki
• autoumount
• python-daemon
• logging — Logging facility for Python
• init
• LSBInitScripts
• Upstart
• Supervisord
• ClipIt
• micron