LUCHS.AT - System Administration - The Eternal systemd Discussion and Implications for System Administration

Recently we had a power outage at one of our sites. A construction site brought down a power line, and the subsequent efforts to repair the damage took the power company longer than the UPS could handle. So all systems shut down cleanly. After the power was back most servers restarted automatically. Two servers were missing. A manual inspection revealed that they hung still in the init phase. According to the logs on the screen systemd was trying to unlock the encrypted harddrives (with a key stored on an external storage media, accessible by the init system). The system were rebooted in Debian 8 System V init mode, and they started immediately. No hangs, no errors, nothing. All systems go, as it should be.

Out of frustration this tweet went out to the world. Unsurprisingly comments about the init system religious war came as reply.

We did not report the error on the servers to the developers for a simple reason. systemd is advertised as a drop in replacement for System V init scripts. It clearly anything but a drop in replacement. If you have to do a migration from System V to systemd, then it is not a drop in replacement. Furthermore everything necessary to get these two servers up and running is stored in their /etc/crypttab and /etc/fstab. It is not really hard to unlock a LUKS-encrypted partition. Even a shell script can do this.

So to cut the bashing short, here is the message: If you want to see your software deployed, do not anger the users (or the sysadmins) using it. We have ditched a lot of software applications, because of misguided development processes. We will replace Debian with Devuan as a lesson from this episode. We already replaced a lot of window managers, because of the ever increasing feature creep, cruft, and bloat. Do not go this way when developing software. Focus!