Race condition between `sops-install-secrets` and `postfix-setup-start`

[jfly]

Seen on umbriel. Note how sops-install-secrets starts, postmap crashes, and then sops-install-secrets finishes.

May 19 06:29:49 umbriel sops-install-secrets[661]: sops-install-secrets: Imported /etc/ssh/ssh_host_ed25519_key as age key with fingerprint age15vcp7875xwtf64j4yshyld0a3hpgzv6n2kxky493s3q0swr9hdaqxugpv6
May 19 06:29:49 umbriel postfix-setup-start[925]: postmap: fatal: open /var/lib/postfix/conf/virtual-mailing-lists: No such file or directory
May 19 06:29:49 umbriel postfix/postmap[925]: fatal: open /var/lib/postfix/conf/virtual-mailing-lists: No such file or directory
...
May 19 06:29:49 umbriel systemd[1]: Finished sops-install-secrets.service.

[jfly]

I believe this started with 23582ea, where we updated sops-nix from nix-community/srvos@c4a21c4 to nix-community/srvos@6c9ab84.

The relevant change is that nix-community/srvos@5e4120c enabled userborn by default, which then triggered us to start using sops-nix's useSystemdActivation.

Clearly we're missing a dependency between sops-install-secrets.service and postfix-setup.service. What I don't know is what the correct fix is. Is sops-install-secrets.service missing a Before directive? Or should sops-install-secrets.service run after sysinit-reactivation.target (which sops-install-secrets runs before) somehow? I could use the help of someone with more knowledge about how these things are supposed to be ordered. @Mic92?

[SuperSandro2000]

We set after for sops-install-secrets for dovecot, hyperkitty*, mailman and postfix and also postfix-setup. Maybe see https://gitea.c3d2.de/c3d2/nix-config/src/branch/master/hosts/mail/default.nix for some inspiration but no guarantees that it is correct.

[Mic92]

@jfly could you print the critical path of the systemd services?

[jfly]

@Mic92, is this what you want?

[root@umbriel:~]# systemd-analyze critical-chain sops-install-secrets.service postfix-setup.service
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

sops-install-secrets.service +170ms
└─userborn.service @1.594s +179ms
  └─systemd-remount-fs.service @1.374s +177ms
    └─systemd-journald.socket
      └─system.slice
        └─-.slice
postfix-setup.service +662ms
└─basic.target @3.023s
  └─dbus-broker.service @3.452s +150ms
    └─dbus.socket @3.008s +139us
      └─sysinit.target @2.972s
        └─systemd-vconsole-setup.service @3.508s +180ms
          └─systemd-journald.socket
            └─system.slice
              └─-.slice

[jfly]

Discussed in today's team meeting. The dependency between postfix-setup.service and basic.target smells off here. It should probably depend on multi-user.target instead. I asked @Mic92 if it would be possible to get secrets installed before basic.target, but secret installation requires users (via userborn) and that might also require basic.target.

So the fix might be to just add a wantedBy = [ "multi-user.target" ]; to postfix-setup.service, but it sure would be nice to make this less footgun-y.