An encrypted server in a datacenter reboots and stops in the initramfs, waiting for a passphrase at a console nobody is sitting at. 3 commands and a kernel argument get NetworkManager and sshd into a Fedora or Red Hat Enterprise Linux (RHEL) initramfs, so a remote server with an encrypted root can be unlocked over the network.
If you run an encrypted server that lives somewhere other than under your desk, you have met this problem where the machine reboots, stops in the initramfs, and waits for someone to type a passphrase (figure 1).
The fix is to put a network stack and an SSH server in the initramfs, log in while it is waiting, and hand it the passphrase from wherever you are. On Fedora and RHEL, this is close to a solved problem, because dracut already knows how to run NetworkManager. It's 1 package, 1 config file, and 1 kernel argument.
The recipe
Install dracut-sshd from EPEL and add the nm dracut module config file:
dnf install dracut-sshd
echo 'add_dracutmodules+=" network-manager "' > /etc/dracut.conf.d/90-nm-initrd.confAdd rd.neednet=1 ip=dhcp to GRUB_CMDLINE_LINUX in /etc/default/grub, then rebuild both:
dracut -f
grub2-mkconfig -o /boot/grub2/grub.cfgdracut-sshd picks up your existing /root/.ssh/authorized_keys and host keys unless you give it dedicated ones, so there is usually nothing else to configure.
Reboot, and while the machine is sitting at the prompt:
$ ssh root@server
Welcome to the early boot SSH environment. You may type
systemd-tty-ask-password-agent
(or press "arrow up") to unlock your disks.
initramfs-ssh:/root# systemd-tty-ask-password-agent
🔐 Please enter passphrase for disk luks-ffe26397-d8be-4f20-99d6-a899886f1169:
(press TAB for no echo) •••••••••••
initramfs-ssh:/root# Connection to server closed by remote host.The session dying is the success signal. sshd lives in the initramfs, so unlocking the root is what destroys it. dracut-sshd prints that banner on login, and seeds the same command into root's .bash_history, so Up arrow works too.
The prompt names the LUKS device the way /etc/crypttab does, so it's luks-<uuid> rather than a friendly name on an Anaconda-partitioned system.
Why rd.neednet=1
Without a network root, dracut has no reason to bring networking up. It only creates the flag file /run/NetworkManager/initrd/neednet when something asks for a network, and that flag is what gates the unit: nm-initrd.service carries ConditionPathExists=/run/NetworkManager/initrd/neednet. No flag means the unit is skipped, and a skipped unit is not a failed unit.
ip=dhcp tells nm-initrd-generator what to configure. For a static address, the syntax is the familiar ip=<client-ip>:<peer>:<gateway>:<netmask>:<hostname>:<interface>:<method> form, but note that NetworkManager's parser diverges from dracut's in places. An empty method means auto rather than dhcp, and an unrecognized method is not an error, it quietly becomes auto.
How it works
NetworkManager upstream ships 3 units: NetworkManager-config-initrd.service, NetworkManager-initrd.service, and NetworkManager-wait-online-initrd.service. It also provides a systemd generator that sets those up. But on Fedora and RHEL, the spec file deletes them:
# Don't use the *-initrd.service files yet, wait dracut to support them
rm -f %{buildroot}%{_systemdgeneratordir}/nm-initrd-generator.sh
rm -f %{buildroot}%{_unitdir}/NetworkManager-config-initrd.service
...That comment is easy to misread. It is not saying initrd networking is unfinished, or that you should wait for something. Dracut has run NetworkManager in the initramfs for years. It means dracut does not consume NetworkManager's own units, so shipping them alongside dracut's would just be two implementations of the same thing in a single image.
What you get instead is dracut's 35network-manager module driving dracut's own nm-initrd.service. It means every search result for nm-initrd.service is about dracut's unit, not NetworkManager's, and the 2 are not the same thing.
The binary is located at /usr/libexec/nm-initrd-generator. On distributions with libexecdir=/usr/lib, it is /usr/lib/nm-initrd-generator, which is why dracut's module globs for both.
Inside the initramfs, from the SSH session, this is what a working setup looks like:
# nmcli -t -f NAME,DEVICE,STATE con show
Wired Connection:enp1s0:activated
lo:lo:activated
# ls /run/NetworkManager/initrd/
neednet
# ls /run/systemd/ask-password/
ask.12669764168055723842 sck.aa80e1a1e7ec9ded
# systemctl list-units --no-legend '*nm*'
nm-initrd.service loaded active running nm-initrd.service
nm-wait-online-initrd.service loaded active exited nm-wait-online-initrd.serviceThe profile follows you into the real root
nm-initrd-generator writes keyfile profiles into /run/NetworkManager/system-connections. /run is not copied at switch-root, it is carried over, and the NetworkManager on the real root reads that same directory through its keyfile plugin. So the connection configured in the initramfs stays configured, with the same UUID:
Wired Connection:10b17d77-2bbc-401d-9229-5c76d2f3abf5:/run/NetworkManager/system-connections/default_connection.nmconnectionThat is deliberate, and it is also the answer to where that Wired Connection profile you never created came from.
Before you rely on it
Your initramfs now contains an SSH host key and an authorized_keys, unencrypted, on /boot. Anyone with physical access to the disk has both. Generate a host key for this purpose rather than reusing the system one, so the key sitting on the unencrypted partition is not the key that identifies your running system. dracut-sshd uses /etc/ssh/dracut_ssh_host_*_key if you create them.
The initramfs and the real system present different host keys, so expect known_hosts complaints on every unlock. Pin them separately rather than turning host key checking off.
Test on a machine you can physically reach before you rely on this for one you cannot. An initramfs that does not come up on the network and has no console means a trip to wherever the box lives or dealing with BMCs.
Conclusion
Just 1 package, 1 config file, 1 kernel argument, and the encrypted machine you could not reach is now one ssh away from finishing its boot. The whole recipe fits in a paragraph because dracut and NetworkManager already did the integration and you are only turning it on.
The 2 things to remember when it does not work: rd.neednet=1 is what makes the unit run at all, and a missing flag file looks exactly like a normal boot. Check /run/NetworkManager/initrd/neednet from a rescue shell before you suspect anything more interesting.
