[pve-devel] [PATCH docs v2 1/2] Add documentation on bootloaders (systemd-boot)

[Stoiko Ivanov]

With the recently added support for booting ZFS on root on EFI systems via
`systemd-boot` the documentation needs adapting (mostly related to editing
the kernel commandline).

This patch adds a short section on Bootloaders to the sysadmin chapter
describing both `grub` and PVE's use of `systemd-boot`

Signed-off-by: Stoiko Ivanov <s.ivanov at proxmox.com>
---
 sysadmin.adoc       |   2 +
 system-booting.adoc | 174 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 176 insertions(+)
 create mode 100644 system-booting.adoc

diff --git a/sysadmin.adoc b/sysadmin.adoc
index 21537f1..e045610 100644
--- a/sysadmin.adoc
+++ b/sysadmin.adoc
@@ -74,6 +74,8 @@ include::local-zfs.adoc[]
 
 include::certificate-management.adoc[]
 
+include::system-booting.adoc[]
+
 endif::wiki[]
 
 
diff --git a/system-booting.adoc b/system-booting.adoc
new file mode 100644
index 0000000..38dacb1
--- /dev/null
+++ b/system-booting.adoc
@@ -0,0 +1,174 @@
+[[system_booting]]
+Bootloaders
+-----------
+ifdef::wiki[]
+:pve-toplevel:
+endif::wiki[]
+
+{pve} uses one of two bootloaders depending on the disk setup selected in the
+installer.
+
+For EFI Systems installed with ZFS as the root filesystem `systemd-boot` is
+used. All other deployments use the standard `grub` bootloader (this usually
+also applies to systems which are installed on top of Debian).
+
+[[installer_partitioning_scheme]]
+Partitioning scheme used by the installer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The {pve} installer creates 3 partitions on the bootable disks selected for
+installation. The bootable disks are:
+
+* For Installations with `ext4` or `xfs` the selected disk
+
+* For ZFS installations all disks belonging to the first `vdev`:
+** The first disk for RAID0
+** All disks for RAID1, RAIDZ1, RAIDZ2, RAIDZ3
+** The first two disks for RAID10
+
+The created partitions are:
+
+* a 1 MB BIOS Boot Partition (gdisk type EF02)
+
+* a 512 MB EFI System Partition (ESP, gdisk type EF00)
+
+* a third partition spanning the set `hdsize` parameter or the remaining space
+    used for the chosen storage type
+
+`grub` in BIOS mode (`--target i386-pc`) is installed onto the BIOS Boot
+Partition of all bootable disks for supporting older systems.
+
+
+Grub
+~~~~
+
+`grub` has been the de-facto standard for booting Linux systems for many years
+and is quite well documented
+footnote:[Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html].
+
+The kernel and initrd images are taken from `/boot` and its configuration file
+`/boot/grub/grub.cfg` gets updated by the kernel installation process.
+
+Configuration
+^^^^^^^^^^^^^
+Changes to the `grub` configuration are done via the defaults file
+`/etc/default/grub` or config snippets in `/etc/default/grub.d`. To regenerate
+the `/boot/grub/grub.cfg` after a change to the configuration run:
+
+----
+`update-grub`.
+----
+
+Systemd-boot
+~~~~~~~~~~~~
+
+`systemd-boot` is a lightweight EFI bootloader. It reads the kernel and initrd
+images directly from the EFI Service Partition (ESP) where it is installed.
+The main advantage of directly loading the kernel from the ESP is that it does
+not need to reimplement the drivers for accessing the storage.  In the context
+of ZFS as root filesystem this means that you can use all optional features on
+your root pool instead of the subset which is also present in the ZFS
+implementation in `grub` or having to create a separate small boot-pool
+footnote:[Booting ZFS on root with grub https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS].
+
+In setups with redundancy (RAID1, RAID10, RAIDZ*) all bootable disks (those
+being part of the first `vdev`) are partitioned with an ESP. This ensures the
+system boots even if the first boot device fails.  The ESPs are kept in sync by
+a kernel postinstall hook script `/etc/kernel/postinst.d/zz-pve-efiboot`. The
+script copies certain kernel versions and the initrd images to `EFI/proxmox/`
+on the root of each ESP and creates the appropriate config files in
+`loader/entries/proxmox-*.conf`.
+
+The following kernel versions are configured by default:
+
+* the currently running kernel
+* the version being newly installed on package updates
+* the two latest kernels
+* the latest version of each kernel series (e.g. 4.15, 5.0).
+
+The ESPs are not kept mounted during regular operation, in contrast to `grub`,
+which keeps an ESP mounted on `/boot/efi`. This helps to prevent filesystem
+corruption to the `vfat` formatted ESPs in case of a system crash, and removes
+the need to manually adapt `/etc/fstab` in case the primary boot device fails.
+
+[[systemd_boot_config]]
+Configuration
+^^^^^^^^^^^^^
+
+`systemd-boot` is configured via the file `loader/loader.conf` in the root
+directory of an EFI System Partition (ESP). See the `loader.conf(5)` manpage
+for details.
+
+Each bootloader entry is placed in a file of its own in the directory
+`loader/entries/`
+
+An example entry.conf looks like this (`/` refers to the root of the ESP):
+
+----
+title    Proxmox
+version  5.0.15-1-pve
+options   root=ZFS=rpool/ROOT/pve-1 boot=zfs
+linux    /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
+initrd   /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
+----
+
+
+.Manually keeping a kernel bootable
+
+Should you wish to add a certain kernel and initrd image to the list of
+bootable kernels you need to:
+
+* create a directory on the ESP (e.g. `/EFI/personalkernel`)
+* copy the kernel and initrd image to that directory
+* create a entry for this kernel in `/loader/entries/*.conf`
+
+NOTE: do not use `/EFI/proxmox` as directory since all entries there can be
+removed by `/etc/kernel/postinst.d/zz-pve-efiboot`
+
+Example (keeping kernel 5.0.15-1-pve and copying to an ESP mounted on
+`/mnt/esp`):
+
+----
+mkdir /mnt/esp/EFI/preferred-kernel
+cp /boot/initrd.img-5.0.15-1-pve /boot/vmlinuz-5.0.15-1-pve /mnt/esp/EFI/preferred-kernel
+echo -n "title Preferred Kernel
+version 5.0.15-1-pve
+linux /mnt/esp/EFI/preferred-kernel/vmlinuz-5.0.15-1-pve
+initrd /mnt/esp/EFI/preferred-kernel/initrd.img-5.0.15-1-pve
+options " > /mnt/esp/loader/entries/preferred.conf
+cat /etc/kernel/cmdline >> /mnt/esp/loader/entries/preferred.conf
+----
+
+
+[[systemd-boot-refresh]]
+.Updating the configuration on all ESPs
+
+To copy and configure all bootable kernels and keep all ESPs in sync you just
+need to run the kernel hook script `/etc/kernel/postinst.d/zz-pve-efiboot`.
+(The equivalent to running `update-grub` on Systems being booted with `grub`).
+
+This is necessary should you make changes to the kernel commandline, or if you
+want to add another ESP (e.g. when replacing a failed disk in a redundant ZFS
+pool).
+
+[[edit_kernel_cmdline]]
+Editing the kernel commandline
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can modify the kernel commandline in the following places, depending on the
+bootloarder used:
+
+.Grub
+
+The kernel commandline needs to be placed in the variable
+`GRUB_CMDLINE_LINUX_DEFAULT` in the file `/etc/default/grub`. Running
+`update-grub` appends its content to all `linux` entries in
+`/boot/grub/grub.cfg`.
+
+.Systemd-boot
+
+The kernel commandline needs to be placed as line in `/etc/kernel/cmdline`
+Running `/etc/kernel/postinst.d/zz-pve-efiboot` sets it as `option` line for
+all config files in `loader/entries/proxmox-*.conf`.
+
+
-- 
2.20.1