[pve-devel] [PATCH docs] sysadmin: add section 'Firmware Updates' and references

Wed Jul 26 12:58:15 CEST 2023

On 7/26/23 12:26, Alexander Zeidler wrote:
> Firmware updates are important, their existence should not be checked
> only when there are already noticeable problems.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler at proxmox.com>
> ---
>   firmware-updates.adoc | 60 +++++++++++++++++++++++++++++++++++++++++++
>   local-lvm.adoc        |  4 ++-
>   qm.adoc               | 15 +++++------
>   sysadmin.adoc         |  4 +++
>   4 files changed, 74 insertions(+), 9 deletions(-)
>   create mode 100644 firmware-updates.adoc
> 
> diff --git a/firmware-updates.adoc b/firmware-updates.adoc
> new file mode 100644
> index 0000000..ef7df78
> --- /dev/null
> +++ b/firmware-updates.adoc
> @@ -0,0 +1,60 @@
> +[[chapter_firmware_updates]]
> +Firmware Updates
> +----------------
> +ifdef::wiki[]
> +:pve-toplevel:
> +endif::wiki[]
> +
> +Firmware updates from this chapter should be applied when running {pve} on a bare-metal server. Whether configuring firmware updates is appropriate within guests, e.g. when using device pass-through, depends strongly on your setup and is therefore out of scope.
> +
> +Regular firmware updates for devices are just as important for proper operation as regular software updates. There are several ways to obtain and apply those updates. The methods listed in this chapter can also be combined to minimize the chance of missing an important update.
> +
> +TIP: When a firmware was updated, a system reboot is the safest way to apply the new version.
> +
> +
> +[[sysadmin_firmware_persistent]]
> +Persistent Firmware
> +~~~~~~~~~~~~~~~~~~~
> +The following methods write the new firmware permanently to the respective device. The firmware therefore remains up to date regardless of the booted operating system.
> +
> +TIP: When using a user space application or 'fwupd', the hardware must usually have been manufactured after 2014, the system must have been booted with UEFI and the EFI partition manually mounted.
> +
> +CAUTION: When updating the BIOS/UEFI itself, its settings are usually reset. Be prepared to reconfigure them afterwards.
> +
> +
> +[[sysadmin_firmware_persistent_vendor_specific]]
> +Vendor-specific
> +^^^^^^^^^^^^^^^
> +Depending on the vendor and hardware, there is a convenient way to update the firmware via an Intelligent Platform Management Interface (IPMI) or a provided user space application. If these are not available, each device firmware can be updated via the respective vendor's website, which is comparatively time-consuming and error-prone.

I would rewrite this a bit, as especially for server HW, this is the main 
option. Maybe something like this?

Firmware updates are usually available from the vendor directly. Please check 
with your vendor what options are available.

Depending on the platform and vendor, there are convenient methods available. 
For servers, for example, Dell's Lifecycle Manager or Service Packs from HPE.
Sometimes there are Linux utilities available as well. Examples are 
https://network.nvidia.com/support/firmware/mlxup-mft/['mlxup'] for NVIDIA 
ConnectX or 
https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/adapters/software-installation/updating-the-firmware/manually-updating-the-adapter-firmware-on-linuxesx.html['bnxtnvm'/'niccli'] 
for Broadcom network cards.

> +
> +
> +[[sysadmin_firmware_persistent_lvfs_fwupd]]
> +Linux Vendor Firmware Service (LVFS) via fwupd
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +On https://fwupd.org['LVFS'], vendors can make their firmware updates available in a standardized way to a wide range of Linux hosts. Here is the growing list of participating https://fwupd.org/lvfs/vendors/[vendors] and their currently supported https://fwupd.org/lvfs/devices/[devices].
> +
> +To use 'fwupd', manually mount your https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot_installer_part_scheme[EFI System Partition] (ESP) you booted from on `/boot/`. After installing the package 'fwupd', update firmware with the following commands:
> +----
> +# fwupdmgr refresh
> +# fwupdmgr get-updates
> +# fwupdmgr update
> +# reboot
> +----
> +
> +
> +[[sysadmin_firmware_runtime_files]]
> +Runtime Firmware Files
> +~~~~~~~~~~~~~~~~~~~~~~
> +The following methods keep the firmware files available at the {pve} host and do not persist it on the device itself. Whenever a device is initialized, usually during the boot process, the corresponding firmware is loaded into the RAM of the respective device. These methods do not provide and can not update firmware that is used in the very early boot process (e.g. BIOS/UEFI, hard disks).
> +
> +In {pve} the package `pve-firmware` is already installed by default. Therefore, with the normal system updates (APT), the included firmware of common hardware is automatically kept up to date. Be aware that CPU microcode updates are located in a separate Debian repository component, which is not configured by default.
> +
> +
> +[[sysadmin_firmware_runtime_files_debian_repo]]
> +Debian Firmware Repository
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +Starting with Debian Bookworm ({pve} 8) non-free firmware (as defined by https://www.debian.org/social_contract#guidelines[DFSG]) has been moved to the newly created Debian repository component `non-free-firmware`. It contains firmware for CPUs (called microcode) as well as other firmware. In the past, CPUs repeatedly had security vulnerabilities beside other issues. Using this update method (additional) to apply microcode updates is convenient, safe and fast.
> +
> +To be able to install microcode updates or other firmware from the `non-free-firmware` component, edit the file `/etc/apt/sources.list`, append `non-free-firmware` to the end of each of the three Debian repository lines and run `apt-get update`.
> +
> +To keep the CPU microcode up to date, depending on the vendor, install the package `intel-microcode` or `amd64-microcode` and reboot your {pve} host afterwards.
> \ No newline at end of file
> diff --git a/local-lvm.adoc b/local-lvm.adoc
> index 8092bb8..b6009c8 100644
> --- a/local-lvm.adoc
> +++ b/local-lvm.adoc
> @@ -64,7 +64,9 @@ Bootloader
>   
>   We install two boot loaders by default. The first partition contains
>   the standard GRUB boot loader. The second partition is an **E**FI **S**ystem
> -**P**artition (ESP), which makes it possible to boot on EFI systems.
> +**P**artition (ESP), which makes it possible to boot on EFI systems and to
> +apply xref:sysadmin_firmware_persistent[persistent firmware updates] from the
> +user space.
>   
>   
>   Creating a Volume Group
> diff --git a/qm.adoc b/qm.adoc
> index b3c3034..c4f1024 100644
> --- a/qm.adoc
> +++ b/qm.adoc
> @@ -352,9 +352,9 @@ CPU Type
>   
>   QEMU can emulate a number different of *CPU types* from 486 to the latest Xeon
>   processors. Each new processor generation adds new features, like hardware
> -assisted 3d rendering, random number generation, memory protection, etc.. Also,
> -a current generation can be upgraded through microcode update with bug or
> -security fixes.
> +assisted 3d rendering, random number generation, memory protection, etc. Also,
> +a current generation can be upgraded through
> +xref:chapter_firmware_updates[microcode update] with bug or security fixes.
>   
>   Usually you should select for your VM a processor type which closely matches the
>   CPU of the host system, as it means that the host CPU features (also called _CPU
> @@ -460,10 +460,9 @@ editing the CPU options in the WebUI, or by setting the 'flags' property of the
>   'cpu' option in the VM configuration file.
>   
>   For Spectre v1,v2,v4 fixes, your CPU or system vendor also needs to provide a
> -so-called ``microcode update'' footnote:[You can use `intel-microcode' /
> -`amd-microcode' from Debian non-free if your vendor does not provide such an
> -update. Note that not all affected CPUs can be updated to support spec-ctrl.]
> -for your CPU.
> +so-called ``microcode update'' for your CPU, see
> +xref:chapter_firmware_updates[chapter Firmware Updates]. Note that not all
> +affected CPUs can be updated to support spec-ctrl.
>   
>   
>   To check if the {pve} host is vulnerable, execute the following command as root:
> @@ -472,7 +471,7 @@ To check if the {pve} host is vulnerable, execute the following command as root:
>   for f in /sys/devices/system/cpu/vulnerabilities/*; do echo "${f##*/} -" $(cat "$f"); done
>   ----
>   
> -A community script is also available to detect is the host is still vulnerable.
> +A community script is also available to detect if the host is still vulnerable.
>   footnote:[spectre-meltdown-checker https://meltdown.ovh/]
>   
>   Intel processors
> diff --git a/sysadmin.adoc b/sysadmin.adoc
> index d3de40a..dd43f73 100644
> --- a/sysadmin.adoc
> +++ b/sysadmin.adoc
> @@ -32,6 +32,8 @@ See Also
>   
>   * link:/wiki/System_Software_Updates[System Software Updates]
>   
> +* link:/wiki/Firmware_Updates[Firmware Updates]
> +
>   * link:/wiki/Host_Bootloader[Host Bootloader]
>   
>   * link:/wiki/Time_Synchronization[Time Synchronization]
> @@ -59,6 +61,8 @@ include::pve-package-repos.adoc[]
>   
>   include::system-software-updates.adoc[]
>   
> +include::firmware-updates.adoc[]
> +
>   include::pve-network.adoc[]
>   
>   include::system-timesync.adoc[]