[PVE-User] ZFS Upgrade...

[Marco Gaiarin]

We have upgraded our server from PVE6 to PVE8, waited some month to be sure
there's no incompatibilities and now we are ready to remove older kernels,
and upgrade ZFS.

Current situation is:

 root at svpve3:~# zfs upgrade
 This system is currently running ZFS filesystem version 5.
 
 All filesystems are formatted with the current version.

and:

 root at svpve3:~# zpool upgrade
 This system supports ZFS pool feature flags.
 
 All pools are formatted using feature flags.
 
 
 Some supported features are not enabled on the following pools. Once a
 feature is enabled the pool may become incompatible with software
 that does not support the feature. See zpool-features(7) for details.
 
 Note that the pool 'compatibility' feature can be used to inhibit
 feature upgrades.
 
 POOL  FEATURE
 ---------------
 rpool
       draid
       zilsaxattr
       head_errlog
       blake3
       block_cloning
       vdev_zaps_v2
 rpool-backup
       draid
       zilsaxattr
       head_errlog
       blake3
       block_cloning
       vdev_zaps_v2

So, there's no zfs upgrade needed, but zpool is.


First question: i've read:
	https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_features
and on detail:
	A system with root on ZFS, that still boots using GRUB will become unbootable if a new feature is active on the rpool, due to the incompatible implementation of ZFS in GRUB. 

looking at 'zpool-features', GRUB 2.06 compatible features seems on file
/usr/share/zfs/compatibility.d/grub2-2.06, that report:

 # Features which are supported by GRUB2 versions prior to v2.12.
 #
 # GRUB is not able to detect ZFS pool if snaphsot of top level boot pool
 # is created. This issue is observed with GRUB versions before v2.12 if
 # extensible_dataset feature is enabled on ZFS boot pool.
 #
 # This file lists all read-only comaptible features except
 # extensible_dataset and any other feature that depends on it.

After upgrading the non-boot ZFS pool, i get this:

 root at svpve3:~# zpool get all rpool-backup | grep feature@ | grep enabled | sort
 rpool-backup  feature at allocation_classes     enabled                        local
 rpool-backup  feature at async_destroy          enabled                        local
 rpool-backup  feature at blake3                 enabled                        local
 rpool-backup  feature at block_cloning          enabled                        local
 rpool-backup  feature at bookmarks              enabled                        local
 rpool-backup  feature at bookmark_v2            enabled                        local
 rpool-backup  feature at bookmark_written       enabled                        local
 rpool-backup  feature at device_rebuild         enabled                        local
 rpool-backup  feature at device_removal         enabled                        local
 rpool-backup  feature at draid                  enabled                        local
 rpool-backup  feature at edonr                  enabled                        local
 rpool-backup  feature at enabled_txg            active                         local
 rpool-backup  feature at encryption             enabled                        local
 rpool-backup  feature at filesystem_limits      enabled                        local
 rpool-backup  feature at large_dnode            enabled                        local
 rpool-backup  feature at livelist               enabled                        local
 rpool-backup  feature at multi_vdev_crash_dump  enabled                        local
 rpool-backup  feature at obsolete_counts        enabled                        local
 rpool-backup  feature at redacted_datasets      enabled                        local
 rpool-backup  feature at redaction_bookmarks    enabled                        local
 rpool-backup  feature at resilver_defer         enabled                        local
 rpool-backup  feature at sha512                 enabled                        local
 rpool-backup  feature at skein                  enabled                        local
 rpool-backup  feature at vdev_zaps_v2           enabled                        local
 rpool-backup  feature at zilsaxattr             enabled                        local
 rpool-backup  feature at zpool_checkpoint       enabled                        local
 rpool-backup  feature at zstd_compress          enabled                        local

but it is unclear to me how to check compatibility issue: seems to there's
no 'extensible_dataset' so i'm safe... or better, seems active BOTH on
upgraded and non-upgraded dataset, and was not on feature list to be enabled
by upgrade...

 root at svpve3:~# zpool get all rpool | grep feature@ | grep extensible_dataset
 rpool  feature at extensible_dataset     active                         local

 root at svpve3:~# zpool get all rpool-backup | grep feature@ | grep extensible_dataset
 rpool-backup  feature at extensible_dataset     active                         local


There's some better description of these incompatibilities, or some
reference ZFS or GRUB docs about this?


Thanks.

--