[pve-devel] [RFC qemu-server] fix #6608: expose viommu driver aw-bits option

[Daniel Kral]

On Wed Aug 27, 2025 at 5:03 PM CEST, Daniel Kral wrote:
> Since QEMU 9.2 [0], the default I/O address space bit width was raised
> from 39 bits to 48 bits for the Intel vIOMMU driver, which makes the
> aw-bits check introduced in [1] to trip for host CPUs with less than 48
> bits physical address width from QEMU 9.2 onwards:
>
> vfio 0000:XX:YY.Z: Failed to set vIOMMU: aw-bits 48 > host aw-bits 39
>
> For VFIO devices where a vIOMMU is in-use, QEMU fetches the IOVA ranges
> with the iommufd ioctl IOMMU_IOAS_IOVA_RANGES or the vfio_iommu_type1's
> VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE info, so 'phys-bits' doesn't change
> the behavior of the check.
>
> Therefore, expose the 'aw-bits' option of the intel-iommu and
> virtio-iommu QEMU drivers to allow users to set the value.
>
> [0] https://lore.kernel.org/qemu-devel/20241212083757.605022-17-zhenzhong.duan@intel.com/
> [1] https://lore.kernel.org/qemu-devel/20240605083043.317831-18-zhenzhong.duan@intel.com/
>
> Signed-off-by: Daniel Kral <d.kral at proxmox.com>
> ---
> There were quite a few changes in the way in qemu upstream since 9.0 for
> the vIOMMU drivers to utilize the Intel VT-d's dual-stage vIOMMU
> translation better, but I'm not entirely sure why the default value was
> changed for legacy mode too, i.e. when scalable mode (x-scalable-mode)
> and first level translation support (x-flts) is off, as I haven't looked
> into it too much whether there are any strict requirements for this in
> the future when 5-level paging is supported.
>
> My CPU itself reports 39 bits physical address size according to
> /proc/cpuinfo and setting aw-bits=39 made the check mentioned above
> happy and the VM startable again. I haven't tested this yet with any CPU
> that has 46 or 48 bit physical address width.

A user reported [0] that both errors vanished (the one mentioned above
in the commit message and the vfio_container_dma_map(...) = -22 one) by
setting the combination of cpu.guest-phys-bits and intel-iommu.aw-bits
so that these are equal on systems where these differ.

It seems like mostly Intel consumer-grade CPUs are the ones where these
mismatch or are below the default 48 bits - it seems the physical
address width ranges from anywhere between 39 and 48 bits on Intel CPUs;
the other 2 AMD CPUs I checked were both 48 bits physical address width
- even though these were quite beefy enthusiast 7900X / 9900X ones.

There was a patch that wasn't applied in qemu upstream [1] that should
warn users about the mismatch but wasn't perfect as one can see in the
replies.

I'll follow up on this patch with a possible check that compares the
cpu's physical bits (or the phys-bits / guest-phys-bits) to the IOMMU's
address width size, which can be found through the iommu's sysfs.

[0] https://forum.proxmox.com/threads/169586/post-795813
[1] https://lore.kernel.org/qemu-devel/20250130134346.1754143-9-clg@redhat.com/