FW: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
DERUMIER, Alexandre
alexandre.derumier at groupe-cyllene.com
Wed Aug 14 15:05:35 CEST 2024
>>I'm talking about virtio-scsi. Our virtio-network device is working
>>fine
Yes, sorry, I wanted to said virtio-scsi.
All pci devices excluding passthrough devices (with pcie=on flag) are
actually plugged to pci bridge
sub get_pci_addr_map {
$pci_addr_map = {
piix3 => { bus => 0, addr => 1, conflict_ok => qw(ehci) },
ehci => { bus => 0, addr => 1, conflict_ok => qw(piix3) }, #
instead of piix3 on arm
vga => { bus => 0, addr => 2, conflict_ok => qw(legacy-igd) },
'legacy-igd' => { bus => 0, addr => 2, conflict_ok => qw(vga)
}, # legacy-igd requires vga=none
balloon0 => { bus => 0, addr => 3 },
watchdog => { bus => 0, addr => 4 },
scsihw0 => { bus => 0, addr => 5, conflict_ok => qw(pci.3) },
'pci.3' => { bus => 0, addr => 5, conflict_ok => qw(scsihw0)
}, # also used for virtio-scsi-single bridge
scsihw1 => { bus => 0, addr => 6 },
ahci0 => { bus => 0, addr => 7 },
qga0 => { bus => 0, addr => 8 },
spice => { bus => 0, addr => 9 },
virtio0 => { bus => 0, addr => 10 },
virtio1 => { bus => 0, addr => 11 },
virtio2 => { bus => 0, addr => 12 },
virtio3 => { bus => 0, addr => 13 },
virtio4 => { bus => 0, addr => 14 },
virtio5 => { bus => 0, addr => 15 },
hostpci0 => { bus => 0, addr => 16 },
hostpci1 => { bus => 0, addr => 17 },
net0 => { bus => 0, addr => 18 },
net1 => { bus => 0, addr => 19 },
net2 => { bus => 0, addr => 20 },
net3 => { bus => 0, addr => 21 },
net4 => { bus => 0, addr => 22 },
net5 => { bus => 0, addr => 23 },
vga1 => { bus => 0, addr => 24 },
vga2 => { bus => 0, addr => 25 },
vga3 => { bus => 0, addr => 26 },
hostpci2 => { bus => 0, addr => 27 },
hostpci3 => { bus => 0, addr => 28 },
#addr29 : usb-host (pve-usb.cfg)
'pci.1' => { bus => 0, addr => 30 },
'pci.2' => { bus => 0, addr => 31 },
'net6' => { bus => 1, addr => 1 },
'net7' => { bus => 1, addr => 2 },
'net8' => { bus => 1, addr => 3 },
'net9' => { bus => 1, addr => 4 },
'net10' => { bus => 1, addr => 5 },
'net11' => { bus => 1, addr => 6 },
'net12' => { bus => 1, addr => 7 },
'net13' => { bus => 1, addr => 8 },
'net14' => { bus => 1, addr => 9 },
'net15' => { bus => 1, addr => 10 },
'net16' => { bus => 1, addr => 11 },
'net17' => { bus => 1, addr => 12 },
'net18' => { bus => 1, addr => 13 },
'net19' => { bus => 1, addr => 14 },
'net20' => { bus => 1, addr => 15 },
'net21' => { bus => 1, addr => 16 },
'net22' => { bus => 1, addr => 17 },
'net23' => { bus => 1, addr => 18 },
'net24' => { bus => 1, addr => 19 },
'net25' => { bus => 1, addr => 20 },
'net26' => { bus => 1, addr => 21 },
'net27' => { bus => 1, addr => 22 },
'net28' => { bus => 1, addr => 23 },
'net29' => { bus => 1, addr => 24 },
'net30' => { bus => 1, addr => 25 },
'net31' => { bus => 1, addr => 26 },
'xhci' => { bus => 1, addr => 27 },
'pci.4' => { bus => 1, addr => 28 },
'rng0' => { bus => 1, addr => 29 },
'pci.2-igd' => { bus => 1, addr => 30 }, # replaces pci.2 in
case a legacy IGD device is passed through
'virtio6' => { bus => 2, addr => 1 },
'virtio7' => { bus => 2, addr => 2 },
'virtio8' => { bus => 2, addr => 3 },
'virtio9' => { bus => 2, addr => 4 },
'virtio10' => { bus => 2, addr => 5 },
'virtio11' => { bus => 2, addr => 6 },
'virtio12' => { bus => 2, addr => 7 },
'virtio13' => { bus => 2, addr => 8 },
'virtio14' => { bus => 2, addr => 9 },
'virtio15' => { bus => 2, addr => 10 },
'ivshmem' => { bus => 2, addr => 11 },
'audio0' => { bus => 2, addr => 12 },
hostpci4 => { bus => 2, addr => 13 },
hostpci5 => { bus => 2, addr => 14 },
hostpci6 => { bus => 2, addr => 15 },
hostpci7 => { bus => 2, addr => 16 },
hostpci8 => { bus => 2, addr => 17 },
hostpci9 => { bus => 2, addr => 18 },
hostpci10 => { bus => 2, addr => 19 },
hostpci11 => { bus => 2, addr => 20 },
hostpci12 => { bus => 2, addr => 21 },
hostpci13 => { bus => 2, addr => 22 },
hostpci14 => { bus => 2, addr => 23 },
hostpci15 => { bus => 2, addr => 24 },
'virtioscsi0' => { bus => 3, addr => 1 },
'virtioscsi1' => { bus => 3, addr => 2 },
'virtioscsi2' => { bus => 3, addr => 3 },
'virtioscsi3' => { bus => 3, addr => 4 },
'virtioscsi4' => { bus => 3, addr => 5 },
'virtioscsi5' => { bus => 3, addr => 6 },
'virtioscsi6' => { bus => 3, addr => 7 },
'virtioscsi7' => { bus => 3, addr => 8 },
'virtioscsi8' => { bus => 3, addr => 9 },
'virtioscsi9' => { bus => 3, addr => 10 },
'virtioscsi10' => { bus => 3, addr => 11 },
'virtioscsi11' => { bus => 3, addr => 12 },
'virtioscsi12' => { bus => 3, addr => 13 },
'virtioscsi13' => { bus => 3, addr => 14 },
'virtioscsi14' => { bus => 3, addr => 15 },
'virtioscsi15' => { bus => 3, addr => 16 },
'virtioscsi16' => { bus => 3, addr => 17 },
'virtioscsi17' => { bus => 3, addr => 18 },
'virtioscsi18' => { bus => 3, addr => 19 },
'virtioscsi19' => { bus => 3, addr => 20 },
'virtioscsi20' => { bus => 3, addr => 21 },
'virtioscsi21' => { bus => 3, addr => 22 },
'virtioscsi22' => { bus => 3, addr => 23 },
'virtioscsi23' => { bus => 3, addr => 24 },
'virtioscsi24' => { bus => 3, addr => 25 },
'virtioscsi25' => { bus => 3, addr => 26 },
'virtioscsi26' => { bus => 3, addr => 27 },
'virtioscsi27' => { bus => 3, addr => 28 },
'virtioscsi28' => { bus => 3, addr => 29 },
'virtioscsi29' => { bus => 3, addr => 30 },
'virtioscsi30' => { bus => 3, addr => 31 },
'scsihw2' => { bus => 4, addr => 1 },
'scsihw3' => { bus => 4, addr => 2 },
'scsihw4' => { bus => 4, addr => 3 },
} if !defined($pci_addr_map);
return $pci_addr_map;
>
> > > Alexandre,
> > >
> > > the statement below is not true for our case. The OpenVMS guest OS is
> > > using a PCIE bus, so the virtio-scsi device should be exposed as
> > > "modern", but is not. Not sure why not at this point
> >
>
> See Fiona response,
>
> the pci express bridge is present, but the virtio-net is plugged on a
> simple pci bridge.
> ²
> pci express slots are mostly used for passthrough devices
>
>
>
> my $pcie_addr_map;
> sub get_pcie_addr_map {
> $pcie_addr_map = {
> vga => { bus => 'pcie.0', addr => 1 },
> hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 },
> hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 },
> hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 },
> hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 },
> hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 },
> hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 },
> hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 },
> hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 },
> hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 },
> hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 },
> hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 },
> hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 },
> hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 },
> hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 },
> hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 },
> hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 },
> # win7 is picky about pcie assignments
> hostpci0bus0 => { bus => "pcie.0", addr => 16 },
> hostpci1bus0 => { bus => "pcie.0", addr => 17 },
> hostpci2bus0 => { bus => "pcie.0", addr => 18 },
> hostpci3bus0 => { bus => "pcie.0", addr => 19 },
> ivshmem => { bus => 'pcie.0', addr => 20 },
> hostpci4bus0 => { bus => "pcie.0", addr => 9 },
> hostpci5bus0 => { bus => "pcie.0", addr => 10 },
> hostpci6bus0 => { bus => "pcie.0", addr => 11 },
> hostpci7bus0 => { bus => "pcie.0", addr => 12 },
> hostpci8bus0 => { bus => "pcie.0", addr => 13 },
> hostpci9bus0 => { bus => "pcie.0", addr => 14 },
> hostpci10bus0 => { bus => "pcie.0", addr => 15 },
> hostpci11bus0 => { bus => "pcie.0", addr => 21 },
> hostpci12bus0 => { bus => "pcie.0", addr => 22 },
> hostpci13bus0 => { bus => "pcie.0", addr => 23 },
> hostpci14bus0 => { bus => "pcie.0", addr => 24 },
> hostpci15bus0 => { bus => "pcie.0", addr => 25 },
> }
>
>
> Christian Moser
> Mobile: +358-40-5022105
> Email: cmos at maklee.com
> URL: www.maklee.com
> -----Original Message-----
> From: DERUMIER, Alexandre <alexandre.derumier at groupe-cyllene.com>
> Sent: Wednesday, August 14, 2024 09:45
> To: pve-devel at lists.proxmox.com; cmos at maklee.com
> Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
>
> Hi,
>
> I didn't see the responde of Fiona but indeed:
>
> https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html
>
> "virtio devices can be exposed in upto three ways
>
> - Legacy - follows virtio 0.9 specification. always uses PCI
> ID range 0x1000-0x103F
> - Transitional - follows virtio 0.9 specification by default, but
> can auto-negotiate with guest for 1.0 spce. Always
> uses PCI ID range 0x1000-0x103F
> - Modern - follows virtio 1.0 specification. always uses PCI
> ID range 0x1040-0x107F
>
> With QEMU, historically devices placed on a PCI bus will always default
> to being in transitional mode, while devices placed on a PCI-E bus will
> always dfault to being in modern mode.
> "
>
>
> -------- Message initial --------
> De: Fiona Ebner <f.ebner at proxmox.com>
> Répondre à: Proxmox VE development discussion <pve-
> devel at lists.proxmox.com>
> À: Proxmox VE development discussion <pve-devel at lists.proxmox.com>,
> Christian Moser <cmos at maklee.com>
> Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
> Date: 13/08/2024 10:55:47
>
> Hi,
>
> Am 12.08.24 um 12:40 schrieb Christian Moser:
> > Hello,
> >
> > I work for VSI (VMS Software Inc) which is porting the OpenVMS
> > operating system to x86. At this point we successfully on various
> > hypervisors, but have some issues on the KVM running on Proxmox.
> >
> > The OpenVMS VM works just fine with SATA disks and it also works with
> > for example virtio-network device etc., but trying to use virtio-scsi
> > hangs the driver. I have debugged this and I can successfully
> > configure the port/controller, send the IO request to the device. It
> > then gets processed by the device, which posts the results and sets
> > the interrupt bit in the ISR register, but it never asserts the
> > interrupt hence the driver never gets notified and the I/O hangs.
> >
> > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no
> > luck. The emulated virtio-scsi device is a legacy device. But then
> > again, the virtio-network device is also a legacy device and here we
> > are getting interrupts. One thing which bothers me is the fact that
> > the “legacy interrupt disable” bit is set in the PCI config space of
> > the virtio-scsi device (i.e. bit 10 at offset 4)
> >
> > Questions:
> > * is there a way to configure a modern virtio-scsi devcie (i.e.
> > disable_legacy=on) ?
> >
>
> I've already answered this question when you asked in a mail addressed
> directly to me:
>
> Am 12.08.24 um 11:58 schrieb Fiona Ebner:
> > Hi,
> >
> > It seems that you either need to attach the "virtio-scsi-pci" device
> > to a pcie bus or explicitly set the "disable_legacy=on" option for
> > the
> > device, neither of which Proxmox VE currently does or allows
> > configuring. The only way right now seems to be to attach the disk
> > yourself via custom arguments (the 'args' in the Proxmox VE VM
> > configuration), but then the disk will be invisible to Proxmox VE
> > operations which look at specific disk keys in the configuration!
> >
> > Feel free to open a feature request on our bug tracker to make this
> > configurable:
> > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr
> > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob
> > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7
> > 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2
> > Fbugzilla.proxmox.com%2F
> >
> > P.S. Please write to the developer list rather than individual
> > developers for such questions in the feature:
> > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr
> > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob
> > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87
> > 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2
> > Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
> >
> > Best Regards,
> > Fiona
> >
>
> > * why is the legacy interrupt bit set in the PCI config space ?
> >
>
> Most likely because the virtio-scsi-pci is configured without the
> "disable_legacy=on" option. If not explicitly set, the option will be
> "disable_legacy=auto" and when not attached to PCIe (which is the case
> for Proxmox VE currently), then legacy mode will be enabled.
>
> > * Are there any working driver for virtio-scsi on this KVM using Q35
> > machine? i.e. any other OS
> >
>
> The virtio drivers for Windows and the ones in Linux work just fine
> with our configuration.
>
>
> > Any thoughts why these interrupts are not getting delivered on the
> > PCIE bus?
> >
>
> We do not configure the virtio-scsi-pci on a PCIe bus currently, see my
> initial answer.
>
> Best Regards,
> Fiona
>
>
>
> pve-devel mailing list
> pve-devel at lists.proxmox.com
> https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK
>
> 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV
>
> xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5
>
> 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists
>
> .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
>
>
>
More information about the pve-devel
mailing list