[pve-devel] FW: FW: issues with Virtio-SCSI devicde on Proxmox...
Christian Moser
cmos at maklee.com
Thu Aug 15 07:35:30 CEST 2024
I finally got it to work. The virtio-scsi device has 2 interrupt models, either the legacy "wire" interrupt or MSIX interrupts. During boot, the OpenVMS bus support code is probing the bus and configures the devices. It will default to the traditional IOAPIC interrupt and if the capability bit is set in the PCI config status register it will parse the device capabilities. Virtio-SCSI reports that it has MSIX capability, so now the bus support code is setting the interrupt mechanism to MSIX. But our OpenVMS driver for the virtio-scsi device does not yet support MSIX, hence the disconnect and hang, because we do not get any interrupts delivered.
I can special case the virtio-scsi device in the PCI config routine to ignore the MSIX capability and then everything is working fine. I checked on other hypervisors, like for example Virtualbox and here the virtio-scsi device is reporting capability as "vendor specific" which are ignored by our bus support code.
I guess there is no way of disabling the MSIX capability on Proxmox with some flags forvirtio-scsi? In the long run, we should modify our driver to support MSIX interrupts. But I had issues enabling them correctly on virtio-scsi, I need to find some driver for Linux or Windows which supports MSIX and verify what is incorrect on our side.
thanks for your help and information
/cmos
_______________________________________________________
Christian Moser
Mobile: +358-40-5022105
Email: cmos at maklee.com
URL: www.maklee.com
-----Original Message-----
From: DERUMIER, Alexandre <alexandre.derumier at groupe-cyllene.com>
Sent: Wednesday, August 14, 2024 16:06
To: cmos at maklee.com
Cc: pve-devel at lists.proxmox.com
Subject: Re: FW: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
>>I'm talking about virtio-scsi. Our virtio-network device is working
>>fine
Yes, sorry, I wanted to said virtio-scsi.
All pci devices excluding passthrough devices (with pcie=on flag) are actually plugged to pci bridge
sub get_pci_addr_map {
$pci_addr_map = {
piix3 => { bus => 0, addr => 1, conflict_ok => qw(ehci) },
ehci => { bus => 0, addr => 1, conflict_ok => qw(piix3) }, # instead of piix3 on arm
vga => { bus => 0, addr => 2, conflict_ok => qw(legacy-igd) },
'legacy-igd' => { bus => 0, addr => 2, conflict_ok => qw(vga) }, # legacy-igd requires vga=none
balloon0 => { bus => 0, addr => 3 },
watchdog => { bus => 0, addr => 4 },
scsihw0 => { bus => 0, addr => 5, conflict_ok => qw(pci.3) },
'pci.3' => { bus => 0, addr => 5, conflict_ok => qw(scsihw0) }, # also used for virtio-scsi-single bridge
scsihw1 => { bus => 0, addr => 6 },
ahci0 => { bus => 0, addr => 7 },
qga0 => { bus => 0, addr => 8 },
spice => { bus => 0, addr => 9 },
virtio0 => { bus => 0, addr => 10 },
virtio1 => { bus => 0, addr => 11 },
virtio2 => { bus => 0, addr => 12 },
virtio3 => { bus => 0, addr => 13 },
virtio4 => { bus => 0, addr => 14 },
virtio5 => { bus => 0, addr => 15 },
hostpci0 => { bus => 0, addr => 16 },
hostpci1 => { bus => 0, addr => 17 },
net0 => { bus => 0, addr => 18 },
net1 => { bus => 0, addr => 19 },
net2 => { bus => 0, addr => 20 },
net3 => { bus => 0, addr => 21 },
net4 => { bus => 0, addr => 22 },
net5 => { bus => 0, addr => 23 },
vga1 => { bus => 0, addr => 24 },
vga2 => { bus => 0, addr => 25 },
vga3 => { bus => 0, addr => 26 },
hostpci2 => { bus => 0, addr => 27 },
hostpci3 => { bus => 0, addr => 28 },
#addr29 : usb-host (pve-usb.cfg)
'pci.1' => { bus => 0, addr => 30 },
'pci.2' => { bus => 0, addr => 31 },
'net6' => { bus => 1, addr => 1 },
'net7' => { bus => 1, addr => 2 },
'net8' => { bus => 1, addr => 3 },
'net9' => { bus => 1, addr => 4 },
'net10' => { bus => 1, addr => 5 },
'net11' => { bus => 1, addr => 6 },
'net12' => { bus => 1, addr => 7 },
'net13' => { bus => 1, addr => 8 },
'net14' => { bus => 1, addr => 9 },
'net15' => { bus => 1, addr => 10 },
'net16' => { bus => 1, addr => 11 },
'net17' => { bus => 1, addr => 12 },
'net18' => { bus => 1, addr => 13 },
'net19' => { bus => 1, addr => 14 },
'net20' => { bus => 1, addr => 15 },
'net21' => { bus => 1, addr => 16 },
'net22' => { bus => 1, addr => 17 },
'net23' => { bus => 1, addr => 18 },
'net24' => { bus => 1, addr => 19 },
'net25' => { bus => 1, addr => 20 },
'net26' => { bus => 1, addr => 21 },
'net27' => { bus => 1, addr => 22 },
'net28' => { bus => 1, addr => 23 },
'net29' => { bus => 1, addr => 24 },
'net30' => { bus => 1, addr => 25 },
'net31' => { bus => 1, addr => 26 },
'xhci' => { bus => 1, addr => 27 },
'pci.4' => { bus => 1, addr => 28 },
'rng0' => { bus => 1, addr => 29 },
'pci.2-igd' => { bus => 1, addr => 30 }, # replaces pci.2 in case a legacy IGD device is passed through
'virtio6' => { bus => 2, addr => 1 },
'virtio7' => { bus => 2, addr => 2 },
'virtio8' => { bus => 2, addr => 3 },
'virtio9' => { bus => 2, addr => 4 },
'virtio10' => { bus => 2, addr => 5 },
'virtio11' => { bus => 2, addr => 6 },
'virtio12' => { bus => 2, addr => 7 },
'virtio13' => { bus => 2, addr => 8 },
'virtio14' => { bus => 2, addr => 9 },
'virtio15' => { bus => 2, addr => 10 },
'ivshmem' => { bus => 2, addr => 11 },
'audio0' => { bus => 2, addr => 12 },
hostpci4 => { bus => 2, addr => 13 },
hostpci5 => { bus => 2, addr => 14 },
hostpci6 => { bus => 2, addr => 15 },
hostpci7 => { bus => 2, addr => 16 },
hostpci8 => { bus => 2, addr => 17 },
hostpci9 => { bus => 2, addr => 18 },
hostpci10 => { bus => 2, addr => 19 },
hostpci11 => { bus => 2, addr => 20 },
hostpci12 => { bus => 2, addr => 21 },
hostpci13 => { bus => 2, addr => 22 },
hostpci14 => { bus => 2, addr => 23 },
hostpci15 => { bus => 2, addr => 24 },
'virtioscsi0' => { bus => 3, addr => 1 },
'virtioscsi1' => { bus => 3, addr => 2 },
'virtioscsi2' => { bus => 3, addr => 3 },
'virtioscsi3' => { bus => 3, addr => 4 },
'virtioscsi4' => { bus => 3, addr => 5 },
'virtioscsi5' => { bus => 3, addr => 6 },
'virtioscsi6' => { bus => 3, addr => 7 },
'virtioscsi7' => { bus => 3, addr => 8 },
'virtioscsi8' => { bus => 3, addr => 9 },
'virtioscsi9' => { bus => 3, addr => 10 },
'virtioscsi10' => { bus => 3, addr => 11 },
'virtioscsi11' => { bus => 3, addr => 12 },
'virtioscsi12' => { bus => 3, addr => 13 },
'virtioscsi13' => { bus => 3, addr => 14 },
'virtioscsi14' => { bus => 3, addr => 15 },
'virtioscsi15' => { bus => 3, addr => 16 },
'virtioscsi16' => { bus => 3, addr => 17 },
'virtioscsi17' => { bus => 3, addr => 18 },
'virtioscsi18' => { bus => 3, addr => 19 },
'virtioscsi19' => { bus => 3, addr => 20 },
'virtioscsi20' => { bus => 3, addr => 21 },
'virtioscsi21' => { bus => 3, addr => 22 },
'virtioscsi22' => { bus => 3, addr => 23 },
'virtioscsi23' => { bus => 3, addr => 24 },
'virtioscsi24' => { bus => 3, addr => 25 },
'virtioscsi25' => { bus => 3, addr => 26 },
'virtioscsi26' => { bus => 3, addr => 27 },
'virtioscsi27' => { bus => 3, addr => 28 },
'virtioscsi28' => { bus => 3, addr => 29 },
'virtioscsi29' => { bus => 3, addr => 30 },
'virtioscsi30' => { bus => 3, addr => 31 },
'scsihw2' => { bus => 4, addr => 1 },
'scsihw3' => { bus => 4, addr => 2 },
'scsihw4' => { bus => 4, addr => 3 },
} if !defined($pci_addr_map);
return $pci_addr_map;
>
> > > Alexandre,
> > >
> > > the statement below is not true for our case. The OpenVMS guest OS
> > > is using a PCIE bus, so the virtio-scsi device should be exposed
> > > as "modern", but is not. Not sure why not at this point
> >
>
> See Fiona response,
>
> the pci express bridge is present, but the virtio-net is plugged on a
> simple pci bridge.
> ²
> pci express slots are mostly used for passthrough devices
>
>
>
> my $pcie_addr_map;
> sub get_pcie_addr_map {
> $pcie_addr_map = {
> vga => { bus => 'pcie.0', addr => 1 },
> hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 },
> hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 },
> hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 },
> hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 },
> hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 },
> hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 },
> hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 },
> hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 },
> hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 },
> hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 },
> hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 },
> hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 },
> hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 },
> hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 },
> hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 },
> hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 },
> # win7 is picky about pcie assignments
> hostpci0bus0 => { bus => "pcie.0", addr => 16 },
> hostpci1bus0 => { bus => "pcie.0", addr => 17 },
> hostpci2bus0 => { bus => "pcie.0", addr => 18 },
> hostpci3bus0 => { bus => "pcie.0", addr => 19 },
> ivshmem => { bus => 'pcie.0', addr => 20 },
> hostpci4bus0 => { bus => "pcie.0", addr => 9 },
> hostpci5bus0 => { bus => "pcie.0", addr => 10 },
> hostpci6bus0 => { bus => "pcie.0", addr => 11 },
> hostpci7bus0 => { bus => "pcie.0", addr => 12 },
> hostpci8bus0 => { bus => "pcie.0", addr => 13 },
> hostpci9bus0 => { bus => "pcie.0", addr => 14 },
> hostpci10bus0 => { bus => "pcie.0", addr => 15 },
> hostpci11bus0 => { bus => "pcie.0", addr => 21 },
> hostpci12bus0 => { bus => "pcie.0", addr => 22 },
> hostpci13bus0 => { bus => "pcie.0", addr => 23 },
> hostpci14bus0 => { bus => "pcie.0", addr => 24 },
> hostpci15bus0 => { bus => "pcie.0", addr => 25 }, }
>
>
> Christian Moser
> Mobile: +358-40-5022105
> Email: cmos at maklee.com
> URL: www.maklee.com
> -----Original Message-----
> From: DERUMIER, Alexandre <alexandre.derumier at groupe-cyllene.com>
> Sent: Wednesday, August 14, 2024 09:45
> To: pve-devel at lists.proxmox.com; cmos at maklee.com
> Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
>
> Hi,
>
> I didn't see the responde of Fiona but indeed:
>
> https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html
>
> "virtio devices can be exposed in upto three ways
>
> - Legacy - follows virtio 0.9 specification. always uses PCI
> ID range 0x1000-0x103F
> - Transitional - follows virtio 0.9 specification by default, but
> can auto-negotiate with guest for 1.0 spce. Always
> uses PCI ID range 0x1000-0x103F
> - Modern - follows virtio 1.0 specification. always uses PCI
> ID range 0x1040-0x107F
>
> With QEMU, historically devices placed on a PCI bus will always
> default to being in transitional mode, while devices placed on a PCI-E
> bus will always dfault to being in modern mode.
> "
>
>
> -------- Message initial --------
> De: Fiona Ebner <f.ebner at proxmox.com>
> Répondre à: Proxmox VE development discussion <pve-
> devel at lists.proxmox.com>
> À: Proxmox VE development discussion <pve-devel at lists.proxmox.com>,
> Christian Moser <cmos at maklee.com>
> Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
> Date: 13/08/2024 10:55:47
>
> Hi,
>
> Am 12.08.24 um 12:40 schrieb Christian Moser:
> > Hello,
> >
> > I work for VSI (VMS Software Inc) which is porting the OpenVMS
> > operating system to x86. At this point we successfully on various
> > hypervisors, but have some issues on the KVM running on Proxmox.
> >
> > The OpenVMS VM works just fine with SATA disks and it also works
> > with for example virtio-network device etc., but trying to use
> > virtio-scsi hangs the driver. I have debugged this and I can
> > successfully configure the port/controller, send the IO request to
> > the device. It then gets processed by the device, which posts the
> > results and sets the interrupt bit in the ISR register, but it
> > never asserts the interrupt hence the driver never gets notified and the I/O hangs.
> >
> > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but
> > no luck. The emulated virtio-scsi device is a legacy device. But
> > then again, the virtio-network device is also a legacy device and
> > here we are getting interrupts. One thing which bothers me is the
> > fact that the “legacy interrupt disable” bit is set in the PCI
> > config space of the virtio-scsi device (i.e. bit 10 at offset 4)
> >
> > Questions:
> > * is there a way to configure a modern virtio-scsi devcie (i.e.
> > disable_legacy=on) ?
> >
>
> I've already answered this question when you asked in a mail addressed
> directly to me:
>
> Am 12.08.24 um 11:58 schrieb Fiona Ebner:
> > Hi,
> >
> > It seems that you either need to attach the "virtio-scsi-pci"
> > device to a pcie bus or explicitly set the "disable_legacy=on"
> > option for the device, neither of which Proxmox VE currently does
> > or allows configuring. The only way right now seems to be to attach
> > the disk yourself via custom arguments (the 'args' in the Proxmox
> > VE VM configuration), but then the disk will be invisible to
> > Proxmox VE operations which look at specific disk keys in the
> > configuration!
> >
> > Feel free to open a feature request on our bug tracker to make this
> > configurable:
> >
> > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbT
> > r
> > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> >
> > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Ro
> > b
> >
> > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada
> > 7
> >
> > 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%
> > 2
> > Fbugzilla.proxmox.com%2F
> >
> > P.S. Please write to the developer list rather than individual
> > developers for such questions in the feature:
> >
> > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbT
> > r
> > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> >
> > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Ro
> > b
> >
> > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8
> > 7
> >
> > 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%
> > 2 Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
> >
> > Best Regards,
> > Fiona
> >
>
> > * why is the legacy interrupt bit set in the PCI config space ?
> >
>
> Most likely because the virtio-scsi-pci is configured without the
> "disable_legacy=on" option. If not explicitly set, the option will be
> "disable_legacy=auto" and when not attached to PCIe (which is the case
> for Proxmox VE currently), then legacy mode will be enabled.
>
> > * Are there any working driver for virtio-scsi on this KVM using
> > Q35 machine? i.e. any other OS
> >
>
> The virtio drivers for Windows and the ones in Linux work just fine
> with our configuration.
>
>
> > Any thoughts why these interrupts are not getting delivered on the
> > PCIE bus?
> >
>
> We do not configure the virtio-scsi-pci on a PCIe bus currently, see
> my initial answer.
>
> Best Regards,
> Fiona
>
>
>
> pve-devel mailing list
> pve-devel at lists.proxmox.com
> https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTra
> K
>
> 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Robk
> V
>
> xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b
> 5
>
> 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flist
> s
>
> .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
>
>
>
More information about the pve-devel
mailing list