[pve-devel] [PATCH] iscsi: fix excessive connection test spam on storage monitoring
Stelios Vailakakis
stelios at libvirt.dev
Sun Jul 6 16:14:34 CEST 2025
Hi Friedrich,
Thank you for the warm welcome!
> This seems like an unrelated change that should be in its own patch,
> could you please re-send it separately?
I submitted a separate patch in the format requested at https://lists.proxmox.com/pipermail/pve-devel/2025-June/071942.html
> This sounds like #957 [1]. A patch [2] with similar objective was merged
> [3] and included in Proxmox VE 8.4. Can you check whether your patch is
> still needed on Proxmox VE 8.4?
On 8.4.1 with the sole Proxmox node that was hosting the TrueNAS instance (proxmox5), I continued to get below errors as well as the errors on the TrueNAS syslog after doing a full pveupgrade.
- proxmox5 does list the hostname instead of the IP address under iscsiadm -m session, while 1-4 list the IP address. Forcing this behavior on proxmox1 did not cause errors to appear (reverted back to IP Address after testing).
- By utilizing my patch above that includes only hostname support, the error continued on proxmox5
- By reverting back to my patched version which included both fixes (excessive connection attempts + hostname support) the errors went away from automated checking
- pvesm status no longer reports the error code 15 when run manually
- This patch is working from an older version of ISCSIPlugin.pm and I haven't checked to see why it's only an issue on this proxmox5 node.
Jul 06 08:30:55 proxmox5 pvestatd[490256]: command '/usr/bin/iscsiadm --mode node --targetname iqn.aaaa-bb.com.redacted.ssd-vm-stor --login' failed: exit code 15
Jul 06 08:30:55 proxmox5 pvestatd[490256]: command '/usr/bin/iscsiadm --mode node --targetname iqn.xxxx-yy.com.redacted.vm-stor --login' failed: exit code 15
root at proxmox1:~# pdsh -g pve pveversion 2>/dev/null | dshbak -c
----------------
proxmox[1-2,4-5]
----------------
pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-11-pve)
root at proxmox1:~# pdsh -g pve 'pvesm status | grep vm-stor' 2>/dev/null | dshbak -c
----------------
proxmox[1-2,4-5]
----------------
lvm-nas-ssd-vm-stor lvm active 6442442752 723525632 5718917120 11.23%
lvm-nas-vm-stor lvm active 2684346368 1006669824 1677676544 37.50%
nas-ssd-vm-stor iscsi active 0 0 0 0.00%
nas-vm-stor iscsi active 0 0 0 0.00%
root at proxmox1:~# pdsh -g pve 'ls -l /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm' 2>/dev/null | dshbak -c
----------------
proxmox[1-2,4]
----------------
-rw-r--r-- 1 root root 16849 Apr 7 15:18 /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm
----------------
proxmox5
----------------
-rw-r--r-- 1 root root 19767 Jul 6 08:33 /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm
Regards,
Stelios Vailakakis
________________________________
From: Friedrich Weber <f.weber at proxmox.com>
Sent: Monday, June 23, 2025 3:24 AM
To: Proxmox VE development discussion <pve-devel at lists.proxmox.com>; Stelios Vailakakis <stelios at libvirt.dev>
Subject: Re: [pve-devel] [PATCH] iscsi: fix excessive connection test spam on storage monitoring
Hi,
Thanks for getting involved!
On 20/06/2025 02:44, Stelios Vailakakis wrote:
> Hi all,
>
> This patch addresses excessive "connection lost" and "connection reset" log
> spam on iSCSI targets caused by Proxmox storage monitoring performing TCP
> connection tests every 10 seconds, even when iSCSI sessions are active.
>
> The issue appears as continuous log entries on iSCSI targets:
> "ctld[PID]: 192.168.1.XX: read: connection lost" every 10 seconds on each node.
>
> Others have experienced this, as seen in forum posts:
> iSCSI Reconnecting every 10 seconds to FreeNAS solution | Proxmox Support Forum<https://forum.proxmox.com/threads/iscsi-reconnecting-every-10-seconds-to-freenas-solution.21205/>
> freenas ctld: read: connection lost | TrueNAS Community<https://www.truenas.com/community/threads/freenas-ctld-read-connection-lost.45513/>
This sounds like #957 [1]. A patch [2] with similar objective was merged
[3] and included in Proxmox VE 8.4. Can you check whether your patch is
still needed on Proxmox VE 8.4?
> Changes:
> - Support hostnames in portal addresses (not just IP addresses) by updating
> ISCSI_TARGET_RE and session parsing regex
This seems like an unrelated change that should be in its own patch,
could you please re-send it separately? If you do, please review our
developer docs [4], especially with regards to the CLA requirement and
instructions how to use git format-patch/send-mail to submit patches
(AFAICT it wasn't used here).
> - Skip TCP connection tests in check_connection when active sessions exist
> - Skip storage activation when sessions are already established
>
> This reduces iSCSI target log spam from continuous entries to only
> legitimate connection issues, while maintaining monitoring functionality.
Best wishes,
Friedrich
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=957
[2]
https://lore.proxmox.com/pve-devel/mailman.14.1742481496.359.pve-devel@lists.proxmox.com/
[3]
https://git.proxmox.com/?p=pve-storage.git;a=commit;h=32e482a6170b3a882dc332263281b7f449066a8d
[4] https://pve.proxmox.com/wiki/Developer_Documentation#Preparing_Patches
>
> Tested with:
> - IPv4 and IPv6 portals
> - Hostname based portals (like nas.example.com)
> - Multiple Proxmox nodes in cluster 8.2.4 + 8.3.0 (Requires latest ISCSIPlugin.pm. taken from raw.githubusercontent.com/proxmox/pve-storage/823707a7ac99eafe345d9131d10837ebefbb9078/src/PVE/Storage/ISCSIPlugin.pm<https://raw.githubusercontent.com/proxmox/pve-storage/823707a7ac99eafe345d9131d10837ebefbb9078/src/PVE/Storage/ISCSIPlugin.pm>)
> - TrueNAS and Linux iSCSI targets
> - Can apply from /usr/share/perl5/PVE/Storage# patch -p0 < /path/to/iscsifix.patch
>
> ---
> PVE/Storage/ISCSIPlugin.pm | 13 +++++++++++--
> 1 file changed, 11 insertions(+), 2 deletions(-)
>
> --- ISCSIPlugin.pm.new-upstream 2025-06-19 16:24:09.220675514 -0500
> +++ ISCSIPlugin.pm 2025-06-19 16:24:42.265694074 -0500
> @@ -33,7 +33,7 @@
> }
>
> # Example: 192.168.122.252:3260,1 iqn.2003-01.org.linux-iscsi.proxmox-nfs.x8664:sn.00567885ba8f
> -my $ISCSI_TARGET_RE = qr/^((?:$IPV4RE|\[$IPV6RE\]):\d+)\,\S+\s+(\S+)\s*$/;
> +my $ISCSI_TARGET_RE = qr/^(\S+:\d+)\,\S+\s+(\S+)\s*$/;
>
> sub iscsi_session_list {
> assert_iscsi_support();
> @@ -49,7 +49,7 @@
> my $line = shift;
> # example: tcp: [1] 192.168.122.252:3260,1 iqn.2003-01.org.linux-iscsi.proxmox-nfs.x8664:sn.00567885ba8f (non-flash)
> if ($line =~
> - m/^tcp:\s+\[(\S+)\]\s+((?:$IPV4RE|\[$IPV6RE\]):\d+)\,\S+\s+(\S+)\s+\S+?\s*$/
> + m/^tcp:\s+\[(\S+)\]\s+(\S+:\d+)\,\S+\s+(\S+)\s+\S+?\s*$/
> ) {
> my ($session_id, $portal, $target) = ($1, $2, $3);
> # there can be several sessions per target (multipath)
> @@ -480,6 +480,10 @@
> sub activate_storage {
> my ($class, $storeid, $scfg, $cache) = @_;
>
> + # Skip if session already exists
> + my $existing_sessions = iscsi_session($cache, $scfg->{target});
> + return 1 if defined($existing_sessions) && scalar(@$existing_sessions) > 0;
> +
> return if !assert_iscsi_support(1);
>
> my $sessions = iscsi_session($cache, $scfg->{target});
> @@ -618,6 +622,11 @@
> sub check_connection {
> my ($class, $storeid, $scfg) = @_;
> my $cache = {};
> + # Check for active session first
> + my $sessions = iscsi_session_list();
> + return 1 if defined($sessions->{$scfg->{target}});
> +
> + # Only check portals if no active session
> my $portals = iscsi_portals($scfg->{target}, $scfg->{portal});
>
> for my $portal (@$portals) {
>
> ---
>
>
> Regards,
> Stelios Vailakakis
> _______________________________________________
> pve-devel mailing list
> pve-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
More information about the pve-devel
mailing list