[pve-devel] iscsi, some hang : pvestatd always do iscsi_session_rescan

[Alexandre DERUMIER]

Hi Dietmar ,

I have some progress about it, we can use sysfs ( echo "- - -" > /sys/class/scsi_host/hostX/scan) to do the same thing async and udevadm settle to check if disks changed are made.

What do you think about it ?
 





I'm also working on a add/remove iscsi disk at vm start/stop.

basicly:

activate storage : login to target, udevadm settle waiting for udev add devices, delete all devices from target (echo to sysfs),udevam settle
(no other way possible, at login, scanning of devices is hardcorded is iscsid and udev add them)

activate volume: simple echo to sysfs to add disk (on each path)

desactivate volume: simple echo to sysfs to delete disk (on each path)

deactivate storage  : logout target
those thinks works fine. 

the only things annoying is iscsi lun listing, as we need to rescan the target, so re-add all devices.

So,I was thinking of adding this in a separate new storage plugin,
maybe with simply specify lunid for volume without needing to rescan.

I also use this method with my nexenta san, works fine too.(no need 1target-1lun).




The main advatange with many luns, is that multipath, it's a lot faster as by exemple with 300luns+2 path, I got 600 devices to monitor on each host, and multipathd use a lot of cpu.
Also system can be unresponsive in case of iscsi problems with so much devices hanging.
And finaly, remove scsi luns if really a pain for the moment, as if you remove a lun in your scan, you need to remove it on each host manually on each path. With this method, device is remove at vm stop :)


What do you think about it ?

----- Mail original ----- 

De: "Dietmar Maurer" <dietmar at proxmox.com> 
À: "Alexandre DERUMIER" <aderumier at odiso.com> 
Cc: pve-devel at pve.proxmox.com 
Envoyé: Lundi 14 Mai 2012 07:00:26 
Objet: RE: [pve-devel] iscsi, some hang : pvestatd always do iscsi_session_rescan 

You want me to implement that? Or will you send a patch? 

> -----Original Message----- 
> From: Alexandre DERUMIER [mailto:aderumier at odiso.com] 
> Sent: Freitag, 27. April 2012 10:39 
> To: Dietmar Maurer 
> Cc: pve-devel at pve.proxmox.com 
> Subject: Re: [pve-devel] iscsi, some hang : pvestatd always do 
> iscsi_session_rescan 
> 
> 
> >> Well, I think the problem is that you iscsi server is slow. 
> Yes indeed, the san was overload yesterday. 
> Also I have 2 scsi controllers (active/passive), failover can take 2min. (vm can 
> handle this) 
> 
> 
> But the problem it's that proxmox become unresponsive when calling the 
> iscsiadm during this time also pvestatd hang, so rrds for vm stats are not 
> updated during this time. 
> And with ha cluster, maybe fencing can be call ? (don't have tested it) 
> 
> 
> >>Forcing a timeout is easy (we have a timeout parameter for 
> run_command()). 
> 
> oh great! 
> 
> >>The question is what timeout do you want? 
> 
> I think 5sec timeout must be enough for session rescan. 
> 
> root at kvm6:~# time /usr/bin/iscsiadm --mode session -r 1 -R 
> 
> real 0m0.616s 
> user 0m0.003s 
> sys 0m0.084s 
> root at kvm6:~# time /usr/bin/iscsiadm --mode session -r 1 -R Rescanning 
> session [sid: 1, target: iqn.1986-03.com.sun:02:316dd6a9-76bc-62ea-93fa- 
> d0140e876a4b, portal: 10.6.0.18,3260] 
> 
> real 0m0.764s 
> user 0m0.006s 
> sys 0m0.114s 
> root at kvm6:~# time /usr/bin/iscsiadm --mode session -r 1 -R Rescanning 
> session [sid: 1, target: iqn.1986-03.com.sun:02:316dd6a9-76bc-62ea-93fa- 
> d0140e876a4b, portal: 10.6.0.18,3260] 
> 
> real 0m0.753s 
> user 0m0.005s 
> sys 0m0.077s 
> root at kvm6:~# time /usr/bin/iscsiadm --mode session -r 1 -R Rescanning 
> session [sid: 1, target: iqn.1986-03.com.sun:02:316dd6a9-76bc-62ea-93fa- 
> d0140e876a4b, portal: 10.6.0.18,3260] 
> 
> real 0m0.734s 
> user 0m0.005s 
> sys 0m0.094s 
> 
> 
> Thanks ! 
> Alexandre 
> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Dietmar Maurer" <dietmar at proxmox.com> 
> À: "Alexandre DERUMIER" <aderumier at odiso.com>, pve- 
> devel at pve.proxmox.com 
> Envoyé: Vendredi 27 Avril 2012 10:12:46 
> Objet: RE: [pve-devel] iscsi, some hang : pvestatd always do 
> iscsi_session_rescan 
> 
> > I check more deeply, 
> > the main problem seem to be that iscsiadm have a long timeout. 
> 
> Well, I think the problem is that you iscsi server is slow. 
> 
> > (In fact the iscsi timeout, so it can be huge in some config with 
> > iscsi failover, or if a path fail). 
> > 
> > Maybe can we implemented some kind of "kill process iscsiadm" if it 
> > take too much time ? 
> > 
> > 
> > I see that "timelimit" package exist in debian, "timelimit 
> > /usr/bin/iscsiadm/ ...." 
> > 
> > maybe this can add some protections to iscsiadm command? 
> 
> Forcing a timeout is easy (we have a timeout parameter for 
> run_command()). 
> 
> The question is what timeout do you want? 
> 
> - Dietmar 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 




-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France