[pve-devel] vm stop tasks hanging

Alexandre DERUMIER aderumier at odiso.com
Fri Apr 20 13:08:13 CEST 2012 

I had launched the command 
/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,lv_name,lv_size,uuid,tags

it take around 60sec to parse all my disks on a test server

But I don't understand why it hang when called on my production server.

Maybe I have a hanging disk lun on my production server...

Do we really need to rescan lvm before desactivate volumes ?

what happen if we remove:

my $lvs = lvm_lvs ()
if ($lvs->{$scfg->{vgname}}->{$name}) { 

?

(I don't have lvm disks to test for the moment...)


----- Mail original ----- 

De: "Dietmar Maurer" <dietmar at proxmox.com> 
À: "Alexandre DERUMIER" <aderumier at odiso.com>, "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
Cc: pve-devel at pve.proxmox.com 
Envoyé: Vendredi 20 Avril 2012 12:28:32 
Objet: RE: [pve-devel] vm stop tasks hanging 

Maybe. I do not have such setup, so its hard to test here. 

> -----Original Message----- 
> From: pve-devel-bounces at pve.proxmox.com [mailto:pve-devel- 
> bounces at pve.proxmox.com] On Behalf Of Alexandre DERUMIER 
> Sent: Freitag, 20. April 2012 12:23 
> To: Stefan Priebe - Profihost AG 
> Cc: pve-devel at pve.proxmox.com 
> Subject: Re: [pve-devel] vm stop tasks hanging 
> 
> also, I have around 600luns, so maybe lvs try to scan each of them ? 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 12:03:43 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> I think I found the problem: 
> 
> in Qemuserver.pm : vm_stop_cleanup sub 
> 
> -> PVE::Storage::deactivate_volumes($storecfg, $vollist); 
> 
> 
> sub deactivate_volumes { 
> my ($cfg, $vollist) = @_; 
> 
> return if !($vollist && scalar(@$vollist)); 
> 
> my $lvs = lvm_lvs (); ------------------------------> this is called, but I don't use lvm 
> 
> my @errlist = (); 
> foreach my $volid (@$vollist) { 
> my ($storeid, $volname) = parse_volume_id ($volid); 
> 
> my $scfg = storage_config ($cfg, $storeid); 
> 
> if ($scfg->{type} eq 'lvm') { 
> my ($name) = parse_volname_lvm ($volname); 
> 
> if ($lvs->{$scfg->{vgname}}->{$name}) { 
> my $path = path ($cfg, $volid); 
> my $cmd = ['/sbin/lvchange', '-aln', $path]; 
> eval { run_command($cmd, errmsg => "can't deactivate LV '$volid'"); }; 
> if (my $err = $@) { 
> warn $err; 
> push @errlist, $volid; 
> } 
> } 
> } 
> } 
> 
> die "volume deativation failed: " . join(' ', @errlist) 
> 
> 
> sub lvm_lvs { 
> my ($vgname) = @_; 
> 
> my $cmd = ['/sbin/lvs', '--separator', ':', '--noheadings', '--units', 'b', 
> '--unbuffered', '--nosuffix', '--options', 
> 'vg_name,lv_name,lv_size,uuid,tags']; 
> 
> 
> 
> 
> maybe adding a 
> 
> return if $scfg->{type} ne 'lvm' 
> 
> at the begin of deactivate_volumes sub should help ? 
> 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 10:39:13 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> interesting: when I stop the process, I have this error: 
> 
> 
> command '/sbin/lvs --separator : --noheadings --units b --unbuffered -- 
> nosuffix --options vg_name,lv_name,lv_size,uuid,tags' failed: received 
> interrupt 
> root at kvm6:~# 
> 
> I'm using scsi lun directly without lvm, maybe this is the cause of the problem 
> ? 
> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 10:36:37 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> Hi, 
> console or proxmox gui, same problem. 
> 
> strace : 
> 
> Process 536632 attached - interrupt to quit 
> select(8, [6], NULL, NULL, {0, 602921}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 263 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 263 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 263 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}^C <unfinished ...> 
> Process 536632 detached 
> 
> 
> ----- Mail original ----- 
> 
> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> À: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 10:00:50 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> Am 20.04.2012 09:58, schrieb Alexandre DERUMIER: 
> > Hi Dietmar, 
> > 
> > I have problems with kvm stopping task, which are hanging. 
> > 
> > vm stop correctly (icon in tree is black), but task continue to run. (so i need 
> to stop the task manually). 
> > 
> > I can reproduce it 100%. 
> > 
> > Do you have an idea ? 
> 
> What does an strace -f -p PID of the vm stop command say? It should do 
> or wait for something. Also does it only happen via Proxmox console or 
> does it also happen when starting then command via shell? 
> 
> Greets 
> Stefan 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France

More information about the pve-devel mailing list