[pve-devel] vm stop tasks hanging

Alexandre DERUMIER aderumier at odiso.com
Fri Apr 20 15:09:04 CEST 2012 

what do you think about this: 
only call lvm_lvs, once only, if storage is lvm.



sub deactivate_volumes {
    my ($cfg, $vollist) = @_;

    return if !($vollist && scalar(@$vollist));

>>  my $lvs=undef;
    my @errlist = ();
    foreach my $volid (@$vollist) { 
        my ($storeid, $volname) = parse_volume_id ($volid);

        my $scfg = storage_config ($cfg, $storeid);

        if ($scfg->{type} eq 'lvm') {
            my ($name) = parse_volname_lvm ($volname);
>>          $lvs = lvm_lvs () if !$lvs;

            if ($lvs->{$scfg->{vgname}}->{$name}) {
                my $path = path ($cfg, $volid);
                my $cmd = ['/sbin/lvchange', '-aln', $path];
                eval { run_command($cmd, errmsg => "can't deactivate LV '$volid'"); };
                if (my $err = $@) {
                    warn $err;
                    push @errlist, $volid;
                }
            }
        }
    }

    die "volume deativation failed: " . join(' ', @errlist)
        if scalar(@errlist);
}


----- Mail original ----- 

De: "Alexandre DERUMIER" <aderumier at odiso.com> 
À: "Dietmar Maurer" <dietmar at proxmox.com> 
Cc: pve-devel at pve.proxmox.com 
Envoyé: Vendredi 20 Avril 2012 13:08:13 
Objet: Re: [pve-devel] vm stop tasks hanging 

I had launched the command 
/sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,lv_name,lv_size,uuid,tags 

it take around 60sec to parse all my disks on a test server 

But I don't understand why it hang when called on my production server. 

Maybe I have a hanging disk lun on my production server... 

Do we really need to rescan lvm before desactivate volumes ? 

what happen if we remove: 

my $lvs = lvm_lvs () 
if ($lvs->{$scfg->{vgname}}->{$name}) { 

? 

(I don't have lvm disks to test for the moment...) 


----- Mail original ----- 

De: "Dietmar Maurer" <dietmar at proxmox.com> 
À: "Alexandre DERUMIER" <aderumier at odiso.com>, "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
Cc: pve-devel at pve.proxmox.com 
Envoyé: Vendredi 20 Avril 2012 12:28:32 
Objet: RE: [pve-devel] vm stop tasks hanging 

Maybe. I do not have such setup, so its hard to test here. 

> -----Original Message----- 
> From: pve-devel-bounces at pve.proxmox.com [mailto:pve-devel- 
> bounces at pve.proxmox.com] On Behalf Of Alexandre DERUMIER 
> Sent: Freitag, 20. April 2012 12:23 
> To: Stefan Priebe - Profihost AG 
> Cc: pve-devel at pve.proxmox.com 
> Subject: Re: [pve-devel] vm stop tasks hanging 
> 
> also, I have around 600luns, so maybe lvs try to scan each of them ? 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 12:03:43 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> I think I found the problem: 
> 
> in Qemuserver.pm : vm_stop_cleanup sub 
> 
> -> PVE::Storage::deactivate_volumes($storecfg, $vollist); 
> 
> 
> sub deactivate_volumes { 
> my ($cfg, $vollist) = @_; 
> 
> return if !($vollist && scalar(@$vollist)); 
> 
> my $lvs = lvm_lvs (); ------------------------------> this is called, but I don't use lvm 
> 
> my @errlist = (); 
> foreach my $volid (@$vollist) { 
> my ($storeid, $volname) = parse_volume_id ($volid); 
> 
> my $scfg = storage_config ($cfg, $storeid); 
> 
> if ($scfg->{type} eq 'lvm') { 
> my ($name) = parse_volname_lvm ($volname); 
> 
> if ($lvs->{$scfg->{vgname}}->{$name}) { 
> my $path = path ($cfg, $volid); 
> my $cmd = ['/sbin/lvchange', '-aln', $path]; 
> eval { run_command($cmd, errmsg => "can't deactivate LV '$volid'"); }; 
> if (my $err = $@) { 
> warn $err; 
> push @errlist, $volid; 
> } 
> } 
> } 
> } 
> 
> die "volume deativation failed: " . join(' ', @errlist) 
> 
> 
> sub lvm_lvs { 
> my ($vgname) = @_; 
> 
> my $cmd = ['/sbin/lvs', '--separator', ':', '--noheadings', '--units', 'b', 
> '--unbuffered', '--nosuffix', '--options', 
> 'vg_name,lv_name,lv_size,uuid,tags']; 
> 
> 
> 
> 
> maybe adding a 
> 
> return if $scfg->{type} ne 'lvm' 
> 
> at the begin of deactivate_volumes sub should help ? 
> 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 10:39:13 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> interesting: when I stop the process, I have this error: 
> 
> 
> command '/sbin/lvs --separator : --noheadings --units b --unbuffered -- 
> nosuffix --options vg_name,lv_name,lv_size,uuid,tags' failed: received 
> interrupt 
> root at kvm6:~# 
> 
> I'm using scsi lun directly without lvm, maybe this is the cause of the problem 
> ? 
> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 10:36:37 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> Hi, 
> console or proxmox gui, same problem. 
> 
> strace : 
> 
> Process 536632 attached - interrupt to quit 
> select(8, [6], NULL, NULL, {0, 602921}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 263 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 263 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 263 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout) 
> open("/proc/536638/stat", O_RDONLY) = 10 
> ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY 
> (Inappropriate ioctl for device) 
> lseek(10, 0, SEEK_CUR) = 0 
> fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0 
> read(10, "536638 (qm) S 536632 536638 5366"..., 4096) = 262 
> close(10) = 0 
> select(8, [6], NULL, NULL, {1, 0}^C <unfinished ...> 
> Process 536632 detached 
> 
> 
> ----- Mail original ----- 
> 
> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> À: pve-devel at pve.proxmox.com 
> Envoyé: Vendredi 20 Avril 2012 10:00:50 
> Objet: Re: [pve-devel] vm stop tasks hanging 
> 
> Am 20.04.2012 09:58, schrieb Alexandre DERUMIER: 
> > Hi Dietmar, 
> > 
> > I have problems with kvm stopping task, which are hanging. 
> > 
> > vm stop correctly (icon in tree is black), but task continue to run. (so i need 
> to stop the task manually). 
> > 
> > I can reproduce it 100%. 
> > 
> > Do you have an idea ? 
> 
> What does an strace -f -p PID of the vm stop command say? It should do 
> or wait for something. Also does it only happen via Proxmox console or 
> does it also happen when starting then command via shell? 
> 
> Greets 
> Stefan 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
> 
> 
> -- 
> 
> -- 
> 
> 
> 
> 
> Alexandre D erumier 
> Ingénieur Système 
> Fixe : 03 20 68 88 90 
> Fax : 03 20 68 90 81 
> 45 Bvd du Général Leclerc 59100 Roubaix - France 
> 12 rue Marivaux 75002 Paris - France 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 



-- 

-- 




Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France

More information about the pve-devel mailing list