[pve-devel] vm stop tasks hanging
Alexandre DERUMIER
aderumier at odiso.com
Fri Apr 20 17:24:58 CEST 2012
it's possible to filter at command line like this:
(exemple for scanning /dev/sda only)
/sbin/lvs --config 'devices { filter = [ "a|^/dev/sda$|", "r/.*/" ] }' --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,lv_name,lv_size,uuid,tags
maybe can we pass device to lvm_lvs () ?
----- Mail original -----
De: "Alexandre DERUMIER" <aderumier at odiso.com>
À: "Dietmar Maurer" <dietmar at proxmox.com>
Cc: pve-devel at pve.proxmox.com
Envoyé: Vendredi 20 Avril 2012 16:15:03
Objet: Re: [pve-devel] vm stop tasks hanging
I didn't know good this part of code,
but yes It can be better.
Also, is it possible to restrict "/sbin/lvs --separator ...." to a specific device ?
I think we could do it in /etc/lvm/lvm.conf
# By default we accept every block device:
filter = [ "a/.*/" ]
maybe can we update this file with device with lvm partitions ? (maybe this is hard to implement....)
----- Mail original -----
De: "Dietmar Maurer" <dietmar at proxmox.com>
À: "Alexandre DERUMIER" <aderumier at odiso.com>
Cc: pve-devel at pve.proxmox.com
Envoyé: Vendredi 20 Avril 2012 15:58:26
Objet: RE: [pve-devel] vm stop tasks hanging
Better, but maybe we can only call it if the device is activated?
> -----Original Message-----
> From: Alexandre DERUMIER [mailto:aderumier at odiso.com]
> Sent: Freitag, 20. April 2012 15:09
> To: Dietmar Maurer
> Cc: pve-devel at pve.proxmox.com
> Subject: Re: [pve-devel] vm stop tasks hanging
>
> what do you think about this:
> only call lvm_lvs, once only, if storage is lvm.
>
>
>
> sub deactivate_volumes {
> my ($cfg, $vollist) = @_;
>
> return if !($vollist && scalar(@$vollist));
>
> >> my $lvs=undef;
> my @errlist = ();
> foreach my $volid (@$vollist) {
> my ($storeid, $volname) = parse_volume_id ($volid);
>
> my $scfg = storage_config ($cfg, $storeid);
>
> if ($scfg->{type} eq 'lvm') {
> my ($name) = parse_volname_lvm ($volname);
> >> $lvs = lvm_lvs () if !$lvs;
>
> if ($lvs->{$scfg->{vgname}}->{$name}) {
> my $path = path ($cfg, $volid);
> my $cmd = ['/sbin/lvchange', '-aln', $path];
> eval { run_command($cmd, errmsg => "can't deactivate LV '$volid'");
> };
> if (my $err = $@) {
> warn $err;
> push @errlist, $volid;
> }
> }
> }
> }
>
> die "volume deativation failed: " . join(' ', @errlist)
> if scalar(@errlist);
> }
>
>
> ----- Mail original -----
>
> De: "Alexandre DERUMIER" <aderumier at odiso.com>
> À: "Dietmar Maurer" <dietmar at proxmox.com>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Vendredi 20 Avril 2012 13:08:13
> Objet: Re: [pve-devel] vm stop tasks hanging
>
> I had launched the command
> /sbin/lvs --separator : --noheadings --units b --unbuffered --nosuffix --
> options vg_name,lv_name,lv_size,uuid,tags
>
> it take around 60sec to parse all my disks on a test server
>
> But I don't understand why it hang when called on my production server.
>
> Maybe I have a hanging disk lun on my production server...
>
> Do we really need to rescan lvm before desactivate volumes ?
>
> what happen if we remove:
>
> my $lvs = lvm_lvs ()
> if ($lvs->{$scfg->{vgname}}->{$name}) {
>
> ?
>
> (I don't have lvm disks to test for the moment...)
>
>
> ----- Mail original -----
>
> De: "Dietmar Maurer" <dietmar at proxmox.com>
> À: "Alexandre DERUMIER" <aderumier at odiso.com>, "Stefan Priebe -
> Profihost AG" <s.priebe at profihost.ag>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Vendredi 20 Avril 2012 12:28:32
> Objet: RE: [pve-devel] vm stop tasks hanging
>
> Maybe. I do not have such setup, so its hard to test here.
>
> > -----Original Message-----
> > From: pve-devel-bounces at pve.proxmox.com [mailto:pve-devel-
> > bounces at pve.proxmox.com] On Behalf Of Alexandre DERUMIER
> > Sent: Freitag, 20. April 2012 12:23
> > To: Stefan Priebe - Profihost AG
> > Cc: pve-devel at pve.proxmox.com
> > Subject: Re: [pve-devel] vm stop tasks hanging
> >
> > also, I have around 600luns, so maybe lvs try to scan each of them ?
> >
> > ----- Mail original -----
> >
> > De: "Alexandre DERUMIER" <aderumier at odiso.com>
> > À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> > Cc: pve-devel at pve.proxmox.com
> > Envoyé: Vendredi 20 Avril 2012 12:03:43
> > Objet: Re: [pve-devel] vm stop tasks hanging
> >
> > I think I found the problem:
> >
> > in Qemuserver.pm : vm_stop_cleanup sub
> >
> > -> PVE::Storage::deactivate_volumes($storecfg, $vollist);
> >
> >
> > sub deactivate_volumes {
> > my ($cfg, $vollist) = @_;
> >
> > return if !($vollist && scalar(@$vollist));
> >
> > my $lvs = lvm_lvs (); ------------------------------> this is called,
> > but I don't use lvm
> >
> > my @errlist = ();
> > foreach my $volid (@$vollist) {
> > my ($storeid, $volname) = parse_volume_id ($volid);
> >
> > my $scfg = storage_config ($cfg, $storeid);
> >
> > if ($scfg->{type} eq 'lvm') {
> > my ($name) = parse_volname_lvm ($volname);
> >
> > if ($lvs->{$scfg->{vgname}}->{$name}) { my $path = path ($cfg,
> > $volid); my $cmd = ['/sbin/lvchange', '-aln', $path]; eval {
> > run_command($cmd, errmsg => "can't deactivate LV '$volid'"); }; if (my
> > $err = $@) { warn $err; push @errlist, $volid; } } } }
> >
> > die "volume deativation failed: " . join(' ', @errlist)
> >
> >
> > sub lvm_lvs {
> > my ($vgname) = @_;
> >
> > my $cmd = ['/sbin/lvs', '--separator', ':', '--noheadings', '--units',
> > 'b', '--unbuffered', '--nosuffix', '--options',
> > 'vg_name,lv_name,lv_size,uuid,tags'];
> >
> >
> >
> >
> > maybe adding a
> >
> > return if $scfg->{type} ne 'lvm'
> >
> > at the begin of deactivate_volumes sub should help ?
> >
> >
> > ----- Mail original -----
> >
> > De: "Alexandre DERUMIER" <aderumier at odiso.com>
> > À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> > Cc: pve-devel at pve.proxmox.com
> > Envoyé: Vendredi 20 Avril 2012 10:39:13
> > Objet: Re: [pve-devel] vm stop tasks hanging
> >
> > interesting: when I stop the process, I have this error:
> >
> >
> > command '/sbin/lvs --separator : --noheadings --units b --unbuffered
> > -- nosuffix --options vg_name,lv_name,lv_size,uuid,tags' failed:
> > received interrupt root at kvm6:~#
> >
> > I'm using scsi lun directly without lvm, maybe this is the cause of
> > the problem ?
> >
> >
> >
> > ----- Mail original -----
> >
> > De: "Alexandre DERUMIER" <aderumier at odiso.com>
> > À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> > Cc: pve-devel at pve.proxmox.com
> > Envoyé: Vendredi 20 Avril 2012 10:36:37
> > Objet: Re: [pve-devel] vm stop tasks hanging
> >
> > Hi,
> > console or proxmox gui, same problem.
> >
> > strace :
> >
> > Process 536632 attached - interrupt to quit select(8, [6], NULL, NULL,
> > {0, 602921}) = 0 (Timeout) open("/proc/536638/stat", O_RDONLY) = 10
> > ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY
> > (Inappropriate ioctl for device) lseek(10, 0, SEEK_CUR) = 0 fstat(10,
> > {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 fcntl(10, F_SETFD,
> > FD_CLOEXEC) = 0 read(10, "536638 (qm) S 536632 536638 5366"..., 4096)
> > = 263
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 263
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 263
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}) = 0 (Timeout)
> > open("/proc/536638/stat", O_RDONLY) = 10 ioctl(10,
> SNDCTL_TMR_TIMEBASE
> > or TCGETS, 0x7fff4daca4b0) = -1 ENOTTY (Inappropriate ioctl for
> > device) lseek(10, 0, SEEK_CUR) = 0 fstat(10, {st_mode=S_IFREG|0444,
> > st_size=0, ...}) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 read(10,
> > "536638 (qm) S 536632 536638 5366"..., 4096) = 262
> > close(10) = 0
> > select(8, [6], NULL, NULL, {1, 0}^C <unfinished ...> Process 536632
> > detached
> >
> >
> > ----- Mail original -----
> >
> > De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> > À: pve-devel at pve.proxmox.com
> > Envoyé: Vendredi 20 Avril 2012 10:00:50
> > Objet: Re: [pve-devel] vm stop tasks hanging
> >
> > Am 20.04.2012 09:58, schrieb Alexandre DERUMIER:
> > > Hi Dietmar,
> > >
> > > I have problems with kvm stopping task, which are hanging.
> > >
> > > vm stop correctly (icon in tree is black), but task continue to run.
> > > (so i need
> > to stop the task manually).
> > >
> > > I can reproduce it 100%.
> > >
> > > Do you have an idea ?
> >
> > What does an strace -f -p PID of the vm stop command say? It should do
> > or wait for something. Also does it only happen via Proxmox console or
> > does it also happen when starting then command via shell?
> >
> > Greets
> > Stefan
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel at pve.proxmox.com
> > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> >
> >
> >
> > --
> >
> > --
> >
> >
> >
> >
> > Alexandre D erumier
> > Ingénieur Système
> > Fixe : 03 20 68 88 90
> > Fax : 03 20 68 90 81
> > 45 Bvd du Général Leclerc 59100 Roubaix - France
> > 12 rue Marivaux 75002 Paris - France
> >
> >
> >
> >
> > --
> >
> > --
> >
> >
> >
> >
> > Alexandre D erumier
> > Ingénieur Système
> > Fixe : 03 20 68 88 90
> > Fax : 03 20 68 90 81
> > 45 Bvd du Général Leclerc 59100 Roubaix - France
> > 12 rue Marivaux 75002 Paris - France
> >
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel at pve.proxmox.com
> > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> >
> >
> >
> > --
> >
> > --
> >
> >
> >
> >
> > Alexandre D erumier
> > Ingénieur Système
> > Fixe : 03 20 68 88 90
> > Fax : 03 20 68 90 81
> > 45 Bvd du Général Leclerc 59100 Roubaix - France
> > 12 rue Marivaux 75002 Paris - France
> >
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel at pve.proxmox.com
> > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> >
> >
> >
> > --
> >
> > --
> >
> >
> >
> >
> > Alexandre D erumier
> > Ingénieur Système
> > Fixe : 03 20 68 88 90
> > Fax : 03 20 68 90 81
> > 45 Bvd du Général Leclerc 59100 Roubaix - France
> > 12 rue Marivaux 75002 Paris - France
> >
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel at pve.proxmox.com
> > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
> --
>
> --
>
>
>
>
> Alexandre D erumier
> Ingénieur Système
> Fixe : 03 20 68 88 90
> Fax : 03 20 68 90 81
> 45 Bvd du Général Leclerc 59100 Roubaix - France
> 12 rue Marivaux 75002 Paris - France
>
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
> --
>
> --
>
>
>
>
> Alexandre D erumier
> Ingénieur Système
> Fixe : 03 20 68 88 90
> Fax : 03 20 68 90 81
> 45 Bvd du Général Leclerc 59100 Roubaix - France
> 12 rue Marivaux 75002 Paris - France
>
--
--
Alexandre D erumier
Ingénieur Système
Fixe : 03 20 68 88 90
Fax : 03 20 68 90 81
45 Bvd du Général Leclerc 59100 Roubaix - France
12 rue Marivaux 75002 Paris - France
_______________________________________________
pve-devel mailing list
pve-devel at pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--
--
Alexandre D erumier
Ingénieur Système
Fixe : 03 20 68 88 90
Fax : 03 20 68 90 81
45 Bvd du Général Leclerc 59100 Roubaix - France
12 rue Marivaux 75002 Paris - France
More information about the pve-devel
mailing list