[PVE-User] pve-user Digest, Vol 60, Issue 23

Ernesto Suarez Ojeda ernesto.suarez at mtz.contraloria.cu
Fri Mar 22 22:06:43 CET 2013


When I try to migrate a container from one node to another node, I get this error:

Mar 22 17:03:05 starting migration of CT 110 to node 'delta' (192.168.104.27)
Mar 22 17:03:05 container data is on shared storage 'local'
Mar 22 17:03:05 dump 2nd level quota
Mar 22 17:03:05 initialize container on remote node 'delta'
Mar 22 17:03:05 initializing remote quota
Mar 22 17:03:05 # /usr/bin/ssh -o 'BatchMode=yes' root at 192.168.104.27 vzctl quotainit 110
Mar 22 17:03:05 vzquota : (error) quota check : stat /var/lib/vz/private/110: No such file or directory
Mar 22 17:03:05 ERROR: Failed to initialize quota: vzquota init failed [1]
Mar 22 17:03:05 start final cleanup
Mar 22 17:03:05 ERROR: migration finished with problems (duration 00:00:00)
TASK ERROR: migration problems

What should I do for migrate this contaniner?
 
           Ernesto Suárez Ojeda
           Especialista B en Ciencias Informáticas
           Contraloría Provincial Matanzas


 
 
-----Original message-----
To:pve-user at pve.proxmox.com; 
From:pve-user-request at pve.proxmox.com
Sent:Fri 22-03-2013 08:27
Subject:pve-user Digest, Vol 60, Issue 23
Send pve-user mailing list submissions to
pve-user at pve.proxmox.com

To subscribe or unsubscribe via the World Wide Web, visit
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
or, via email, send a message with subject or body 'help' to
pve-user-request at pve.proxmox.com

You can reach the person managing the list at
pve-user-owner at pve.proxmox.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of pve-user digest..."


Today's Topics:

  1. Crash after Upgrade PVE2.3 (Martin Schuchmann)
  2. Re: Error openvz with gitlab "running (failure count 20)"
     (Knaupp, Thomas)
  3. Re: Error openvz with gitlab "running (failure count 20)"
     (maykel at maykel.sytes.net)
  4. Re: Crash after Upgrade PVE2.3 / Cron backup crashes with
     2.6.32-19 but not with 2.6.32-17 (Martin Schuchmann)


----------------------------------------------------------------------

Message: 1
Date: Thu, 21 Mar 2013 12:46:56 +0100
From: Martin Schuchmann <ms at city-pc.de>
To: pve-user at pve.proxmox.com
Subject: [PVE-User] Crash after Upgrade PVE2.3
Message-ID: <514AF330.7090700 at city-pc.de>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed

Hi there,

Yesterday I did the upgrade from 2.2 up to 2.3 (pveversion see below) on 
all three nodes of our cluster (no HA).
At 23:00 the usual backup of a KVM Machine (801) started via vzdump.cron 
on Node 3 and ended with errors (see syslog below).

After this crash the VMs on Node 3 and the Webinterface had not been 
reachable anymore.

We restarted pvedaemond and pvestatd and had been able to reach the 
webinterface.

We tried to stop the vms but the processes "vzctl stop xxx" remained in 
the process list, even kill -9 did not work for removing them.
"reboot" via ssh failed also - we had to execute an "echo b > 
/proc/sysrq-trigger" to restart the host.

After reboot everthing was fine, the VMs started again.

Now we have on the two other nodes (no reboot) still an issue in syslog:

Mar 21 12:09:18 promo2 pvestatd[101835]: WARNING: command 'df -P -B 1 
/mnt/pve/p3_storage' failed: got timeout"But an

But on the bash the "df -P -B 1 /mnt/pve/p3_storage" works fine on every 
of the three hosts.


Had this heavy backup issue been reported earlier?
Any hints to prevent from that?

Regards, Martin


pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-93
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-19-pve: 2.6.32-93
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-18
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-8
ksm-control-daemon: 1.1-1


Mar 20 23:00:01 promo3 /USR/SBIN/CRON[150583]: (root) CMD (vzdump 801 
306 --quiet 1 --mode snapshot --compress lzo --storage p2_storage)
Mar 20 23:00:02 promo3 vzdump[150584]: <root at pam> starting task 
UPID:promo3:00024C3A:00785E0A:514A3162:vzdump::root at pam:
Mar 20 23:00:02 promo3 vzdump[150586]: INFO: starting new backup job: 
vzdump 801 306 --quiet 1 --mode snapshot --compress lzo --storage p2_storage
Mar 20 23:00:02 promo3 vzdump[150586]: INFO: Starting Backup of VM 306 
(openvz)
Mar 20 23:00:31 promo3 pvestatd[2328]: WARNING: unable to connect to VM 
801 socket - timeout after 31 retries
...
Mar 20 23:03:11 promo3 pvestatd[2328]: WARNING: unable to connect to VM 
801 socket - timeout after 31 retries
Mar 20 23:03:18 promo3 kernel: INFO: task kvm:2585 blocked for more than 
120 seconds.
Mar 20 23:03:18 promo3 kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 23:03:18 promo3 kernel: kvm           D ffff88107a480da0 0  
2585      1    0 0x00000000
Mar 20 23:03:18 promo3 kernel: ffff88107a92fd08 0000000000000082 
0000000000000000 ffff880879df35c8
Mar 20 23:03:18 promo3 kernel: ffff880878cc08c0 00000000000000db 
ffff88107c415810 ffff88107a92fab8
Mar 20 23:03:18 promo3 kernel: ffff88107c415800 0000000104af1976 
ffff88107a481368 000000000001e9c0
Mar 20 23:03:18 promo3 kernel: Call Trace:
Mar 20 23:03:18 promo3 kernel: [<ffffffff8119ad69>] 
__sb_start_write+0x169/0x1a0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81097200>] ? 
autoremove_wake_function+0x0/0x40
Mar 20 23:03:18 promo3 kernel: [<ffffffff81127489>] 
generic_file_aio_write+0x69/0x100
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e325b>] 
aio_rw_vect_retry+0xbb/0x220
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e4bc4>] aio_run_iocb+0x64/0x170
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e614c>] do_io_submit+0x2bc/0x670
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e6510>] sys_io_submit+0x10/0x20
Mar 20 23:03:18 promo3 kernel: [<ffffffff8100b102>] 
system_call_fastpath+0x16/0x1b
Mar 20 23:03:18 promo3 kernel: INFO: task lvcreate:150596 blocked for 
more than 120 seconds.
Mar 20 23:03:18 promo3 kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 23:03:18 promo3 kernel: lvcreate      D ffff88087aae6d20 0 150596 
150595    0 0x00000000
Mar 20 23:03:18 promo3 kernel: ffff8802fc5bbc48 0000000000000082 
0000000000000000 00000000000000d2
Mar 20 23:03:18 promo3 kernel: ffffe8ffffffffff ffff88087bec5760 
ffffffff81ac37d0 ffffffff8141c110
Mar 20 23:03:18 promo3 kernel: 0000000000000000 0000000104af1b10 
ffff88087aae72e8 000000000001e9c0
Mar 20 23:03:18 promo3 kernel: Call Trace:
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141c110>] ? copy_params+0x90/0x110
Mar 20 23:03:18 promo3 kernel: [<ffffffff8119ab6d>] sb_wait_write+0x9d/0xb0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81097200>] ? 
autoremove_wake_function+0x0/0x40
Mar 20 23:03:18 promo3 kernel: [<ffffffff8119c2d0>] freeze_super+0x60/0x140
Mar 20 23:03:18 promo3 kernel: [<ffffffff811d5ad8>] freeze_bdev+0x98/0xe0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81415697>] dm_suspend+0x97/0x270
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141a1dc>] ? 
__find_device_hash_cell+0xac/0x170
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141b4a6>] dev_suspend+0x76/0x250
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141c344>] ctl_ioctl+0x1b4/0x270
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141b430>] ? dev_suspend+0x0/0x250
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141c413>] dm_ctl_ioctl+0x13/0x20
Mar 20 23:03:18 promo3 kernel: [<ffffffff811ac622>] vfs_ioctl+0x22/0xa0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81061bcf>] ? 
pick_next_task_fair+0x16f/0x1f0
Mar 20 23:03:18 promo3 kernel: [<ffffffff8109e52d>] ? 
sched_clock_cpu+0xcd/0x110
Mar 20 23:03:18 promo3 kernel: [<ffffffff811ac7ca>] do_vfs_ioctl+0x8a/0x590
Mar 20 23:03:18 promo3 kernel: [<ffffffff8151dc50>] ? 
thread_return+0xbe/0x88e
Mar 20 23:03:18 promo3 kernel: [<ffffffff8108e675>] ? set_one_prio+0x75/0xd0
Mar 20 23:03:18 promo3 kernel: [<ffffffff811acd1f>] sys_ioctl+0x4f/0x80
Mar 20 23:03:18 promo3 kernel: [<ffffffff8100b102>] 
system_call_fastpath+0x16/0x1b
Mar 20 23:03:21 promo3 pvestatd[2328]: WARNING: unable to connect to VM 
801 socket - timeout after 31 retries
...





------------------------------

Message: 2
Date: Thu, 21 Mar 2013 13:39:45 +0000
From: "Knaupp, Thomas" <Thomas.Knaupp at schwarz.de>
To: "'maykel at maykel.sytes.net'" <maykel at maykel.sytes.net>,
"pve-user at pve.proxmox.com" <pve-user at pve.proxmox.com>
Subject: Re: [PVE-User] Error openvz with gitlab "running (failure
count 20)"
Message-ID:
<C3289A7ABB6A5342945D393E64AADEA60AB3E57A at SCSMSX5.ADSCS.LAN>
Content-Type: text/plain; charset="utf-8"

Hello,

> I have the problem, I backup the container openvz en other machine
> and I restore  the backup in other machine best performance.
> All ok, but I start the openvz the received this error:
> running (failure count 20)

If you run   cat /proc/user_beancounters   inside the openvz machine
-> do you see any failcounts?  And if yes, which ones?


Regards
Tom




________________________________

--
Schwarz Computer Systeme GmbH
Altenhofweg 2a
92318 Neumarkt
http://www.schwarz.de
___________________________________________

Geschaeftsfuehrer: Manfred Schwarz
Sitz der Gesellschaft: Neumarkt i.d.Opf.
Registergericht: AG Nuernberg, HRB 11908
___________________________________________

Diese eMail enthaelt moeglicherweise vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese eMail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.

This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorized copying, disclosure or distribution of the material in this email is strictly forbidden.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5832 bytes
Desc: not available
URL: <http://pve.proxmox.com/pipermail/pve-user/attachments/20130321/1edd108c/attachment-0001.bin>

------------------------------

Message: 3
Date: Thu, 21 Mar 2013 16:47:51 +0100
From: maykel at maykel.sytes.net
To: <pve-user at pve.proxmox.com>
Subject: Re: [PVE-User] Error openvz with gitlab "running (failure
count 20)"
Message-ID: <20dcfe945f39550b649a56b439d82711 at maykel.sytes.net>
Content-Type: text/plain; charset=UTF-8; format=flowed

El 2013-03-21 14:39, Knaupp, Thomas escribi?:
> Hello,
>
>> I have the problem, I backup the container openvz en other machine
>> and I restore  the backup in other machine best performance.
>> All ok, but I start the openvz the received this error:
>> running (failure count 20)
>
> If you run   cat /proc/user_beancounters   inside the openvz machine
> -> do you see any failcounts?  And if yes, which ones?
>
>
> Regards
> Tom
>
>
>
>
> ________________________________
>
> --
> Schwarz Computer Systeme GmbH
> Altenhofweg 2a
> 92318 Neumarkt
> http://www.schwarz.de
> ___________________________________________
>
> Geschaeftsfuehrer: Manfred Schwarz
> Sitz der Gesellschaft: Neumarkt i.d.Opf.
> Registergericht: AG Nuernberg, HRB 11908
> ___________________________________________
>
> Diese eMail enthaelt moeglicherweise vertrauliche und/oder rechtlich
> geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind
> oder diese eMail irrtuemlich erhalten haben, informieren Sie bitte
> sofort den Absender und vernichten Sie diese eMail. Das unerlaubte
> Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht
> gestattet.
>
> This email may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this email in
> error) please notify the sender immediately and destroy this email.
> Any unauthorized copying, disclosure or distribution of the material
> in this email is strictly forbidden.

Thanks for your response. The output of command is 7 failcount in 
physpages:

physpages                  171989               262144                  
 0               262144                    7


I have give more resources???

Thanks in advanced.


------------------------------

Message: 4
Date: Fri, 22 Mar 2013 02:31:01 +0100
From: Martin Schuchmann <ms at city-pc.de>
To: pve-user at pve.proxmox.com
Subject: Re: [PVE-User] Crash after Upgrade PVE2.3 / Cron backup
crashes with 2.6.32-19 but not with 2.6.32-17
Message-ID: <514BB455.6050301 at city-pc.de>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed

Hi,

Today I had the same behaviour as yesterday - at 23:00h cron started the 
backup job and immediately the whole node was out of order: Via 
webinterface I could see the running VMs but they had not been reachable 
via RDP/SSH anymore. Also VZCTL ENTER did not work.

Again there had been kernel errors in syslog.

After a hart-reset via we recognized a logical volume created at the 
same time as the crashed:

lvdisplay

--- Logical volume ---
  LV Path                /dev/promo3/vzsnap-promo3-0
  LV Name                vzsnap-promo3-0
  VG Name                promo3
  LV UUID                DswPod-t1lR-yKen-vwDH-sG5D-Djpl-wo9iSX
  LV Write Access        read/write
  LV Creation host, time promo3, 2013-03-21 23:00:02 +0100
  LV Status              available
  # open                 0
  LV Size                4,00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3


After deleting via lvremove I started a manual backup (snapshot) for CTs 
and VMs - no problem occured.

Now I created again a new cron backup via webinterface of the same 
machine which had been successfully updated.

A few seconds after start the following errors occured:

Mar 22 01:00:01 promo3 vzdump[12622]: <root at pam> starting task 
UPID:promo3:00003150:0004D259:514B9F01:vzdump::root at pam:
Mar 22 01:00:01 promo3 vzdump[12624]: INFO: starting new backup job: 
vzdump 306 --quiet 1 --mailto ms at city-pc.de --mode snapshot --compress 
lzo --storage p2_storage
Mar 22 01:00:01 promo3 vzdump[12624]: INFO: Starting Backup of VM 306 
(openvz)
Mar 22 01:00:02 promo3 pmxcfs[4048]: [status] notice: received log
Mar 22 01:00:02 promo3 kernel: EXT3-fs: barriers disabled
Mar 22 01:00:02 promo3 kernel: kjournald starting.  Commit interval 5 
seconds
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): using internal journal
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148485
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148484
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148481
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148429
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148427
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70156303
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): 6 orphan inodes deleted
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): recovery complete
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): mounted filesystem with 
ordered data mode
Mar 22 01:00:02 promo3 pmxcfs[4048]: [status] notice: received log
Mar 22 01:01:06 promo3 pvestatd[4511]: WARNING: command 'df -P -B 1 
/mnt/pve/p3_storage' failed: got timeout
Mar 22 01:01:36 promo3 pvestatd[4511]: WARNING: command 'df -P -B 1 
/mnt/pve/p3_storage' failed: got timeout
Mar 22 01:02:58 promo3 kernel: device-mapper: snapshots: Invalidating 
snapshot: Unable to allocate exception.
Mar 22 01:03:05 promo3 kernel: Aborting journal on device dm-3.
Mar 22 01:03:05 promo3 kernel: Buffer I/O error on device dm-3, logical 
block 342819330
Mar 22 01:03:05 promo3 kernel: lost page write due to I/O error on dm-3
Mar 22 01:03:05 promo3 kernel: JBD: I/O error detected when updating 
journal superblock for dm-3.
Mar 22 01:03:05 promo3 kernel: EXT3-fs (dm-3): error: 
ext3_journal_start_sb: Detected aborted journal
Mar 22 01:03:05 promo3 kernel: EXT3-fs (dm-3): error: remounting 
filesystem read-only
Mar 22 01:03:09 promo3 kernel: EXT3-fs (dm-3): error: ext3_put_super: 
Couldn't clean up the journal
Mar 22 01:03:10 promo3 vzdump[12624]: ERROR: Backup of VM 306 failed - 
command '(cd /mnt/vzsnap0/private/306;find . '(' -regex '^\.$' ')' -o 
'(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals 
--sparse --numeric-owner --no-recursion --one-file-system --null -T 
-|lzop) 
>/mnt/pve/p2_storage/dump/vzdump-openvz-306-2013_03_22-01_00_01.tar.dat' failed: exit code 2
Mar 22 01:03:10 promo3 vzdump[12624]: INFO: Backup job finished with errors
Mar 22 01:03:12 promo3 citadel: 1 unique messages to be merged
Mar 22 01:03:12 promo3 citadel: 1 unique messages to be merged
Mar 22 01:03:12 promo3 vzdump[12624]: job errors
Mar 22 01:03:12 promo3 vzdump[12622]: <root at pam> end task 
UPID:promo3:00003150:0004D259:514B9F01:vzdump::root at pam: job errors
Mar 22 01:03:12 promo3 /USR/SBIN/CRON[12617]: (CRON) error (grandchild 
#12620 failed with exit status 255)

This time the backup did not crash the whole node, but it failed.
Also the lvdisplay did show the lv again during the failed backup:

--- Logical volume ---
  LV Path                /dev/promo3/vzsnap-promo3-0
  LV Name                vzsnap-promo3-0
  VG Name                promo3
  LV UUID                UkxdQW-GGM7-raEO-MSxS-k9jZ-s0D1-g2ZO9M
  LV Write Access        read/write
  LV Creation host, time promo3, 2013-03-22 01:00:01 +0100
  LV snapshot status     active destination for data
  LV Status              available
  # open                 1
  LV Size                2,55 TiB
  Current LE             669651
  COW-table size         4,00 GiB
  COW-table LE           1024
  Allocated to snapshot  60,92%
  Snapshot chunk size    4,00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3


I added a new backup cron a few minutes later. It started, and I tried 
to have a look on the lv's - but lvdisplay did not answer.
I started a new SSH console and tried lvscan - and it also did not 
answer during the backup, CTRL-C endet with:

promo3:~# lvscan
^C  CTRL-c detected: giving up waiting for lock
  /run/lock/lvm/V_promo3: flock failed: Unterbrechung w?hrend des 
Betriebssystemaufrufs
  Can't get lock for promo3
  Skipping volume group promo3

Syslog this time.

Mar 22 01:08:01 promo3 /USR/SBIN/CRON[13592]: (root) CMD (vzdump 306 
--quiet 1 --mode snapshot --mailto ms at city-pc.de --compress lzo 
--storage p2_storage)
Mar 22 01:08:02 promo3 vzdump[13593]: <root at pam> starting task 
UPID:promo3:0000351B:00058E23:514BA0E2:vzdump::root at pam:
Mar 22 01:08:02 promo3 vzdump[13595]: INFO: starting new backup job: 
vzdump 306 --quiet 1 --mailto ms at city-pc.de --mode snapshot --compress 
lzo --storage p2_storage
Mar 22 01:08:02 promo3 vzdump[13595]: INFO: Starting Backup of VM 306 
(openvz)
Mar 22 01:09:54 promo3 rrdcached[4027]: flushing old values
Mar 22 01:09:54 promo3 rrdcached[4027]: rotating journals
Mar 22 01:09:54 promo3 rrdcached[4027]: started new journal 
/var/lib/rrdcached/journal//rrd.journal.1363910994.615643
Mar 22 01:11:07 promo3 kernel: ct0 nfs: server 10.1.0.3 not responding, 
still trying
Mar 22 01:11:22 promo3 kernel: INFO: task nfsd:3957 blocked for more 
than 120 seconds.
Mar 22 01:11:22 promo3 kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 01:11:22 promo3 kernel: nfsd          D ffff880879f2d1e0 0  
3957      2    0 0x00000000
Mar 22 01:11:22 promo3 kernel: ffff880879f2f900 0000000000000046 
ffff8808619a4fc0 0000000000000001
Mar 22 01:11:22 promo3 kernel: 00000000000005a8 ffff88087bb77e00 
0000000000000080 0000000000000004
Mar 22 01:11:22 promo3 kernel: ffff880879f2f8d0 ffffffff81182e1b 
ffff880879f2d7a8 000000000001e9c0
Mar 22 01:11:22 promo3 kernel: Call Trace:
Mar 22 01:11:22 promo3 kernel: [<ffffffff81182e1b>] ? 
cache_flusharray+0xab/0x100
Mar 22 01:11:22 promo3 kernel: [<ffffffff810974ee>] ? 
prepare_to_wait+0x4e/0x80
Mar 22 01:11:22 promo3 kernel: [<ffffffff8119ad69>] 
__sb_start_write+0x169/0x1a0
Mar 22 01:11:22 promo3 kernel: [<ffffffff81097200>] ? 
autoremove_wake_function+0x0/0x40
Mar 22 01:11:22 promo3 kernel: [<ffffffff81127489>] 
generic_file_aio_write+0x69/0x100
Mar 22 01:11:22 promo3 kernel: [<ffffffff81127420>] ? 
generic_file_aio_write+0x0/0x100
Mar 22 01:11:22 promo3 kernel: [<ffffffff8119872b>] 
do_sync_readv_writev+0xfb/0x140
Mar 22 01:11:22 promo3 kernel: [<ffffffff811b3e40>] ? iput+0x30/0x70
Mar 22 01:11:22 promo3 kernel: [<ffffffff81097200>] ? 
autoremove_wake_function+0x0/0x40
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0361e70>] ? 
nfsd_acceptable+0x0/0x120 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffff81198557>] ? 
rw_copy_check_uvector+0x97/0x120
Mar 22 01:11:22 promo3 kernel: [<ffffffff81199696>] 
do_readv_writev+0xd6/0x1f0
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0361ff2>] ? 
nfsd_setuser_and_check_port+0x62/0xb0 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0571e99>] ? 
vzquota_qlnk_destroy+0x29/0x110 [vzdquota]
Mar 22 01:11:22 promo3 kernel: [<ffffffff811997f8>] vfs_writev+0x48/0x60
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0363a25>] 
nfsd_vfs_write+0x115/0x480 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0365cbb>] ? 
nfsd_open+0x23b/0x2c0 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0366107>] 
nfsd_write+0xe7/0x100 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa036e1df>] 
nfsd3_proc_write+0xaf/0x140 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa035e52e>] 
nfsd_dispatch+0xfe/0x240 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa027e174>] 
svc_process_common+0x344/0x650 [sunrpc]
Mar 22 01:11:22 promo3 kernel: [<ffffffff8105a620>] ? 
default_wake_function+0x0/0x20
Mar 22 01:11:22 promo3 kernel: [<ffffffffa027e7b2>] 
svc_process+0x102/0x150 [sunrpc]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa035ee5d>] nfsd+0xcd/0x180 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa035ed90>] ? nfsd+0x0/0x180 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffff81096c26>] kthread+0x96/0xa0
Mar 22 01:11:22 promo3 kernel: [<ffffffff8100c1aa>] child_rip+0xa/0x20
Mar 22 01:11:22 promo3 kernel: [<ffffffff81096b90>] ? kthread+0x0/0xa0
Mar 22 01:11:22 promo3 kernel: [<ffffffff8100c1a0>] ? child_rip+0x0/0x20
Mar 22 01:11:22 promo3 kernel: INFO: task nfsd:3958 blocked for more 
than 120 seconds.
Mar 22 01:11:22 promo3 kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 01:11:22 promo3 kernel: nfsd          D ffff880879f2c700 0  
3958      2    0 0x00000000
Mar 22 01:11:22 promo3 kernel: ffff88087aa97900 0000000000000046 
ffff88087aa978a0 ffff881064c4b000
Mar 22 01:11:22 promo3 kernel: ffff8808619a4fc0 ffff88087bb77e00 
ffff88086de62bf8 0000000000000020
Mar 22 01:11:22 promo3 kernel: ffff88087aa978d0 ffffffff81182e1b 
ffff880879f2ccc8 000000000001e9c0



During the job also the listing of the mounted pve storage did not work:

ls /mnt/pve/

ended with an hung up.

All machines on the node had been inaccassibe again.

I did a reboot with kernel  2.6.32-17.

Entered a new cronjob and the backup worked as it did for 12 month 
before, here the syslog:

Mar 22 02:22:01 promo3 /USR/SBIN/CRON[3738]: (root) CMD (vzdump 306 
--quiet 1 --mode snapshot  --compress lzo --storage p2_storage)
Mar 22 02:22:01 promo3 vzdump[3739]: <root at pam> starting task 
UPID:promo3:00000E9D:00007A5A:514BB239:vzdump::root at pam:
Mar 22 02:22:01 promo3 vzdump[3741]: INFO: starting new backup job: 
vzdump 306 --quiet 1 --mailto ms at city-pc.de --mode snapshot --compress 
lzo --storage p2_storage
Mar 22 02:22:01 promo3 vzdump[3741]: INFO: Starting Backup of VM 306 
(openvz)
Mar 22 02:22:02 promo3 kernel: EXT3-fs: barriers disabled
Mar 22 02:22:02 promo3 kernel: kjournald starting.  Commit interval 5 
seconds
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): using internal journal
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148484
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148481
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148465
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148429
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70148427
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting 
unreferenced inode 70156303
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): 6 orphan inodes deleted
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): recovery complete
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): mounted filesystem with 
ordered data mode
Mar 22 02:22:30 promo3 ntpd[1884]: Listen normally on 38 veth306.0 
fe80::c04e:44ff:fe61:ecfe UDP 123
Mar 22 02:25:49 promo3 vzdump[3741]: INFO: Finished Backup of VM 306 
(00:03:48)
Mar 22 02:25:49 promo3 vzdump[3741]: INFO: Backup job finished successfully
Mar 22 02:25:49 promo3 citadel: 1 unique messages to be merged
Mar 22 02:25:49 promo3 citadel: 1 unique messages to be merged
Mar 22 02:25:49 promo3 vzdump[3739]: <root at pam> end task 
UPID:promo3:00000E9D:00007A5A:514BB239:vzdump::root at pam: OK


Any hints to pevent kernel  2.6.32-19 from that issue?

Regards, Martin




------------------------------

_______________________________________________
pve-user mailing list
pve-user at pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user


End of pve-user Digest, Vol 60, Issue 23
****************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://pve.proxmox.com/pipermail/pve-user/attachments/20130322/ad6eb634/attachment-0013.html>


More information about the pve-user mailing list