[PVE-User] pve-user Digest, Vol 60, Issue 23
Ernesto Suarez Ojeda
ernesto.suarez at mtz.contraloria.cu
Fri Mar 22 22:06:43 CET 2013
When I try to migrate a container from one node to another node, I get this error:
Mar 22 17:03:05 starting migration of CT 110 to node 'delta' (192.168.104.27)
Mar 22 17:03:05 container data is on shared storage 'local'
Mar 22 17:03:05 dump 2nd level quota
Mar 22 17:03:05 initialize container on remote node 'delta'
Mar 22 17:03:05 initializing remote quota
Mar 22 17:03:05 # /usr/bin/ssh -o 'BatchMode=yes' root at 192.168.104.27 vzctl quotainit 110
Mar 22 17:03:05 vzquota : (error) quota check : stat /var/lib/vz/private/110: No such file or directory
Mar 22 17:03:05 ERROR: Failed to initialize quota: vzquota init failed [1]
Mar 22 17:03:05 start final cleanup
Mar 22 17:03:05 ERROR: migration finished with problems (duration 00:00:00)
TASK ERROR: migration problems
What should I do for migrate this contaniner?
Ernesto Suárez Ojeda
Especialista B en Ciencias Informáticas
Contraloría Provincial Matanzas
-----Original message-----
To:pve-user at pve.proxmox.com;
From:pve-user-request at pve.proxmox.com
Sent:Fri 22-03-2013 08:27
Subject:pve-user Digest, Vol 60, Issue 23
Send pve-user mailing list submissions to
pve-user at pve.proxmox.com
To subscribe or unsubscribe via the World Wide Web, visit
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
or, via email, send a message with subject or body 'help' to
pve-user-request at pve.proxmox.com
You can reach the person managing the list at
pve-user-owner at pve.proxmox.com
When replying, please edit your Subject line so it is more specific
than "Re: Contents of pve-user digest..."
Today's Topics:
1. Crash after Upgrade PVE2.3 (Martin Schuchmann)
2. Re: Error openvz with gitlab "running (failure count 20)"
(Knaupp, Thomas)
3. Re: Error openvz with gitlab "running (failure count 20)"
(maykel at maykel.sytes.net)
4. Re: Crash after Upgrade PVE2.3 / Cron backup crashes with
2.6.32-19 but not with 2.6.32-17 (Martin Schuchmann)
----------------------------------------------------------------------
Message: 1
Date: Thu, 21 Mar 2013 12:46:56 +0100
From: Martin Schuchmann <ms at city-pc.de>
To: pve-user at pve.proxmox.com
Subject: [PVE-User] Crash after Upgrade PVE2.3
Message-ID: <514AF330.7090700 at city-pc.de>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Hi there,
Yesterday I did the upgrade from 2.2 up to 2.3 (pveversion see below) on
all three nodes of our cluster (no HA).
At 23:00 the usual backup of a KVM Machine (801) started via vzdump.cron
on Node 3 and ended with errors (see syslog below).
After this crash the VMs on Node 3 and the Webinterface had not been
reachable anymore.
We restarted pvedaemond and pvestatd and had been able to reach the
webinterface.
We tried to stop the vms but the processes "vzctl stop xxx" remained in
the process list, even kill -9 did not work for removing them.
"reboot" via ssh failed also - we had to execute an "echo b >
/proc/sysrq-trigger" to restart the host.
After reboot everthing was fine, the VMs started again.
Now we have on the two other nodes (no reboot) still an issue in syslog:
Mar 21 12:09:18 promo2 pvestatd[101835]: WARNING: command 'df -P -B 1
/mnt/pve/p3_storage' failed: got timeout"But an
But on the bash the "df -P -B 1 /mnt/pve/p3_storage" works fine on every
of the three hosts.
Had this heavy backup issue been reported earlier?
Any hints to prevent from that?
Regards, Martin
pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-93
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-19-pve: 2.6.32-93
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-18
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-8
ksm-control-daemon: 1.1-1
Mar 20 23:00:01 promo3 /USR/SBIN/CRON[150583]: (root) CMD (vzdump 801
306 --quiet 1 --mode snapshot --compress lzo --storage p2_storage)
Mar 20 23:00:02 promo3 vzdump[150584]: <root at pam> starting task
UPID:promo3:00024C3A:00785E0A:514A3162:vzdump::root at pam:
Mar 20 23:00:02 promo3 vzdump[150586]: INFO: starting new backup job:
vzdump 801 306 --quiet 1 --mode snapshot --compress lzo --storage p2_storage
Mar 20 23:00:02 promo3 vzdump[150586]: INFO: Starting Backup of VM 306
(openvz)
Mar 20 23:00:31 promo3 pvestatd[2328]: WARNING: unable to connect to VM
801 socket - timeout after 31 retries
...
Mar 20 23:03:11 promo3 pvestatd[2328]: WARNING: unable to connect to VM
801 socket - timeout after 31 retries
Mar 20 23:03:18 promo3 kernel: INFO: task kvm:2585 blocked for more than
120 seconds.
Mar 20 23:03:18 promo3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 23:03:18 promo3 kernel: kvm D ffff88107a480da0 0
2585 1 0 0x00000000
Mar 20 23:03:18 promo3 kernel: ffff88107a92fd08 0000000000000082
0000000000000000 ffff880879df35c8
Mar 20 23:03:18 promo3 kernel: ffff880878cc08c0 00000000000000db
ffff88107c415810 ffff88107a92fab8
Mar 20 23:03:18 promo3 kernel: ffff88107c415800 0000000104af1976
ffff88107a481368 000000000001e9c0
Mar 20 23:03:18 promo3 kernel: Call Trace:
Mar 20 23:03:18 promo3 kernel: [<ffffffff8119ad69>]
__sb_start_write+0x169/0x1a0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81097200>] ?
autoremove_wake_function+0x0/0x40
Mar 20 23:03:18 promo3 kernel: [<ffffffff81127489>]
generic_file_aio_write+0x69/0x100
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e325b>]
aio_rw_vect_retry+0xbb/0x220
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e4bc4>] aio_run_iocb+0x64/0x170
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e614c>] do_io_submit+0x2bc/0x670
Mar 20 23:03:18 promo3 kernel: [<ffffffff811e6510>] sys_io_submit+0x10/0x20
Mar 20 23:03:18 promo3 kernel: [<ffffffff8100b102>]
system_call_fastpath+0x16/0x1b
Mar 20 23:03:18 promo3 kernel: INFO: task lvcreate:150596 blocked for
more than 120 seconds.
Mar 20 23:03:18 promo3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 23:03:18 promo3 kernel: lvcreate D ffff88087aae6d20 0 150596
150595 0 0x00000000
Mar 20 23:03:18 promo3 kernel: ffff8802fc5bbc48 0000000000000082
0000000000000000 00000000000000d2
Mar 20 23:03:18 promo3 kernel: ffffe8ffffffffff ffff88087bec5760
ffffffff81ac37d0 ffffffff8141c110
Mar 20 23:03:18 promo3 kernel: 0000000000000000 0000000104af1b10
ffff88087aae72e8 000000000001e9c0
Mar 20 23:03:18 promo3 kernel: Call Trace:
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141c110>] ? copy_params+0x90/0x110
Mar 20 23:03:18 promo3 kernel: [<ffffffff8119ab6d>] sb_wait_write+0x9d/0xb0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81097200>] ?
autoremove_wake_function+0x0/0x40
Mar 20 23:03:18 promo3 kernel: [<ffffffff8119c2d0>] freeze_super+0x60/0x140
Mar 20 23:03:18 promo3 kernel: [<ffffffff811d5ad8>] freeze_bdev+0x98/0xe0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81415697>] dm_suspend+0x97/0x270
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141a1dc>] ?
__find_device_hash_cell+0xac/0x170
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141b4a6>] dev_suspend+0x76/0x250
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141c344>] ctl_ioctl+0x1b4/0x270
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141b430>] ? dev_suspend+0x0/0x250
Mar 20 23:03:18 promo3 kernel: [<ffffffff8141c413>] dm_ctl_ioctl+0x13/0x20
Mar 20 23:03:18 promo3 kernel: [<ffffffff811ac622>] vfs_ioctl+0x22/0xa0
Mar 20 23:03:18 promo3 kernel: [<ffffffff81061bcf>] ?
pick_next_task_fair+0x16f/0x1f0
Mar 20 23:03:18 promo3 kernel: [<ffffffff8109e52d>] ?
sched_clock_cpu+0xcd/0x110
Mar 20 23:03:18 promo3 kernel: [<ffffffff811ac7ca>] do_vfs_ioctl+0x8a/0x590
Mar 20 23:03:18 promo3 kernel: [<ffffffff8151dc50>] ?
thread_return+0xbe/0x88e
Mar 20 23:03:18 promo3 kernel: [<ffffffff8108e675>] ? set_one_prio+0x75/0xd0
Mar 20 23:03:18 promo3 kernel: [<ffffffff811acd1f>] sys_ioctl+0x4f/0x80
Mar 20 23:03:18 promo3 kernel: [<ffffffff8100b102>]
system_call_fastpath+0x16/0x1b
Mar 20 23:03:21 promo3 pvestatd[2328]: WARNING: unable to connect to VM
801 socket - timeout after 31 retries
...
------------------------------
Message: 2
Date: Thu, 21 Mar 2013 13:39:45 +0000
From: "Knaupp, Thomas" <Thomas.Knaupp at schwarz.de>
To: "'maykel at maykel.sytes.net'" <maykel at maykel.sytes.net>,
"pve-user at pve.proxmox.com" <pve-user at pve.proxmox.com>
Subject: Re: [PVE-User] Error openvz with gitlab "running (failure
count 20)"
Message-ID:
<C3289A7ABB6A5342945D393E64AADEA60AB3E57A at SCSMSX5.ADSCS.LAN>
Content-Type: text/plain; charset="utf-8"
Hello,
> I have the problem, I backup the container openvz en other machine
> and I restore the backup in other machine best performance.
> All ok, but I start the openvz the received this error:
> running (failure count 20)
If you run cat /proc/user_beancounters inside the openvz machine
-> do you see any failcounts? And if yes, which ones?
Regards
Tom
________________________________
--
Schwarz Computer Systeme GmbH
Altenhofweg 2a
92318 Neumarkt
http://www.schwarz.de
___________________________________________
Geschaeftsfuehrer: Manfred Schwarz
Sitz der Gesellschaft: Neumarkt i.d.Opf.
Registergericht: AG Nuernberg, HRB 11908
___________________________________________
Diese eMail enthaelt moeglicherweise vertrauliche und/oder rechtlich geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese eMail irrtuemlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese eMail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet.
This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorized copying, disclosure or distribution of the material in this email is strictly forbidden.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5832 bytes
Desc: not available
URL: <http://pve.proxmox.com/pipermail/pve-user/attachments/20130321/1edd108c/attachment-0001.bin>
------------------------------
Message: 3
Date: Thu, 21 Mar 2013 16:47:51 +0100
From: maykel at maykel.sytes.net
To: <pve-user at pve.proxmox.com>
Subject: Re: [PVE-User] Error openvz with gitlab "running (failure
count 20)"
Message-ID: <20dcfe945f39550b649a56b439d82711 at maykel.sytes.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
El 2013-03-21 14:39, Knaupp, Thomas escribi?:
> Hello,
>
>> I have the problem, I backup the container openvz en other machine
>> and I restore the backup in other machine best performance.
>> All ok, but I start the openvz the received this error:
>> running (failure count 20)
>
> If you run cat /proc/user_beancounters inside the openvz machine
> -> do you see any failcounts? And if yes, which ones?
>
>
> Regards
> Tom
>
>
>
>
> ________________________________
>
> --
> Schwarz Computer Systeme GmbH
> Altenhofweg 2a
> 92318 Neumarkt
> http://www.schwarz.de
> ___________________________________________
>
> Geschaeftsfuehrer: Manfred Schwarz
> Sitz der Gesellschaft: Neumarkt i.d.Opf.
> Registergericht: AG Nuernberg, HRB 11908
> ___________________________________________
>
> Diese eMail enthaelt moeglicherweise vertrauliche und/oder rechtlich
> geschuetzte Informationen. Wenn Sie nicht der richtige Adressat sind
> oder diese eMail irrtuemlich erhalten haben, informieren Sie bitte
> sofort den Absender und vernichten Sie diese eMail. Das unerlaubte
> Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht
> gestattet.
>
> This email may contain confidential and/or privileged information. If
> you are not the intended recipient (or have received this email in
> error) please notify the sender immediately and destroy this email.
> Any unauthorized copying, disclosure or distribution of the material
> in this email is strictly forbidden.
Thanks for your response. The output of command is 7 failcount in
physpages:
physpages 171989 262144
0 262144 7
I have give more resources???
Thanks in advanced.
------------------------------
Message: 4
Date: Fri, 22 Mar 2013 02:31:01 +0100
From: Martin Schuchmann <ms at city-pc.de>
To: pve-user at pve.proxmox.com
Subject: Re: [PVE-User] Crash after Upgrade PVE2.3 / Cron backup
crashes with 2.6.32-19 but not with 2.6.32-17
Message-ID: <514BB455.6050301 at city-pc.de>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Hi,
Today I had the same behaviour as yesterday - at 23:00h cron started the
backup job and immediately the whole node was out of order: Via
webinterface I could see the running VMs but they had not been reachable
via RDP/SSH anymore. Also VZCTL ENTER did not work.
Again there had been kernel errors in syslog.
After a hart-reset via we recognized a logical volume created at the
same time as the crashed:
lvdisplay
--- Logical volume ---
LV Path /dev/promo3/vzsnap-promo3-0
LV Name vzsnap-promo3-0
VG Name promo3
LV UUID DswPod-t1lR-yKen-vwDH-sG5D-Djpl-wo9iSX
LV Write Access read/write
LV Creation host, time promo3, 2013-03-21 23:00:02 +0100
LV Status available
# open 0
LV Size 4,00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3
After deleting via lvremove I started a manual backup (snapshot) for CTs
and VMs - no problem occured.
Now I created again a new cron backup via webinterface of the same
machine which had been successfully updated.
A few seconds after start the following errors occured:
Mar 22 01:00:01 promo3 vzdump[12622]: <root at pam> starting task
UPID:promo3:00003150:0004D259:514B9F01:vzdump::root at pam:
Mar 22 01:00:01 promo3 vzdump[12624]: INFO: starting new backup job:
vzdump 306 --quiet 1 --mailto ms at city-pc.de --mode snapshot --compress
lzo --storage p2_storage
Mar 22 01:00:01 promo3 vzdump[12624]: INFO: Starting Backup of VM 306
(openvz)
Mar 22 01:00:02 promo3 pmxcfs[4048]: [status] notice: received log
Mar 22 01:00:02 promo3 kernel: EXT3-fs: barriers disabled
Mar 22 01:00:02 promo3 kernel: kjournald starting. Commit interval 5
seconds
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): using internal journal
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148485
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148484
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148481
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148429
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148427
Mar 22 01:00:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70156303
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): 6 orphan inodes deleted
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): recovery complete
Mar 22 01:00:02 promo3 kernel: EXT3-fs (dm-3): mounted filesystem with
ordered data mode
Mar 22 01:00:02 promo3 pmxcfs[4048]: [status] notice: received log
Mar 22 01:01:06 promo3 pvestatd[4511]: WARNING: command 'df -P -B 1
/mnt/pve/p3_storage' failed: got timeout
Mar 22 01:01:36 promo3 pvestatd[4511]: WARNING: command 'df -P -B 1
/mnt/pve/p3_storage' failed: got timeout
Mar 22 01:02:58 promo3 kernel: device-mapper: snapshots: Invalidating
snapshot: Unable to allocate exception.
Mar 22 01:03:05 promo3 kernel: Aborting journal on device dm-3.
Mar 22 01:03:05 promo3 kernel: Buffer I/O error on device dm-3, logical
block 342819330
Mar 22 01:03:05 promo3 kernel: lost page write due to I/O error on dm-3
Mar 22 01:03:05 promo3 kernel: JBD: I/O error detected when updating
journal superblock for dm-3.
Mar 22 01:03:05 promo3 kernel: EXT3-fs (dm-3): error:
ext3_journal_start_sb: Detected aborted journal
Mar 22 01:03:05 promo3 kernel: EXT3-fs (dm-3): error: remounting
filesystem read-only
Mar 22 01:03:09 promo3 kernel: EXT3-fs (dm-3): error: ext3_put_super:
Couldn't clean up the journal
Mar 22 01:03:10 promo3 vzdump[12624]: ERROR: Backup of VM 306 failed -
command '(cd /mnt/vzsnap0/private/306;find . '(' -regex '^\.$' ')' -o
'(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals
--sparse --numeric-owner --no-recursion --one-file-system --null -T
-|lzop)
>/mnt/pve/p2_storage/dump/vzdump-openvz-306-2013_03_22-01_00_01.tar.dat' failed: exit code 2
Mar 22 01:03:10 promo3 vzdump[12624]: INFO: Backup job finished with errors
Mar 22 01:03:12 promo3 citadel: 1 unique messages to be merged
Mar 22 01:03:12 promo3 citadel: 1 unique messages to be merged
Mar 22 01:03:12 promo3 vzdump[12624]: job errors
Mar 22 01:03:12 promo3 vzdump[12622]: <root at pam> end task
UPID:promo3:00003150:0004D259:514B9F01:vzdump::root at pam: job errors
Mar 22 01:03:12 promo3 /USR/SBIN/CRON[12617]: (CRON) error (grandchild
#12620 failed with exit status 255)
This time the backup did not crash the whole node, but it failed.
Also the lvdisplay did show the lv again during the failed backup:
--- Logical volume ---
LV Path /dev/promo3/vzsnap-promo3-0
LV Name vzsnap-promo3-0
VG Name promo3
LV UUID UkxdQW-GGM7-raEO-MSxS-k9jZ-s0D1-g2ZO9M
LV Write Access read/write
LV Creation host, time promo3, 2013-03-22 01:00:01 +0100
LV snapshot status active destination for data
LV Status available
# open 1
LV Size 2,55 TiB
Current LE 669651
COW-table size 4,00 GiB
COW-table LE 1024
Allocated to snapshot 60,92%
Snapshot chunk size 4,00 KiB
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3
I added a new backup cron a few minutes later. It started, and I tried
to have a look on the lv's - but lvdisplay did not answer.
I started a new SSH console and tried lvscan - and it also did not
answer during the backup, CTRL-C endet with:
promo3:~# lvscan
^C CTRL-c detected: giving up waiting for lock
/run/lock/lvm/V_promo3: flock failed: Unterbrechung w?hrend des
Betriebssystemaufrufs
Can't get lock for promo3
Skipping volume group promo3
Syslog this time.
Mar 22 01:08:01 promo3 /USR/SBIN/CRON[13592]: (root) CMD (vzdump 306
--quiet 1 --mode snapshot --mailto ms at city-pc.de --compress lzo
--storage p2_storage)
Mar 22 01:08:02 promo3 vzdump[13593]: <root at pam> starting task
UPID:promo3:0000351B:00058E23:514BA0E2:vzdump::root at pam:
Mar 22 01:08:02 promo3 vzdump[13595]: INFO: starting new backup job:
vzdump 306 --quiet 1 --mailto ms at city-pc.de --mode snapshot --compress
lzo --storage p2_storage
Mar 22 01:08:02 promo3 vzdump[13595]: INFO: Starting Backup of VM 306
(openvz)
Mar 22 01:09:54 promo3 rrdcached[4027]: flushing old values
Mar 22 01:09:54 promo3 rrdcached[4027]: rotating journals
Mar 22 01:09:54 promo3 rrdcached[4027]: started new journal
/var/lib/rrdcached/journal//rrd.journal.1363910994.615643
Mar 22 01:11:07 promo3 kernel: ct0 nfs: server 10.1.0.3 not responding,
still trying
Mar 22 01:11:22 promo3 kernel: INFO: task nfsd:3957 blocked for more
than 120 seconds.
Mar 22 01:11:22 promo3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 01:11:22 promo3 kernel: nfsd D ffff880879f2d1e0 0
3957 2 0 0x00000000
Mar 22 01:11:22 promo3 kernel: ffff880879f2f900 0000000000000046
ffff8808619a4fc0 0000000000000001
Mar 22 01:11:22 promo3 kernel: 00000000000005a8 ffff88087bb77e00
0000000000000080 0000000000000004
Mar 22 01:11:22 promo3 kernel: ffff880879f2f8d0 ffffffff81182e1b
ffff880879f2d7a8 000000000001e9c0
Mar 22 01:11:22 promo3 kernel: Call Trace:
Mar 22 01:11:22 promo3 kernel: [<ffffffff81182e1b>] ?
cache_flusharray+0xab/0x100
Mar 22 01:11:22 promo3 kernel: [<ffffffff810974ee>] ?
prepare_to_wait+0x4e/0x80
Mar 22 01:11:22 promo3 kernel: [<ffffffff8119ad69>]
__sb_start_write+0x169/0x1a0
Mar 22 01:11:22 promo3 kernel: [<ffffffff81097200>] ?
autoremove_wake_function+0x0/0x40
Mar 22 01:11:22 promo3 kernel: [<ffffffff81127489>]
generic_file_aio_write+0x69/0x100
Mar 22 01:11:22 promo3 kernel: [<ffffffff81127420>] ?
generic_file_aio_write+0x0/0x100
Mar 22 01:11:22 promo3 kernel: [<ffffffff8119872b>]
do_sync_readv_writev+0xfb/0x140
Mar 22 01:11:22 promo3 kernel: [<ffffffff811b3e40>] ? iput+0x30/0x70
Mar 22 01:11:22 promo3 kernel: [<ffffffff81097200>] ?
autoremove_wake_function+0x0/0x40
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0361e70>] ?
nfsd_acceptable+0x0/0x120 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffff81198557>] ?
rw_copy_check_uvector+0x97/0x120
Mar 22 01:11:22 promo3 kernel: [<ffffffff81199696>]
do_readv_writev+0xd6/0x1f0
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0361ff2>] ?
nfsd_setuser_and_check_port+0x62/0xb0 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0571e99>] ?
vzquota_qlnk_destroy+0x29/0x110 [vzdquota]
Mar 22 01:11:22 promo3 kernel: [<ffffffff811997f8>] vfs_writev+0x48/0x60
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0363a25>]
nfsd_vfs_write+0x115/0x480 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0365cbb>] ?
nfsd_open+0x23b/0x2c0 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa0366107>]
nfsd_write+0xe7/0x100 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa036e1df>]
nfsd3_proc_write+0xaf/0x140 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa035e52e>]
nfsd_dispatch+0xfe/0x240 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa027e174>]
svc_process_common+0x344/0x650 [sunrpc]
Mar 22 01:11:22 promo3 kernel: [<ffffffff8105a620>] ?
default_wake_function+0x0/0x20
Mar 22 01:11:22 promo3 kernel: [<ffffffffa027e7b2>]
svc_process+0x102/0x150 [sunrpc]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa035ee5d>] nfsd+0xcd/0x180 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffffa035ed90>] ? nfsd+0x0/0x180 [nfsd]
Mar 22 01:11:22 promo3 kernel: [<ffffffff81096c26>] kthread+0x96/0xa0
Mar 22 01:11:22 promo3 kernel: [<ffffffff8100c1aa>] child_rip+0xa/0x20
Mar 22 01:11:22 promo3 kernel: [<ffffffff81096b90>] ? kthread+0x0/0xa0
Mar 22 01:11:22 promo3 kernel: [<ffffffff8100c1a0>] ? child_rip+0x0/0x20
Mar 22 01:11:22 promo3 kernel: INFO: task nfsd:3958 blocked for more
than 120 seconds.
Mar 22 01:11:22 promo3 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 22 01:11:22 promo3 kernel: nfsd D ffff880879f2c700 0
3958 2 0 0x00000000
Mar 22 01:11:22 promo3 kernel: ffff88087aa97900 0000000000000046
ffff88087aa978a0 ffff881064c4b000
Mar 22 01:11:22 promo3 kernel: ffff8808619a4fc0 ffff88087bb77e00
ffff88086de62bf8 0000000000000020
Mar 22 01:11:22 promo3 kernel: ffff88087aa978d0 ffffffff81182e1b
ffff880879f2ccc8 000000000001e9c0
During the job also the listing of the mounted pve storage did not work:
ls /mnt/pve/
ended with an hung up.
All machines on the node had been inaccassibe again.
I did a reboot with kernel 2.6.32-17.
Entered a new cronjob and the backup worked as it did for 12 month
before, here the syslog:
Mar 22 02:22:01 promo3 /USR/SBIN/CRON[3738]: (root) CMD (vzdump 306
--quiet 1 --mode snapshot --compress lzo --storage p2_storage)
Mar 22 02:22:01 promo3 vzdump[3739]: <root at pam> starting task
UPID:promo3:00000E9D:00007A5A:514BB239:vzdump::root at pam:
Mar 22 02:22:01 promo3 vzdump[3741]: INFO: starting new backup job:
vzdump 306 --quiet 1 --mailto ms at city-pc.de --mode snapshot --compress
lzo --storage p2_storage
Mar 22 02:22:01 promo3 vzdump[3741]: INFO: Starting Backup of VM 306
(openvz)
Mar 22 02:22:02 promo3 kernel: EXT3-fs: barriers disabled
Mar 22 02:22:02 promo3 kernel: kjournald starting. Commit interval 5
seconds
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): using internal journal
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148484
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148481
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148465
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148429
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70148427
Mar 22 02:22:02 promo3 kernel: ext3_orphan_cleanup: deleting
unreferenced inode 70156303
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): 6 orphan inodes deleted
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): recovery complete
Mar 22 02:22:02 promo3 kernel: EXT3-fs (dm-3): mounted filesystem with
ordered data mode
Mar 22 02:22:30 promo3 ntpd[1884]: Listen normally on 38 veth306.0
fe80::c04e:44ff:fe61:ecfe UDP 123
Mar 22 02:25:49 promo3 vzdump[3741]: INFO: Finished Backup of VM 306
(00:03:48)
Mar 22 02:25:49 promo3 vzdump[3741]: INFO: Backup job finished successfully
Mar 22 02:25:49 promo3 citadel: 1 unique messages to be merged
Mar 22 02:25:49 promo3 citadel: 1 unique messages to be merged
Mar 22 02:25:49 promo3 vzdump[3739]: <root at pam> end task
UPID:promo3:00000E9D:00007A5A:514BB239:vzdump::root at pam: OK
Any hints to pevent kernel 2.6.32-19 from that issue?
Regards, Martin
------------------------------
_______________________________________________
pve-user mailing list
pve-user at pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
End of pve-user Digest, Vol 60, Issue 23
****************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20130322/ad6eb634/attachment.htm>
More information about the pve-user
mailing list