[PVE-User] KVM Guest Ubuntu goes RO-Filesystem every some night

[kai at zimmer.net]

Hi Chris, 

some more information would be helpful: 

- Which version of Proxmox are you running? 

- Are the RAIDs local or network attached (NAS)? 

- What type of disk type do you use (virtio, raw), which disk format
(qcow2) do you use? Do you use caching? 

- How large are your virtual disks, how much are they used in % - and
especially - is there enough free disk space on the host (are you
over-provisioning)? 

- Did you try disabling HPET (in the vm configuration)? 

My first guess would be, that your virtual disk usage grows at night
(maybe dumping databases for backup purposes), so that you don't have
enough disk space on the host left. But it could also be a problem with
NAS devices if network traffic for backup raises at night... 

Best, 

Kai

Am 20.05.2016 08:19, schrieb Christopher Meyering:

> Hi folks,
> 
> Maybe someone of you has some idea how to get rid of my strange "ro-filesytem" problem.
> 
> First of all some basics:
> Proxmox runs on a potent host, powering the kvm-guests with 2 raids. (1 ssd raid & 2 sas raid)
> In our pve cluster we currently run 4 identical hosts with overall 50 kvm guests on it.
> 
> Every guest runs ubuntu server 14.04 with webservers, mysql & solr services.
> 
> Since 3 weeks ago some guests started to fall into ro-state root filesystems at night without any noticeable reason.
> The stress on the hosts and guests is quite normal at these times and the problems occure on different times between 01:00 and 06:30 in the morning.
> Occurently i can see some serious cpu soft locks with over 40s stuck time.
> 
> By now i don't have much ideas left how to debug / solve this one out.
> 
> Any ideas?
> 
> Greetings,
> Chris
> 
> -- 
> 
> Mit freundlichem Gruß
> Christopher Meyering
> Technical Consultant
> 
> DIA Connecting Software
> ------------------------------------------------------------------------
> 
> DIA Connecting Software GmbH & Co. KG · Menkestraße 23 · 26419 Schortens
> Fon 04461 / 899 89 39 · Fax 04461 / 899 89 41
> www.dia-software.de [1] <http://www.dia-software.de> · christopher.meyering at dia-software.de <mailto:christopher.meyering at dia-software.de>
> 
> Persönlich haftende Gesellschafterin: DIA Connecting Software Beteiligungs GmbH
> Handelsregister Wilhelmshaven HR B 131 604
> Geschäftsführer: Dipl.-Ing. Ansgar Gallas, Marco Walther
> Registergericht: Amtsgericht Oldenburg
> Handelsregister Wilhelmshaven: HR A 13 08 89
> Umsatzsteuer-Identifikationsnummer: DE813971952
> 
> Die Information in dieser Email ist vertraulich und ist ausschließlich für den Adressaten bestimmt. Jeglicher Zugriff auf diese Email durch andere Personen als den Adressaten ist untersagt.
> Sollten Sie nicht der für diese Email bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Maßnahmen im Vertrauen
> auf erlangte Information untersagt. In dieser Email enthaltene Meinungen oder Empfehlungen unterliegen den Bedingungen des jeweiligen Kundenverhältnisses mit dem Adressaten.
> 
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
 

Links:
------
[1] http://www.dia-software.de