[PVE-User] [DRBD-user] Stability issue Proxmox4/DRBD9

Wed Apr 13 11:29:18 CEST 2016

IIRC the kernel module is done by dkms so it is automatically build through apt process. I think this is a ok, isn’t it 

On 13 Apr 2016, at 11:15, Jean-Laurent Ivars <jl.ivars at ipgenius.fr> wrote:

> Hello Patrick,
> 
> I don’t understand what you mean since drbd is kernel related, if I upgrade the kernel the drbd module will be automatically upgraded too… So It is necessary to compile the module every time a new kernel upgrade is done. There’s no way to have it managed by apt.
> 
> If you want to understand more, you should read this page :http://coolsoft.altervista.org/en/blog/2016/04/proxmox-41-kernel-panic-downgrade-drbd-resources-drbd-9-84
> 
> Best regards,
> 
> 	
> Jean-Laurent Ivars 
> Responsable Technique | Technical Manager
> 22, rue Robert - 13007 Marseille 
> Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47 
> Linkedin <http://fr.linkedin.com/in/jlivars/>   |  Viadeo <http://www.viadeo.com/fr/profile/jean-laurent.ivars>   |  www.ipgenius.fr <https://www.ipgenius.fr/>
>> Le 12 avr. 2016 à 22:14, Patrick Kaiser <pk at argonius.de> a écrit :
>> 
>> Hey Jean-Laurent,
>> 
>> ok, your decision. But if you exclude drbd packages from pve repo, than you can stick on the default debian rebo and your system is managed by apt and you will have drbd8 (as it is packaged by debian)
>> 
>> 
>> On 11 Apr 2016, at 20:00, Jean-Laurent Ivars <jl.ivars at ipgenius.fr <mailto:jl.ivars at ipgenius.fr>> wrote:
>> 
>>> Hey Patrick,
>>> 
>>> I do think the crash is related to DRBD but if I downgrade to 8.4, I will have to compile the module every time there is a new kernel… and with no guaranties that there will not have a problem one day, I would feel more comfortable with a system completely managed by apt…
>>> 
>>> 
>>> 
>>> 	
>>> Jean-Laurent Ivars 
>>> Responsable Technique | Technical Manager
>>> 22, rue Robert - 13007 Marseille 
>>> Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47 
>>> Linkedin <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>>   |  Viadeo <http://www.viadeo.com/fr/profile/jean-laurent.ivars <http://www.viadeo.com/fr/profile/jean-laurent.ivars>>   |  www.ipgenius.fr <http://www.ipgenius.fr/> <https://www.ipgenius.fr/<https://www.ipgenius.fr/>>
>>>> Le 11 avr. 2016 à 19:08, Patrick Kaiser <pk at argonius.de <mailto:pk at argonius.de>> a écrit :
>>>> 
>>>> Hey Jean-Laurent,
>>>> 
>>>> i do not see any need for downgrading to PVE 3. The only reason to downgrade would be to have the quorum feature back, so you can track for example gateways and other stuff
>>>> and take this information for your ha decisions.
>>>> 
>>>> related to the DRBD issue: it is only related to DRBD not Proxmox. Using PVE 4.1 with DRBD 8.4 from debian stable should be no problem. 
>>>> Or is the kernel crash related to some proxmox specific stuff? I think it is just based on the drbd module…
>>>> 
>>>> 
>>>> On 11 Apr 2016, at 16:10, Jean-Laurent Ivars <jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr> <mailto:jl.ivars at ipgenius.fr <mailto:jl.ivars at ipgenius.fr>>> wrote:
>>>> 
>>>>> Actually I am thinking about doing the same thing, some kind people gave me procedure to do it but I am not very comfortable with a production system that can be broken by a simple update… 
>>>>> So I am wondering very strongly if I should go that way or downgrade the whole Proxmox Install to 3.4…
>>>>> 
>>>>> 	
>>>>> Jean-Laurent Ivars 
>>>>> Responsable Technique | Technical Manager
>>>>> 22, rue Robert - 13007 Marseille 
>>>>> Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47 
>>>>> Linkedin <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>>>   |  Viadeo <http://www.viadeo.com/fr/profile/jean-laurent.ivars <http://www.viadeo.com/fr/profile/jean-laurent.ivars> <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars>>>   |  www.ipgenius.fr<http://www.ipgenius.fr/> <http://www.ipgenius.fr/ <http://www.ipgenius.fr/>> <https://www.ipgenius.fr/<https://www.ipgenius.fr/><https://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E>>
>>>>>> Le 11 avr. 2016 à 16:02, Patrick Kaiser <pk at argonius.de <mailto:pk at argonius.de> <mailto:pk at argonius.de <mailto:pk at argonius.de>>> a écrit :
>>>>>> 
>>>>>> Ah great. So on pve wiki i’ve already seen, but I never have seen that on linbit’s page… So i think i will stick 
>>>>>> on PVE 4.1 but switch Proxmox Back to DRBD 8.4.  
>>>>>> 
>>>>>> Do you think, this combination will do better? :)
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 11 Apr 2016, at 15:53, Jean-Laurent Ivars <jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr> <mailto:jl.ivars at ipgenius.fr <mailto:jl.ivars at ipgenius.fr>> <mailto:jl.ivars at ipgenius.fr <mailto:jl.ivars at ipgenius.fr> <mailto:jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr>>>> wrote:
>>>>>> 
>>>>>>> Hi there !
>>>>>>> 
>>>>>>> I did the same as you, I set up my DRBD9 PVE4 the old way (2 ressources…) but I have issues. 
>>>>>>> Maybe you did not notice it because your system is not loaded, for example, in my case, everything works perfectly until I try do clone a VM especially when I clone from one ressource to an other one. Sometimes it happen when I choose to « move » a disk from one ressource to the other. 
>>>>>>> What happen exactly, you can see it in my first mail to the mailing list. Basically the kernel crash.
>>>>>>> 
>>>>>>> Maybe you are lucky and on your material you will have no problem ? I really don’t know, it’s just a suspicion but i think it’s CPU relative.
>>>>>>> 
>>>>>>> You will find that it is not production ready on Proxmox DRBD9 wiki page and on Linbit pages :
>>>>>>> 
>>>>>>> https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9> <https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9>> <https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9> <https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9>>> <https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9> <https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9>> <https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9> <https://pve.proxmox.com/wiki/DRBD9 <https://pve.proxmox.com/wiki/DRBD9>>>>
>>>>>>> https://www.linbit.com/en/partners/11-en/products-and-services/drbd<https://www.linbit.com/en/partners/11-en/products-and-services/drbd><https://www.linbit.com/en/partners/11-en/products-and-services/drbd%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3E> <https://www.linbit.com/en/partners/11-en/products-and-services/drbd%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3E%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3E%3E><https://www.linbit.com/en/partners/11-en/products-and-services/drbd<https://www.linbit.com/en/partners/11-en/products-and-services/drbd><https://www.linbit.com/en/partners/11-en/products-and-services/drbd%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3E> <https://www.linbit.com/en/partners/11-en/products-and-services/drbd%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3E%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3Chttps://www.linbit.com/en/partners/11-en/products-and-services/drbd%3E%3E>>
>>>>>>> 
>>>>>>> regards,
>>>>>>> 
>>>>>>> 	
>>>>>>> Jean-Laurent Ivars 
>>>>>>> Responsable Technique | Technical Manager
>>>>>>> 22, rue Robert - 13007 Marseille 
>>>>>>> Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47 
>>>>>>> Linkedin <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>>>>   |  Viadeo <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars> <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars>> <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars>> <http://www.viadeo.com/fr/profile/jean-laurent.ivars%3Chttp://www.viadeo.com/fr/profile/jean-laurent.ivars%3E%3E>>   | www.ipgenius.fr <http://www.ipgenius.fr/><http://www.ipgenius.fr/<http://www.ipgenius.fr/>> <http://www.ipgenius.fr/ <http://www.ipgenius.fr/> <http://www.ipgenius.fr/ <http://www.ipgenius.fr/>>> <https://www.ipgenius.fr/<https://www.ipgenius.fr/><https://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E><https://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E%3Chttps://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E%3E>>
>>>>>>>> Le 11 avr. 2016 à 14:53, Patrick Kaiser <pk at argonius.de <mailto:pk at argonius.de> <mailto:pk at argonius.de <mailto:pk at argonius.de>> <mailto:pk at argonius.de<mailto:pk at argonius.de> <mailto:pk at argonius.de <mailto:pk at argonius.de>>>> a écrit :
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> i was also just setting up PVE 4.1 with DRBD9. But I’ve setup drbd9 in the “old-way”.
>>>>>>>> 
>>>>>>>> Currently there is no load on the system and everything is working like expected. But now I think about resetup everything with latest Proxmox 3.x and latest DRBD 8.
>>>>>>>> 
>>>>>>>> I do not see any noticeable feature in PVE 4 I will miss in PVE3 but I am thinking more about problems in Upgrade process later on…
>>>>>>>> 
>>>>>>>> Where do you guys got the information, that DRBD 9 is NOT production ready?
>>>>>>>> 
>>>>>>>> thanks,
>>>>>>>> patrick
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 11 Apr 2016, at 12:08, Jean-Laurent Ivars <jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr><mailto:jl.ivars at ipgenius.fr <mailto:jl.ivars at ipgenius.fr>><mailto:jl.ivars at ipgenius.fr <mailto:jl.ivars at ipgenius.fr> <mailto:jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr>>> <mailto:jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr> <mailto:jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr>> <mailto:jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr><mailto:jl.ivars at ipgenius.fr<mailto:jl.ivars at ipgenius.fr>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Claudio,
>>>>>>>>> 
>>>>>>>>> I wonder the same as you and I’m a little bit afraid about the production ready thing…
>>>>>>>>> 
>>>>>>>>> I am wondering about downgrading my whole Proxmox installation to have a stable system rather than using something that can brake any day with a simple update… Do you think the version 4 features worth the risk and I should go your way ?
>>>>>>>>> 
>>>>>>>>> Thank you for your answer again
>>>>>>>>> Best regards,
>>>>>>>>> 
>>>>>>>>> 	
>>>>>>>>> Jean-Laurent Ivars 
>>>>>>>>> Responsable Technique | Technical Manager
>>>>>>>>> 22, rue Robert - 13007 Marseille 
>>>>>>>>> Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47 
>>>>>>>>> Linkedin <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>>> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/> <http://fr.linkedin.com/in/jlivars/ <http://fr.linkedin.com/in/jlivars/>>>>>   |  Viadeo <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars> <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars>> <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars><http://www.viadeo.com/fr/profile/jean-laurent.ivars%3Chttp://www.viadeo.com/fr/profile/jean-laurent.ivars%3E>> <http://www.viadeo.com/fr/profile/jean-laurent.ivars<http://www.viadeo.com/fr/profile/jean-laurent.ivars><http://www.viadeo.com/fr/profile/jean-laurent.ivars%3Chttp://www.viadeo.com/fr/profile/jean-laurent.ivars%3E>> <http://www.viadeo.com/fr/profile/jean-laurent.ivars%3Chttp://www.viadeo.com/fr/profile/jean-laurent.ivars%3E%3E<http://www.viadeo.com/fr/profile/jean-laurent.ivars%3Chttp://www.viadeo.com/fr/profile/jean-laurent.ivars%3E%3E>>>   |www.ipgenius.fr <http://www.ipgenius.fr/> <http://www.ipgenius.fr/<http://www.ipgenius.fr/>><http://www.ipgenius.fr/<http://www.ipgenius.fr/><http://www.ipgenius.fr/%3Chttp://www.ipgenius.fr/%3E>> <http://www.ipgenius.fr/<http://www.ipgenius.fr/> <http://www.ipgenius.fr/ <http://www.ipgenius.fr/>> <http://www.ipgenius.fr/ <http://www.ipgenius.fr/> <http://www.ipgenius.fr/<http://www.ipgenius.fr/>>>> <https://www.ipgenius.fr/<https://www.ipgenius.fr/><https://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E> <https://www.ipgenius.fr/<https://www.ipgenius.fr/> <https://www.ipgenius.fr/ <https://www.ipgenius.fr/>>> <https://www.ipgenius.fr/<https://www.ipgenius.fr/><https://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E>> <https://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E%3E<https://www.ipgenius.fr/%3Chttps://www.ipgenius.fr/%3E%3E>>>
>>>>>>>>>> Le 11 avr. 2016 à 10:13, Claudio Nicora <claudio.nicora at gmail.com<mailto:claudio.nicora at gmail.com>> a écrit :
>>>>>>>>>> 
>>>>>>>>>> Welcome aboard, and thanks for bringing back an issue that nobody took care of since now.
>>>>>>>>>> It means I was not the alone...
>>>>>>>>>> 
>>>>>>>>>> I've had the same issue a few months ago, I've reported it here (and on Proxmox forums) but the issue is still there.
>>>>>>>>>> 
>>>>>>>>>> Linbit: nobody took care of it (however it's a kernel crash, should be critical enough...)
>>>>>>>>>> Proxmox: a kernel with an updated DRBD module (still DRBD 9.0) was released but the issue persisted. Still wondering how a "non-production ready" feature could end up in a "production ready" bundle like Proxmox 4 without a choice to get DRBD 8.4 back.
>>>>>>>>>> 
>>>>>>>>>> I've sorted it out by downgrading DRBD module to 8.4 and building the kernel module + DRBD tools on my own.
>>>>>>>>>> This is the post on Proxmox forum with a backlink to my blog post where I detailed the actions to downgrade the module.
>>>>>>>>>> I've also set up a script to run after each Proxmox kernel upgrade, just in case.
>>>>>>>>>> 
>>>>>>>>>> PS: still waiting for a feedback from Linbit/Proxmox guys ;)
>>>>>>>>>> _______________________________________________
>>>>>>>>>> drbd-user mailing list
>>>>>>>>>> drbd-user at lists.linbit.com <mailto:drbd-user at lists.linbit.com>
>>>>>>>>>> http://lists.linbit.com/mailman/listinfo/drbd-user<http://lists.linbit.com/mailman/listinfo/drbd-user>
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> pve-user mailing list
>>>>>>>>> pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>>> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>>>>
>>>>>>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user<http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user> <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user<http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user>> <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user<http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user><http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E>> <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user<http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user><http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E><http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E%3E>>
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> pve-user mailing list
>>>>>>>> pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>>> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com> <mailto:pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>>>>
>>>>>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user> <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user>> <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user<http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user><http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E>> <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user<http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user><http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E><http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3Chttp://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user%3E%3E>>
>>>>>>> _______________________________________________
>>>>>>> pve-user mailing list
>>>>>>> pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>
>>>>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user>
>>>>>> 
>>>>>> _______________________________________________
>>>>>> pve-user mailing list
>>>>>> pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>
>>>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user>
>>>>> _______________________________________________
>>>>> pve-user mailing list
>>>>> pve-user at pve.proxmox.com
>>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>> 
>>>> _______________________________________________
>>>> pve-user mailing list
>>>> pve-user at pve.proxmox.com
>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> 
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user>
>> 
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user <http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user