[pve-devel] qemu memory hotplug limited to 64 dimm

[Alexandre DERUMIER]

>>3) Drop support for hot*UN*plugging altogether and simply always hot-add
>>one big dimm. ;-)
>>On that note, my guests currently hang when I unplug memory and I bet
>>that happens to all of you and nobody ever noticed because nobody needs
>>that feature anyway ;-)

yes, it's a linux guest limitation. (qemu implementation is ok), but linux kernel memory is not movable currently, and if it's located on a dimm, you can't unplug it, as the kernel memory can't be move at another location.




----- Mail original -----
De: "Wolfgang Bumiller" <w.bumiller at proxmox.com>
À: "Alexandre Derumier" <aderumier at odiso.com>
Cc: "dietmar" <dietmar at proxmox.com>, "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Mercredi 14 Décembre 2016 15:07:16
Objet: Re: [pve-devel] qemu memory hotplug limited to 64 dimm

So I see a few options one or more of which we should consider: 

1) Make the *static* memory size configurable: 

It doesn't make sense to me to eg. want to go from 1G to 200G in 
small steps. Worse is the fact that even when starting with a higher 
amount knowing that I'm only going to increase it, I start with lots and 
lots of tiny dimms. Eg. 8G = 1G static + 14 x 512M dimms. 

This could be added as a separate option or even to the existing 
`memory` option without breaking existing configs (by turning it into a 
property string with the current amount as the default key.) 
I could then do: 

memory: 8192,static=8192 

This example frees up 14 dimm slots alone. 
The downside is this VM cannot go below 8G without a reboot. 

The dimm size should then probably not start at 512M either. That's 
where the next change comes in: 

2) Change the steps one way or another: 

a) Generally change the number of steps before increasing the step 
size: 
The current 32 makes sense for the first few gigs, but if we eg. have 
a machine with 101G memory, we'd still be doing 2G steps. That 
amounts to a 2% granularity. At 113G we bump this up to 4G steps, 
which is a 3.5% granularity. 
I propose we use no more than 16 dimms of each size. (Heck I'd even 
be fine with 8 personally, although 14 of the 512M dimms would be 
good to get to a "nice" 8G in small steps. ;-) ) 

This change is not backward compatible and would break memory 
hot-unplugging of running machines after the upgrade. 
Or well, it would still "happen", but our count would be out of 
sync. 

b) Make the _initial_ dimm size configurable: 
This would work well in combination with (1). (This initial dimm 
size would not be hot-pluggable as that would throw off our 
counter.) 

c) Have the option to use only 1 fixed dimm size. 
Since we already have a way to get a list of a running VM's dimms, 
this option could even be hot-pluggable. The unplug code would have 
to be changed to fetch the list of dimms from qemu rather than 
assume our fixed dimm list is used. 
[*] This would probably be a good idea in any case ;-) 
Of course this would allow the user to add a 16G dimm, change the 
dimm size to 1G and then try to reduce memory by 1 and fail. 
(I see no reason to protect people from too many of their own 
mistakes, but we *could* of course limit the hot-plugging of the 
dimm size to increasing only...) 

c) Use an explicit dimm list. 
This would be an alternative to (1), you define the static memory 
and the dimms, the latter can be hot plugged: 

memory: 8192,dimms=512x4:1024x4 
or 
memory: 8192 
dimms: 512x4 1024x4 

Would be the equivalent of 8G static memory plus 4 dimms of 512M 
and 1G each - a total of 14G. 

For backward compatibility we could require dimms to be "defined 
but empty" to use the new method, and keep the old method 
otherwise. 

Downside: who wants to make a GUI for that? ;-) 

3) Drop support for hot*UN*plugging altogether and simply always hot-add 
one big dimm. ;-) 
On that note, my guests currently hang when I unplug memory and I bet 
that happens to all of you and nobody ever noticed because nobody needs 
that feature anyway ;-)