[pve-devel] [PATCH v8 0/8] Add basics for custom CPU models
Stefan Reiter
s.reiter at proxmox.com
Thu Feb 13 16:53:40 CET 2020
Hi,
On 2/13/20 4:22 PM, Eneko Lacunza wrote:
> Hi again,
>
> It is also documented elsewhere:
> https://www.linux-kvm.org/page/Migration#Note
>
First of all I want to point out that the information you posted was
last updated in 2009, and since then CPUs have gotten more and more
complex (and, if Intel's security issues are anything to go by, also
come with more bugs then ever). It _might_ have been true back then, but
I have a feeling that even then it was more wishful thinking then actual
success stories.
> Cheers
>
> El 13/2/20 a las 16:12, Eneko Lacunza escribió:
>> Hi Stefan,
>>
>> El 13/2/20 a las 15:58, Stefan Reiter escribió:
>>> Hi,
>>>
>>> On 2/13/20 1:27 PM, Eneko Lacunza wrote:
>>>> Hi Stefan,
>>>>
>>>> Do you think this could help specifying a custom CPU model that will
>>>> migrate VMs with more than 1 core between Intel and AMD CPUs?
>>>> https://bugzilla.proxmox.com/show_bug.cgi?id=1660
>>>>
>>>
>>> as discussed on the bug report you linked, live migration between
>>> hosts using different vendors' CPUs is very much unsupported in
>>> general (with KVM-enabled VMs). Even if it _appears_ to work, there
>>> are no guarantees about stability.
>>>
>>> That said, if you find a specific CPU flag configuration that seems
>>> to work for you (like comment #12 in the report mentions), this
>>> series would indeed allow you to specify it as a custom CPU model
>>> instead of having to use the '-args' parameter.
>> I don't understand why you insist that this isn't supported. Linux KVM
>> FAQ says otherwise:
>>
>> https://www.linux-kvm.org/page/FAQ#Does_KVM_support_live_migration_from_an_AMD_host_to_an_Intel_host_and_back.3F
>>
>>
>> "
>> Does KVM support live migration from an AMD host to an Intel host and
>> back?
>>
>> Yes. [...]
>> "
>>
>> I understand that there can be bugs, or maybe Proxmox VE as a product
>> can decide no to support such a feature; but technically upstrean says
>> that it should work. Shall I report the issue upstream directly?
PVE _does_ allow you to attempt live-migration between different CPU
hosts, but since time and experience has told us that it seldomly works
as intended, we don't officially support it.
This is in contrast to most other virtualization providers that simply
forbid cross-vendor migrations entirely [0][1] - and I suppose they do
it for good reason too.
Of course you can always report kernel/KVM/QEMU/etc. bugs directly
upstream, this will indeed have the highest chance of getting your
specific issues fixed. Although I personally wouldn't hold my breath for
a 'complete fix' (as in, cross-vendor live-migration working just as
well as between same-vendor hosts) anytime soon.
[0]
https://kb.vmware.com/s/article/1005764#Does%20EVC%20allow%20AMD%20and%20Intel%20CPUs%20to%20be%20vMotion%20compatible?
[1]
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn859550(v=ws.11)#when-to-use-processor-compatibility-mode
>>
>> Thanks a lot
>> Eneko
>>>
>>> Hope that helps,
>>>
>>> Stefan
>>>
>>>> Thanks a lot
>>>> Eneko
>>>>
>>>>
>>>> El 12/2/20 a las 16:11, Stefan Reiter escribió:
>>>>> Based on the RFC and following on- and off-list discussion about
>>>>> custom CPU
>>>>> models [0].
>>>>>
>>>>> In essence, this revised patch allows a user to specify custom CPU
>>>>> models in
>>>>> /etc/pve/cpu-models.conf (section-config style [1]), where VMs
>>>>> using that CPU
>>>>> model inherit details from the definition. This removes any fragile
>>>>> "auto-magical" CPU flag detection, while still giving the user a
>>>>> way to create
>>>>> VMs with the best possible subset of CPU features maintaining
>>>>> live-migration
>>>>> compatibility.
>>>>>
>>>>> Includes the infrastructure for broadcasting supported CPU flags
>>>>> for each
>>>>> cluster-node via the key-value store - this is not necessary for the
>>>>> custom-cpu feature in particular, but I think could prove useful for
>>>>> implementing the GUI part (e.g. show the user which flags are
>>>>> supported on which
>>>>> nodes).
>>>>>
>>>>>
>>>>> v7 -> v8:
>>>>> * rebase on master, fix tests now using +pve0
>>>>> * fixed nits noted by Thomas
>>>>> * moved live-migration/snapshot patches forward to avoid temporary
>>>>> breakage
>>>>> * fix CPU hotplug with custom CPUs
>>>>> * guard mkdir with check_cfs_is_mounted and also run before write
>>>>> * fix rebase-error in cfg2cmd tests (getaddrinfo_all)
>>>>> * dropped applied patches
>>>>>
>>>>> I deliberately didn't add any version_guard checks, since I don't
>>>>> think a bump
>>>>> is the right thing to do here (I say, after I waited for the
>>>>> pve-version series
>>>>> to be applied for making this v8).
>>>>>
>>>>> Doing so would break custom CPUs on older machine versions (since
>>>>> I'd have to
>>>>> pin it to 4.1~pve2) for no reason. The idea would have been to show
>>>>> a neat error
>>>>> message, but since up to this point we didn't even check the
>>>>> pve-version on
>>>>> migration, it wouldn't help much. Besides, migration to older
>>>>> versions already
>>>>> fails cleanly with a semi-readable message ('Unknown option:
>>>>> force-cpu').
>>>>>
>>>>>
>>>>> < see [2] for older history >
>>>>>
>>>>>
>>>>> [0]: https://pve.proxmox.com/pipermail/pve-devel/2019-July/038268.html
>>>>> [1]: e.g.:
>>>>> cpu-model: custom-cpu-name
>>>>> host-phys-bits 1
>>>>> flags +aes;+avx;+avx2
>>>>> reported-model kvm64
>>>>> [2]
>>>>> https://pve.proxmox.com/pipermail/pve-devel/2020-January/041278.html
>>>>>
>>>>>
>>>>> qemu-server: Stefan Reiter (8):
>>>>> Adapt CPUConfig to handle custom models
>>>>> Verify VM-specific CPU configs seperately
>>>>> Include "-cpu" parameter with live-migration
>>>>> Include "-cpu" parameter with snapshots/suspend
>>>>> Add helpers to better structure CPU option handling
>>>>> Rework get_cpu_options and allow custom CPU models
>>>>> fix #2318: allow phys-bits and host-phys-bits CPU settings
>>>>> cfg2cmd: add test cases for custom CPU models
>>>>>
>>>>> PVE/API2/Qemu.pm | 10 +-
>>>>> PVE/QemuConfig.pm | 36 +-
>>>>> PVE/QemuMigrate.pm | 22 +
>>>>> PVE/QemuServer.pm | 44 +-
>>>>> PVE/QemuServer/CPUConfig.pm | 477
>>>>> +++++++++++++++---
>>>>> test/cfg2cmd/custom-cpu-model-defaults.conf | 8 +
>>>>> .../custom-cpu-model-defaults.conf.cmd | 24 +
>>>>> test/cfg2cmd/custom-cpu-model.conf | 8 +
>>>>> test/cfg2cmd/custom-cpu-model.conf.cmd | 27 +
>>>>> test/cfg2cmd/i440fx-win10-hostpci.conf.cmd | 2 +-
>>>>> test/cfg2cmd/minimal-defaults.conf.cmd | 2 +-
>>>>> test/cfg2cmd/pinned-version.conf.cmd | 2 +-
>>>>> .../q35-linux-hostpci-multifunction.conf.cmd | 2 +-
>>>>> test/cfg2cmd/q35-linux-hostpci.conf.cmd | 2 +-
>>>>> test/cfg2cmd/q35-win10-hostpci.conf.cmd | 2 +-
>>>>> test/cfg2cmd/simple1.conf.cmd | 2 +-
>>>>> test/cfg2cmd/spice-enhancments.conf.cmd | 2 +-
>>>>> test/cfg2cmd/spice-linux-4.1.conf.cmd | 2 +-
>>>>> test/cfg2cmd/spice-usb3.conf.cmd | 2 +-
>>>>> test/cfg2cmd/spice-win.conf.cmd | 2 +-
>>>>> test/run_config2command_tests.pl | 23 +
>>>>> 21 files changed, 609 insertions(+), 92 deletions(-)
>>>>> create mode 100644 test/cfg2cmd/custom-cpu-model-defaults.conf
>>>>> create mode 100644 test/cfg2cmd/custom-cpu-model-defaults.conf.cmd
>>>>> create mode 100644 test/cfg2cmd/custom-cpu-model.conf
>>>>> create mode 100644 test/cfg2cmd/custom-cpu-model.conf.cmd
>>>>>
>>>>
>>>>
>>>
>>
>>
>
>
More information about the pve-devel
mailing list