[pve-devel] [RFC qemu-server++ 0/22] remote migration

[alexandre derumier]

>>looking forward to feedback :) you'll need to put the 
>>proxmox-websocket-tunnel binary into $PATH of pveproxy/qm, after 
>>building it with 'cargo build'.

oh, I didn't known that the rust version was already ready. great :)
I'll try to do my first rust build :) (I really need to learn it, seem to be a great language)


>>if your inter-DC link is fast enough, you'll likely be hit by the 
>>pveproxy bottleneck. it would still be interesting to get some 
>>real-world numbers, I haven't tested with baremetal and fast storage 
>>yet.

yes, I have enough bandwidth (25gbs), so no problem here. I'll try with && without any storage to compare.
(as I'm using ceph with nvme, I think it'll be the bottleneck with qemu mirror)

>>please be aware that this is very much experimental code still!

yes, sure , no problem ;)


On 15/04/2021 16:32, Fabian Grünbichler wrote:
> On April 15, 2021 4:04 pm, alexandre derumier wrote:
>> Hi,
>>
>> thanks for working on this !
>>
>> I'll be able to test it soon as I'll need to migrate 200-300 vms between
>> 2 datacenter soon.
> looking forward to feedback :) you'll need to put the
> proxmox-websocket-tunnel binary into $PATH of pveproxy/qm, after
> building it with 'cargo build'.
>
> if your inter-DC link is fast enough, you'll likely be hit by the
> pveproxy bottleneck. it would still be interesting to get some
> real-world numbers, I haven't tested with baremetal and fast storage
> yet.
>
> please be aware that this is very much experimental code still!
>
>> I think it could be great to add optionnal "tag" option to targetbridge,
>> as it could be different on target cluster.
> hmm, we could have another (optional) map for VLAN tags? since tags and
> bridges are not one entity (you can have on interface on bridge A with
> tag X, and another interface on bridge A with tag Y, and those need to
> be mapped to bridge B with tag P and bridge B with tag Q, for example).
>
>> Also, we should transfert vm firewall config.
> yes, that's definitely true. another source of potential
> mismatches/things to check before migrating (security groups/aliases!)
>
>> On 13/04/2021 14:16, Fabian Grünbichler wrote:
>>> this series adds remote migration for VMs. there's still plenty of
>>> TODOs/FIXMEs/stuff that requires discussion, hence the RFC. live
>>> migration with NBD and storage-migrated disks should work already.
>>>
>>> the performance bottle neck (~190MB/s on loopback) for the websocket
>>> connection seems to be in pveproxy at the moment - the rust code should
>>> manage about 700MB/s.
>>>
>>> overview over affected repos and changes, see individual patches for
>>> more details.
>>>
>>> proxmox:
>>>
>>> some compatible changes to make websocket code usable for client-side
>>> connections, required by proxmox-websocket-tunnel
>>>
>>> proxmox-websocket-tunnel:
>>>
>>> new tunnel helper tool for forwarding commands and data over websocket
>>> connections, required by qemu-server on source side
>>> TODO: better error handling
>>> TODO: fingerprint checking/valid certs/..
>>> TODO: WS key generation
>>> TODO: decide on mask?
>>> TODO: investigate performance bottlenecks once PVE api server gets
>>> faster
>>>
>>> pve-access-control:
>>>
>>> new ticket type, required by qemu-server on target side
>>>
>>> pve-cluster:
>>>
>>> new remote.cfg and related helpers, required by qemu-server on source
>>> side
>>> TODO: ACLs, CLI, API for managing config
>>> TODO: handling of discovered nodes with valid certificates
>>> TODO: add additional information like default bwlimits, storage/bridge
>>> mappings
>>>
>>> pve-common:
>>>
>>> bridgepair format akin to storage pair, pve-bridge-id option, required
>>> by qemu-server
>>> TODO: adapt pve-container
>>>
>>> pve-guest-common:
>>>
>>> handle remote migration (no SSH) in AbstractMigrate,
>>> required by qemu-server
>>>
>>> pve-manager:
>>>
>>> new 'addr' endpoint for retrieving remote node IPs, required on target
>>> node
>>>
>>> pve-storage:
>>>
>>> extend 'pvesm import' to allow import from UNIX socket, required on
>>> target node by qemu-server
>>>
>>> qemu-server:
>>>
>>> some refactoring, new mtunnel endpoints, new remote_migration endpoints
>>> TODO: check remote ACLs
>>> TODO: handle pending changes and snapshots
>>> TODO: CLI for remote migration
>>> potential TODO: expose remote info via additional endpoints (resources? vmids?
>>> permissions? ...)
>>>
>>> as usual, some of the patches are best viewed with '-w', especially in
>>> qemu-server..
>>>
>>>
>>> _______________________________________________
>>> pve-devel mailing list
>>> pve-devel at lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>>
>>