[PVE-User] [pve-devel] online migration broken in latest updates - "unknown command 'mtunnel'"

[Thomas Lamprecht]

Hi,


I'm removing the pve-devel list from the reply-to as one is enough.

On 11/18/2016 06:01 PM, Chance Ellis wrote:
> Hi Tom,
>
> You have the essential issue.
>
> I have my original cluster. All nodes are running these versions: http://pastebin.com/vquKJaKJ
>
> As a test, I added a new node to the cluster, running these versions: http://pastebin.com/Jg5LH0RD
>
> When I try to migrate from old->new, I get the following error: http://pastebin.com/YazWBtn2

Here the migration network patch has now fault, it comes to this line:

  > ...
  > Nov 18 11:59:27 starting online/live migration on tcp:localhost:60000

Here anything related to a dedicated migration network happened already, 
the rest of the code is independent from it.

But I find it interesting that you have "tcp:localhost:60000" in the log.
This means that your node still uses the old TCP forward ssh tunnel for 
the migration.
Those did not open reliable so we switched to unix sockets, so the line 
should be something like:

  > Nov 18 17:42:38 starting online/live migration on 
unix:/run/qemu-server/167.migrate

As a workaround disable migration_unsecure or delete it from 
/etc/pve/datacenter.cfg then it should work.
I have to look into that, there may be a bug when migrating from old -> 
new and migration_unsecure on.

>
> When I try to migrate from new-> old, I get the following error: http://pastebin.com/hBfBnsYP

This is expected. But you can solve it by updating at least the 
pve-cluster pcakage on the old node, then it should work also.

cheers,
Thomas

> Thanks!
>
>
>
>
>
>
> On 11/18/16, 11:44 AM, "pve-user on behalf of Thomas Lamprecht" <pve-user-bounces at pve.proxmox.com on behalf of t.lamprecht at proxmox.com> wrote:
>
>      Hi,
>      
>      
>      On 11/18/2016 05:02 PM, Chance Ellis wrote:
>      > Hello,
>      >
>      > I am running a small cluster of 3 nodes. I would like to upgrade those nodes to the newer versions. The problem I will run into is a requirement for no down time.
>      >
>      > My normal upgrade plan is to live migrate vms off of node-1  to the remaining 2 nodes. I then upgrade and reboot node-1. Once node-1 is operational, I move vms from node-2 to node-1. I upgrade and reboot node-2. I follow the same for node-3.
>      >
>      > The issue I will run into is that once I upgrade node-1, I won’t be able to migrate vms from node-2 back to it because of the version mismatch on qemu-server and pve-cluster. The migration will fail. The only option I will have is to shutdown the vm and move the conf file. Back to the no down time requirement, this is less than ideal.
>      
>      If I understand you correctly you have
>      new node1
>      old node2
>      
>      And migration from node2 -> node1 does not work?
>      That should not be, if you run into this can you post the error from the
>      migrate command?
>      
>      We normally try to guarantee that old -> new works, the other way around
>      cannot be always guaranteed.
>      
>      I tested this also now and it worked. I down graded a test node of mine,
>      started a VM there and live migrated it successfully to a upgraded VM.
>      
>      > Is there another way to migrate the vms with the new version packages using the old method that doesn’t detect or a separate migration network?
>      
>      If you have not configured it it will not be used.
>      Migrate a unimportant test VM first to see if it works.
>      
>      cheers,
>      Thomas
>      
>      >
>      > Thanks!
>      >
>      >
>      > On 11/11/16, 2:05 AM, "pve-user on behalf of Thomas Lamprecht" <pve-user-bounces at pve.proxmox.com on behalf of t.lamprecht at proxmox.com> wrote:
>      >
>      >      On 11/10/2016 10:35 PM, Lindsay Mathieson wrote:
>      >      > On 11/11/2016 7:11 AM, Thomas Lamprecht wrote:
>      >      >> Are you sure you upgraded all, i.e. used:
>      >      >> apt update
>      >      >> apt full-upgrade
>      >      >
>      >      > Resolved it thanks Thomas - I hadn't updated the *destination* server.
>      >      >
>      >
>      >
>      >      makes sense, should have been made sense a few days ago this, would not
>      >      be too hard to catch :/
>      >
>      >      anyway, for anyone reading this:
>      >      When upgrading qemu-server to version 4.0.93 or newer you should upgrade
>      >      all other nodes pve-cluster package to version 4.0-47 or newer, else
>      >      migrations to those nodes will not work - as we use a new command to
>      >      detect if we should send the traffic over a separate migration network.
>      >
>      >      cheers,
>      >      Thomas
>      >
>      >
>      >
>      >      _______________________________________________
>      >      pve-user mailing list
>      >      pve-user at pve.proxmox.com
>      >      http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>      >
>      >
>      >
>      > _______________________________________________
>      > pve-user mailing list
>      > pve-user at pve.proxmox.com
>      > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>      
>      
>      _______________________________________________
>      pve-user mailing list
>      pve-user at pve.proxmox.com
>      http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>      
>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user