[pve-devel] KVM Templates on CephFS are only shown on one node - not all nodes in the cluster

[Dominik Csapak]

On 3/10/20 12:10 PM, Victor Hooi wrote:
> Hi Dominik,
> 

Hi,

> Aha - I apologise - let me explain the use-case.
> 
> We have two Proxmox clusters in our lab - both are HA clusters, using 
> hyper-converged Ceph to store the VM disk images.
> 
> In each cluster, we have a template of various VM images (e.g. Windows 
> 7, Windows 8.1, Windows 10 etc), in various different 
> configurations/software combinations.
> 
> We want the same templates to appear /under/ each node because:
> 
>   * Redundancy/HA - if one of the nodes is down - Ceph can still
>     function, and we can still spin-up VMs. However, if a template only
>     appears under node1, and node1 dies - then we cannot use any
>     templates anymore and the cluster is useless (even if the files are
>     technically still accessible via Ceph)

as i said, mark the template with ha as well, when node1 dies,
the template config gets moved to a node in the quorate partition.

>   * Distributing VMs across the cluster - If all the templates appear
>     under node1 - then when a user goes to clone it, it will default to
>     node1. Proxmox currently doesn't have a feature to automatically
>     distribute VMs across the cluster (see this thread for a discussion
>     on the feature
>     <https://forum.proxmox.com/threads/3-node-proxmox-ceph-cluster-how-to-automatically-distribute-vms-among-nodes.52443/>).
>     If the templates appear under each node (even if they point to the
>     same underlying file), it makes it easier to get users to understand
>     you can spin them up there

i get it, but even if this would work, the user would have to evenly
select the different nodes, also i would find it confusing seeing
the same template multiple times, as i would wonder if they are really
the same or not...

> 
> 
> I tried to do it just now - this is what I did.
> 
> I had a VM on node1. I converted it into a template:
> 
> https://i.imgur.com/xlLXUy5.png <https://i.imgur.com/xlLXUy5.png> - this 
> refers to vm_storage:base-500-disk-0/base-100-disk-0,size=30G
> 
> I then did a linked clone of this onto node2 - which I also converted 
> into a template
> 
> https://i.imgur.com/PUel6RB.png <https://i.imgur.com/PUel6RB.png> - this 
> refers to vm_storage:base-100-disk-0/base-104-disk-0,size=30G

so where is the problem?
vm 104 now has a disk 'base-104-disk-0' whose parent disk is
'base-100-disk-0' whose parent disk is 'base-500-disk-0'
(we only save one level of the tree)

so no data is actually duplicated, you can verify this by
using 'rbd ls -l' to see that they are still chained

i hope i wrote it clear enough

> 
> Do you know the correct steps to go from a VM, to two templates on two 
> different nodes, pointing to the same base image?
no, each vm has its own images, but this is only a logical step, and
does not produce a new full image (besides some metadata on storage level)


hope this helps :)
kind regards
Dominik