[PVE-User] lxc container can't start on a newly added cluster node

[Stoiko Ivanov]

On Fri, 11 Jan 2019 15:58:03 +0100
"Marco M. Gabriel" <marcomgabriel at gmail.com> wrote:

> I just tried to set up a new ubuntu 16.04 and an ubuntu 18.04
> container on the new node - and both are running fine. but the old
> one still doesn't start.
> 
> After this, I wanted to trace a new and the old one and compare the
> logs of both, but it seems that neither the old one nor the new one
> start when I use
> 
> "/usr/bin/lxc-start -n <vmid> -F -f /etc/pve/lxc/<vmid>.conf --logfile
> /tmp/lxc<vmid>.log --logpriority TRACE"
> 
> The new and the old one do not start when I use that command to get a
> log. The logs are the same for both containers.
> 
> If that command above does not work, then how to debug a container
> that does not want to start?

lxc-start -n <vmid> -l TRACE -o /tmp/<vmid>.log works for me (and
starts the container in the background (then you can run all `pct`
commands for stopping/entering/configuring it...

the problem is most likely the `-f` switch - that expects an
lxc-config, and PVE uses its own config-format.

see also:
https://pve.proxmox.com/pve-docs/chapter-pct.html#_obtaining_debugging_logs

Hope that helps!
stoiko

> 
> Thanks,
> Marco
> 
> Am Fr., 11. Jan. 2019 um 14:21 Uhr schrieb Stoiko Ivanov <
> s.ivanov at proxmox.com>:  
> 
> > Sorry - I misread your original mail (and thought 3 of your nodes
> > were still running PVE 4.x).
> >
> > do you have the same versions of all relevant packages
> > (pve-container, lxcfs, lxc-pve,..) on all nodes?
> > does creating a fresh container work on the new node (also ubuntu
> > 16.04)?
> >
> > FWIW - could not reproduce the issue locally (created 1 privileged
> > and 1 unprivileged Ubuntu 16.04 container and both started).
> >
> > When starting with loglevel TRACE I do see a few more lines between
> > the 'Populated "/dev"' and the error. Also the console and
> > tty-creation does log a few lines further up - maybe they provide
> > some hints where the problem lies.
> >
> >
> > On Fri, 11 Jan 2019 12:48:33 +0100
> > "Marco M. Gabriel" <marcomgabriel at gmail.com> wrote:
> >  
> > > the /etc/fstab is there, but it is quite empty:
> > >
> > > cat fstab
> > > # UNCONFIGURED FSTAB FOR BASE SYSTEM
> > >
> > > thanks,
> > > Marco
> > >
> > >
> > > Am Fr., 11. Jan. 2019 um 12:44 Uhr schrieb Stoiko Ivanov <  
> > > s.ivanov at proxmox.com>:  
> > >  
> > > > Hi,
> > > >
> > > > just a guess - does the container have an `/etc/fstab` inside?
> > > > and does the `/etc/fstab` contain an entry for `/dev/pts`?
> > > >
> > > > If yes - I guess the container would start on PVE 5.3 when you
> > > > remove the line (or the complete `/etc/fstab`). This might
> > > > result in it not starting on PVE 4.x.
> > > >
> > > > Could you test this?
> > > >
> > > > Best regards,
> > > > stoiko
> > > >
> > > >
> > > >
> > > > On Fri, 11 Jan 2019 11:18:09 +0100
> > > > "Marco M. Gabriel" <marcomgabriel at gmail.com> wrote:
> > > >  
> > > > > Hi there,
> > > > >
> > > > > we recently added a 4th proxmox node to our 3-node proxmox
> > > > > cluster. When trying to migrate or start an lxc container to
> > > > > the new node, it doesn't start because it can't
> > > > > mount /dev/pts/x to /dev/console. I can move the lxc
> > > > > container to each of the 3 old nodes and start it there, but
> > > > > it refuses to start on the new node.
> > > > >
> > > > > Nodes 1 to 3 were installed with Version 4.4 and upgraded
> > > > > until the current 5.3-6/37b3c8df (running kernel:
> > > > > 4.15.18-9-pve) Node 4 was installed a few days ago, but is on
> > > > > exactly the same version now.
> > > > >
> > > > > The lxc container is an ubuntu 16.04 based container from the
> > > > > proxmox repository and was installed about a month ago.
> > > > >
> > > > > Trace log of lxc-start (shortened):
> > > > >
> > > > > lxc-start 116 20190111094747.940 INFO     conf -
> > > > > conf.c:lxc_fill_autodev:1220 - Populating "/dev"
> > > > > lxc-start 116 20190111094747.940 DEBUG    conf -
> > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/full"
> > > > > lxc-start 116 20190111094747.940 DEBUG    conf -
> > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/null"
> > > > > lxc-start 116 20190111094747.940 DEBUG    conf -
> > > > > conf.c:lxc_fill_autodev:1235 - Created device node
> > > > > "/dev/random" lxc-start 116 20190111094747.940 DEBUG    conf -
> > > > > conf.c:lxc_fill_autodev:1235 - Created device node "/dev/tty"
> > > > > lxc-start 116 20190111094747.940 DEBUG    conf -
> > > > > conf.c:lxc_fill_autodev:1235 - Created device node
> > > > > "/dev/urandom" lxc-start 116 20190111094747.940 DEBUG    conf
> > > > > - conf.c:lxc_fill_autodev:1235 - Created device node
> > > > > "/dev/zero" lxc-start 116 20190111094747.940 INFO     conf -
> > > > > conf.c:lxc_fill_autodev:1297 - Populated "/dev"
> > > > > lxc-start 116 20190111094747.940 ERROR    utils -
> > > > > utils.c:safe_mount:1791 - No such file or directory - Failed
> > > > > to mount /dev/pts/3 onto /dev/console lxc-start 116
> > > > > 20190111094747.940 ERROR    conf -
> > > > > conf.c:lxc_setup_dev_console:1765 - Failed to mount
> > > > > "/dev/pts/3" on "/dev/console" lxc-start 116
> > > > > 20190111094747.940 ERROR    conf - conf.c:lxc_setup:3656 -
> > > > > Failed to setup console lxc-start 116 20190111094747.940
> > > > > ERROR    start - start.c:do_start:1246 - Failed to setup
> > > > > container "116"
> > > > >
> > > > > Any hints how to fix this or what happens here?
> > > > >
> > > > > Best regards,
> > > > > Marco
> > > > > _______________________________________________
> > > > > pve-user mailing list
> > > > > pve-user at pve.proxmox.com
> > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user  
> > > >
> > > >
> > > >  
> >
> >
> >