[PVE-User] VMs created in rapid succession are assigned the same IPv4 address
Adrian Petrescu
adrian at apetre.sc
Mon Dec 2 23:18:59 CET 2019
Hey all, I have a pretty intriguing issue.
I'm spinning up VMs through a Terraform provider(https://github.com/Telmate/terraform-provider-proxmox
if it matters), which goes through the /api2/json endpoints. They are
all full clones of a simple ubuntu1804 template. Everything is working
just fine when I spin them up one at a time. The VMs are all just using
a simple vmbr0 bridge with CIDR 192.168.128.207/16.
However, if I use `count = N` (with N > 1) to create multiple VMs "at
once" (I'm using scare quotes because they are still just individual
calls to `POST /api2/json/nodes/<NODE>/qemu/<ID>/clone` being fired off
in rapid succession), then once everything comes up, I find that all the
VMs in that batch were assigned the same IPv4 address, which makes all
but one of them inaccessible.
The IPv6 address is different, the MAC addresses are different, and if I
reboot the VM, the IPv4 address gets reassigned to something unique as
well, so it's not as if the parameterization is somehow forcing it. If I
slow the calls down and make them one at a time, everything is fine. So
it really does seem like the DHCP server has some sort of strange race
condition that ignores the MAC. But surely any reasonable DHCP
implementation can deal with such a basic case, so I must be missing
something.
If it helps, below is an example of 3 consecutive clones to the API which
produced this outcome (with Cookie headers redacted).
Any ideas what I might be doing wrong? Please let me know if I'm leaving
out any other details that might be important :)
=====================================================================
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 >>>>>>>>>> REQUEST:
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: POST /api2/json/nodes/dilithium/qemu/34000/clone HTTP/1.1
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: Host: 192.168.128.207:8006
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: User-Agent: Go-http-client/1.1
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Length: 79
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept: application/json
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Type: application/x-www-form-urlencoded
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept-Encoding: gzip
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.057-0500 [DEBUG] plugin.terraform-provider-proxmox: full=1&name=k8s-worker-5&newid=107&pool=k8s-pool&storage=local&target=dilithium
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 <<<<<<<<<< RESULT:
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: HTTP/1.1 200 OK
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Connection: close
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Length: 76
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Cache-Control: max-age=0
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Type: application/json;charset=UTF-8
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Date: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Expires: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Pragma: no-cache
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox: Server: pve-api-daemon/3.0
2019-12-02T16:10:10.087-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox: {"data":"UPID:dilithium:000051EF:000B902C:5DE57DB2:qmclone:34000:root at pam:"}
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 >>>>>>>>>> REQUEST:
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox: GET /api2/json/nodes/dilithium/tasks/UPID:dilithium:000051EF:000B902C:5DE57DB2:qmclone:34000:root at pam:/status HTTP/1.1
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox: Host: 192.168.128.207:8006
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox: User-Agent: Go-http-client/1.1
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept: application/json
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept-Encoding: gzip
2019-12-02T16:10:10.088-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 <<<<<<<<<< RESULT:
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: HTTP/1.1 200 OK
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Length: 2334
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Cache-Control: max-age=0
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Connection: Keep-Alive
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Type: application/json;charset=UTF-8
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Date: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Expires: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Pragma: no-cache
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Server: pve-api-daemon/3.0
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: {"data":[{"diskwrite":0,"template":1,"netout":0,"cpu":0,"mem":0,"maxcpu":2,"maxdisk":8589934592,"node":"dilithium","id":"qemu/34000","maxmem":1073741824,"name":"ubuntu1804-tmpl","type":"qemu","diskread":0,"uptime":0,"status":"stopped","netin":0,"disk":0,"vmid":34000},{"node":"dilithium","maxdisk":4294967296,"maxcpu":1,"mem":0,"cpu":0,"id":"qemu/106","netout":0,"diskwrite":0,"template":0,"name":"testvm","maxmem":2147483648,"type":"qemu","diskread":0,"vmid":106,"disk":0,"netin":0,"status":"stopped","uptime":0},{"id":"qemu/107","lock":"clone","status":"unknown","type":"qemu","node":"dilithium","vmid":107},{"lock":"clone","id":"qemu/105","node":"dilithium","vmid":105,"status":"unknown","type":"qemu"},{"maxmem":68719476736,"name":"k8s-worker-1","diskwrite":360395776,"netout":117805,"template":0,"id":"qemu/101","maxdisk":8589934592,"node":"dilithium","mem":733495296,"cpu":0.0018635563792405,"maxcpu":4,"uptime":1317,"pool":"k8s-pool","vmid":101,"netin":11964331,"disk":0,"status":"running","diskread":511095448,"type":"qemu"},{"status":"running","netin":7005759,"disk":0,"vmid":103,"pool":"k8s-pool","uptime":1065,"type":"qemu","diskread":503227032,"name":"k8s-worker-4","maxmem":68719476736,"mem":684974080,"cpu":0.0045257797781555,"maxcpu":4,"maxdisk":8589934592,"node":"dilithium","id":"qemu/103","template":0,"diskwrite":268002304,"netout":45075},{"status":"running","netin":11849412,"disk":0,"vmid":104,"pool":"k8s-pool","uptime":1067,"type":"qemu","diskread":501944984,"name":"k8s-worker-3","maxmem":68719476736,"mem":708276224,"cpu":0.001597334039349,"maxcpu":4,"node":"dilithium","maxdisk":8589934592,"id":"qemu/104","netout":88985,"diskwrite":299877376,"template":0},{"maxmem":68719476736,"name":"k8s-worker-2","template":0,"diskwrite":300356608,"netout":79704,"id":"qemu/102","node":"dilithium","maxdisk":8589934592,"mem":709390336,"cpu":0.002129778719132,"maxcpu":4,"uptime":1317,"pool":"k8s-pool","vmid":102,"netin":12080125,"status":"running","disk":0,"diskread":501949080,"type":"qemu"},{"diskwrite":301437952,"template":0,"netout":89181,"cpu":0.002129778719132,"mem":729665536,"maxcpu":4,"node":"dilithium","maxdisk":8589934592,"id":"qemu/100","maxmem":68719476736,"name":"k8s-worker-0","type":"qemu","diskread":512447128,"pool":"k8s-pool","uptime":2211,"disk":0,"netin":13768347,"status":"running","vmid":100}]}
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 [DEBUG] cloning VM
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 >>>>>>>>>> REQUEST:
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: POST /api2/json/nodes/dilithium/qemu/34000/clone HTTP/1.1
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Host: 192.168.128.207:8006
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: User-Agent: Go-http-client/1.1
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Length: 79
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept: application/json
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Type: application/x-www-form-urlencoded
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept-Encoding: gzip
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: full=1&name=k8s-worker-7&newid=108&pool=k8s-pool&storage=local&target=dilithium
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 <<<<<<<<<< RESULT:
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: HTTP/1.1 200 OK
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Length: 222
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Cache-Control: max-age=0
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Connection: Keep-Alive
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Connection: Keep-Alive
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Type: application/json;charset=UTF-8
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Date: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Expires: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox: Pragma: no-cache
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox: Server: pve-api-daemon/3.0
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox: {"data":{"upid":"UPID:dilithium:000051EF:000B902C:5DE57DB2:qmclone:34000:root at pam:","node":"dilithium","id":"34000","pid":20975,"type":"qmclone","user":"root at pam","status":"running","pstart":757804,"starttime":1575321010}}
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 >>>>>>>>>> REQUEST:
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: POST /api2/json/nodes/dilithium/qemu/34000/clone HTTP/1.1
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Host: 192.168.128.207:8006
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: User-Agent: Go-http-client/1.1
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Length: 79
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept: application/json
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Type: application/x-www-form-urlencoded
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: Accept-Encoding: gzip
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.105-0500 [DEBUG] plugin.terraform-provider-proxmox: full=1&name=k8s-worker-7&newid=108&pool=k8s-pool&storage=local&target=dilithium
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: 2019/12/02 16:10:10 <<<<<<<<<< RESULT:
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: HTTP/1.1 200 OK
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Length: 222
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Cache-Control: max-age=0
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Connection: Keep-Alive
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Connection: Keep-Alive
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Content-Type: application/json;charset=UTF-8
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Date: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.109-0500 [DEBUG] plugin.terraform-provider-proxmox: Expires: Mon, 02 Dec 2019 21:10:10 GMT
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox: Pragma: no-cache
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox: Server: pve-api-daemon/3.0
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox:
2019-12-02T16:10:10.110-0500 [DEBUG] plugin.terraform-provider-proxmox: {"data":{"upid":"UPID:dilithium:000051EF:000B902C:5DE57DB2:qmclone:34000:root at pam:","node":"dilithium","id":"34000","pid":20975,"type":"qmclone","user":"root at pam","status":"running","pstart":757804,"starttime":1575321010}}
--
Adrian
More information about the pve-user
mailing list