PBS backup/restore issue

Eneko Lacunza elacunza at binovo.es
Wed Feb 24 12:53:23 CET 2021 

Hi Stefan,

El 24/2/21 a las 11:34, Stefan Reiter escribió:
> On 2/23/21 8:46 AM, Eneko Lacunza wrote:
>> Hi all,
>>
>> Yesterday I hit a problem with PBS. My sequence of actions was:
>>
>> 1.a Install debian 10 VM (updated to latest, only system utilities 
>> and SSH server)
>> 1.b Add SSH access key to root
>> 1.c Shutdown VM
>> 1.d Backup to PBS and to NFS vía regular backup.
>> 1.e Start VM and install various packages - all OK.
>> 1.f Stop VM
>>
>> 2.a Restore VM from PBS backup
>> 2.b Start VM and install various packages - apt complains about no 
>> newline in linux-image-amd64 file manifest, can't install new packages.
>> 2.c Stop VM
>>
>> 3.a Restore VM from NFS regular backup
>> 3.b Start VM and install various packages - all OK.
>>
>
> Hi,
>
> I tried to replicate your problem by following pretty much the exact 
> steps you mention above, but to no avail.
Sorry for not providing enough info to replicate.
>
> Fresh Debian 10 VM, PBS backup + VMA to local disk, restored both and 
> both work. I compared the disk images of the restored VMs with md5sum 
> and the hashes matched, as expected. Tested with the latest package 
> versions, though using QEMU 5.2.0, but the backup and restore 
> functionality shouldn't have changed there.
>
> Could you try to see what the difference is between the restored disks 
> from the VMA and the PBS backups (if there is any)? I.e. restore them 
> both to seperate VMIDs and then compare hashes and/or run (v)bindiff 
> on the disk images (best restore to a directory storage in 'raw' mode) 
> to see where they differ - of course, this only makes sense if the 
> backups are created when the VM was shut down.
I have restored both to local storage:

# ls -lhs ./901/vm-901-disk-0.raw ./900/vm-900-disk-0.raw
2,4G -rw-r----- 1 root root 10G feb 24 12:06 ./900/vm-900-disk-0.raw
2,0G -rw-r----- 1 root root 10G feb 24 12:22 ./901/vm-901-disk-0.raw

900 is from PBS, 901 from vzdump:
900: vm/133/2021-02-22T17:32:23Z (date is shown as 2121-02-22 18:32:23)
901: 
/mnt/pve/txoritokieta-proxmox/dump/vzdump-qemu-133-2021_02_22-18_33_03.vma.zst 


sha256 are the same:
a3299c08a038a381a7b292a1538c975007f8637f0f3012a319d378090f434ebe 
./901/vm-901-disk-0.raw
a3299c08a038a381a7b292a1538c975007f8637f0f3012a319d378090f434ebe 
./900/vm-900-disk-0.raw

I have started vm900 and apt has run fine!
"apt install debconf php php-cli rsnapshot apache2 mariadb-server 
php-mysql acl bzip2 php-xml libapache2-mod-php libssh2-1 mariadb-client gpg"

My original restores were to a ceph storage (r2 on HDD with SSD 
journal/DB). So i have restored again both, to that ceph storage:

# rbd export -p proxmox vm-900-disk-0 - | sha256sum
Exporting image: 100% complete...done.
d05b88dc0f72f9900b2f1cddac021e9f6221eadb2b29b594eb4f3d1db34342e2  -

(OOPS!!)

# rbd export -p proxmox vm-901-disk-0 - | sha256sum
Exporting image: 100% complete...done.
a3299c08a038a381a7b292a1538c975007f8637f0f3012a319d378090f434ebe  -

(GOOD)

>
> Potentially something about your NFS/PBS setup is corrupting data? You 
> mentioned in another mail that they "are the same disks", how is that 
> configured?

 From my new data, PBS store seems good which should be quite relaxing 
for all :-) . PBS node has a mdraid. The volume is used both as NFS 
export and as PBS store.

It seems there's a bad interaction betweeen ceph storage and PBS restore???

That ceph storage has hot data and running VMs, and vzdump restores to 
it never had given any problem.

I restored VM 900 from PBS again but to another Ceph pool, r3 SSD backed:

# rbd export -p proxmox_r3_ssd vm-900-disk-0 - | sha256sum
Exporting image: 100% complete...done.
cb304ca02126d741791922dfbc484e1bdc14556a3c277aa14249cf1fd4773dca  -

This is wrong too (and has different SHA256 to the restore to HDD pool).

>
> Also please post the exact error messages APT is giving you, and your 
> VM config.

# apt install debconf php php-cli rsnapshot apache2 mariadb-server 
php-mysql acl bzip2 php-xml libapache2-mod-php libssh2-1 mariadb-client gpg
Leyendo lista de paquetes... Hecho
Creando árbol de dependencias
Leyendo la información de estado... Hecho
bzip2 ya está en su versión más reciente (1.0.6-9.2~deb10u1).
fijado bzip2 como instalado manualmente.
debconf ya está en su versión más reciente (1.5.71).
libssh2-1 ya está en su versión más reciente (1.8.0-2.1).
fijado libssh2-1 como instalado manualmente.
Se instalarán los siguientes paquetes adicionales:
   apache2-bin apache2-data apache2-utils dirmngr galera-3 gawk gnupg 
gnupg-l10n gnupg-utils gpg-agent gpg-wks-client gpg-wks-server gpgconf 
gpgsm libapache2-mod-php7.3 libapr1 libaprutil1
   libaprutil1-dbd-sqlite3 libaprutil1-ldap libassuan0 libbrotli1 
libcgi-fast-perl libcgi-pm-perl libconfig-inifiles-perl libcurl4 
libdbd-mysql-perl libdbi-perl libencode-locale-perl libfcgi-perl
   libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl 
libhttp-date-perl libhttp-message-perl libio-html-perl libjansson4 
libksba8 liblchown-perl liblua5.2-0 liblwp-mediatypes-perl libmariadb3
   libmpfr6 libnpth0 libsigsegv2 libsnappy1v5 libsodium23 
libterm-readkey-perl libtimedate-perl liburi-perl libxslt1.1 
mariadb-client-10.3 mariadb-client-core-10.3 mariadb-common 
mariadb-server-10.3
   mariadb-server-core-10.3 mysql-common php-common php7.3 php7.3-cli 
php7.3-common php7.3-json php7.3-mysql php7.3-opcache php7.3-readline 
php7.3-xml pinentry-curses psmisc rsync socat ssl-cert
Paquetes sugeridos:
   apache2-doc apache2-suexec-pristine | apache2-suexec-custom 
www-browser dbus-user-session pinentry-gnome3 tor gawk-doc parcimonie 
xloadimage scdaemon php-pear libclone-perl libmldbm-perl libnet-daemon-perl
   libsql-statement-perl libdata-dump-perl libipc-sharedcache-perl 
libwww-perl mailx mariadb-test netcat-openbsd tinyca pinentry-doc 
openssl-blacklist
Se instalarán los siguientes paquetes NUEVOS:
   acl apache2 apache2-bin apache2-data apache2-utils dirmngr galera-3 
gawk gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client 
gpg-wks-server gpgconf gpgsm libapache2-mod-php libapache2-mod-php7.3
   libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap 
libassuan0 libbrotli1 libcgi-fast-perl libcgi-pm-perl 
libconfig-inifiles-perl libcurl4 libdbd-mysql-perl libdbi-perl 
libencode-locale-perl
   libfcgi-perl libhtml-parser-perl libhtml-tagset-perl 
libhtml-template-perl libhttp-date-perl libhttp-message-perl 
libio-html-perl libjansson4 libksba8 liblchown-perl liblua5.2-0 
liblwp-mediatypes-perl
   libmariadb3 libmpfr6 libnpth0 libsigsegv2 libsnappy1v5 libsodium23 
libterm-readkey-perl libtimedate-perl liburi-perl libxslt1.1 
mariadb-client mariadb-client-10.3 mariadb-client-core-10.3 mariadb-common
   mariadb-server mariadb-server-10.3 mariadb-server-core-10.3 
mysql-common php php-cli php-common php-mysql php-xml php7.3 php7.3-cli 
php7.3-common php7.3-json php7.3-mysql php7.3-opcache php7.3-readline
   php7.3-xml pinentry-curses psmisc rsnapshot rsync socat ssl-cert
0 actualizados, 81 nuevos se instalarán, 0 para eliminar y 0 no 
actualizados.
Se necesita descargar 37,2 MB de archivos.
Se utilizarán 215 MB de espacio de disco adicional después de esta 
operación.
¿Desea continuar? [S/n]
Des:1 http://security.debian.org/debian-security buster/updates/main 
amd64 php7.3-common amd64 7.3.27-1~deb10u1 [960 kB]
[... remove dowload logs ...]
Des:81 http://deb.debian.org/debian buster/main amd64 ssl-cert all 
1.0.39 [20,8 kB]
Descargados 37,2 MB en 2s (16,8 MB/s)
debconf: Perl may be unconfigured (strict.pm did not return a true value 
at (eval 1) line 2.
BEGIN failed--compilation aborted at (eval 1) line 2.
) -- aborting
Seleccionando el paquete libapr1:amd64 previamente no seleccionado.
dpkg: error fatal irrecuperable, abortando:
  falta un salto de línea nueva al final del fichero de lista de 
ficheros del paquete `laptop-detect'
E: Sub-process /usr/bin/dpkg returned an error code (2)


>
>> I hit the issue 3 times, and tried creating a new VM and a new Debian 
>> installation each time.
>>
>> Any ideas? I have the "broken" PBS backups saved.
>>
>> VM node:
>> # pveversion -v [...]
>>
>> PBS node:
>> # pveversion -v [...]
>
> FYI, the more useful command for PBS would be:
> proxmox-backup-manager versions --verbose
>
Thanks:

# proxmox-backup-manager versions --verbose
proxmox-backup             unknown    running kernel: 5.4.78-2-pve
proxmox-backup-server      1.0.6-1    running version: 1.0.6
pve-kernel-5.4             6.3-5
pve-kernel-helper          6.3-5
pve-kernel-5.3             6.1-6
pve-kernel-5.4.78-2-pve    5.4.78-2
pve-kernel-5.4.65-1-pve    5.4.65-1
pve-kernel-5.4.44-2-pve    5.4.44-2
pve-kernel-5.3.18-3-pve    5.3.18-3
ifupdown2                  unknown
libjs-extjs                6.0.1-10
proxmox-backup-docs        1.0.6-1
proxmox-backup-client      1.0.8-1
proxmox-mini-journalreader 1.1-1
proxmox-widget-toolkit     2.4-5
pve-xtermjs                4.7.0-3
smartmontools              7.1-pve2
zfsutils-linux             2.0.3-pve1

Hope this info helps you to replicate the issue.

Cheers


-- 


      EnekoLacunza

Director Técnico | Zuzendari teknikoa

Binovo IT Human Project




	943 569 206 <tel:943 569 206>

	elacunza at binovo.es <mailto:elacunza at binovo.es>

	binovo.es <//binovo.es>

	Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun




	
youtube <https://www.youtube.com/user/CANALBINOVO/> 	
	linkedin <https://www.linkedin.com/company/37269706/>

More information about the pve-user mailing list