[pve-devel] [PATCH v3 0/13] Fix: #2124 zstd

[Alwin Antreich]

Zstandard (zstd) [0] is a data compression algorithm, in addition to
gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But
by default it has one compression and one writer thread.


Here some quick tests I made on my workstation. The files where placed
on a ram disk. And with dd filled from /dev/urandom and /dev/zero.

__Compression__
file size: 1073741824 bytes
  = urandom =         = zero =
  995ms 1073766414    328ms 98192    zstd -k
  732ms 1073766414    295ms 98192    zstd -k -T4
  906ms 1073791036    562ms 4894779  lzop -k
31992ms 1073915558   5594ms 1042087  gzip -k
30832ms 1074069541   5776ms 1171491  pigz -k -p 1
 7814ms 1074069541   1567ms 1171491  pigz -k -p 4

__Decompression__
file size: 1073741824 bytes
= urandom =   = zero =
   712ms      869ms  zstd -d
   685ms      872ms  zstd -k -d -T4
   841ms     2462ms  lzop -d
  5417ms     4754ms  gzip -k -d
  1248ms     3118ms  pigz -k -d -p 1
  1236ms     2379ms  pigz -k -d -p 4


And I used the same ramdisk to move a VM onto it and run a quick
backup/restore.

__vzdump backup__
INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1
INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4
INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo
INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip
INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4

__qmrestore__
progress 100% (read 34359738368 bytes, duration 36 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4

progress 100% (read 34359738368 bytes, duration 38 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo

progress 100% (read 34359738368 bytes, duration 175 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4


v2 -> v3:
    * split archive_info into decompressor_info and archive_info
    * "compact" regex pattern is now a constant and used in
      multiple modules
    * added tests for regex matching
    * bug fix for ctime of backup files

v1 -> v2:
    * factored out the decompressor info first, as Thomas suggested
    * made the regex pattern of backup files more compact, easier to
      read (hopefully)
    * less code changes for container restores

Thanks for any comment or suggestion in advance.

[0] https://facebook.github.io/zstd/

Alwin Antreich (13):
__pve-storage__
  storage: test: split archive format/compressor
  test: parse_volname
  test: list_volumes
  Fix: backup: ctime taken from stat not file name
  test: path_to_volume_id
  backup: more compact regex for backup file filter
  Fix: #2124 storage: add zstd support

 test/Makefile                  |   5 +-
 PVE/Storage.pm                 |  85 ++++++---
 PVE/Storage/Plugin.pm          |   9 +-
 test/run_parser_tests.pl       |  17 ++
 test/test_archive_info.pm      |  57 ++++++
 test/test_list_volumes.pm      | 313 +++++++++++++++++++++++++++++++++
 test/test_parse_volname.pm     |  98 +++++++++++
 test/test_path_to_volume_id.pm | 104 +++++++++++
 8 files changed, 661 insertions(+), 27 deletions(-)
 create mode 100755 test/run_parser_tests.pl
 create mode 100644 test/test_archive_info.pm
 create mode 100644 test/test_list_volumes.pm
 create mode 100644 test/test_parse_volname.pm
 create mode 100644 test/test_path_to_volume_id.pm

__pve-container__
  Fix: #2124 add zstd

 src/PVE/LXC/Create.pm | 1 +
 1 file changed, 1 insertion(+)

__qemu-server__
  restore: replace archive format/compression regex
  Fix #2124: Add support for zstd

 PVE/QemuServer.pm | 38 +++++++-------------------------------
 1 file changed, 7 insertions(+), 31 deletions(-)

__pve-manager__
  Fix #2124: Add support for zstd

 PVE/VZDump.pm                            | 11 +++++++++--
 debian/control                           |  1 +
 www/manager6/form/CompressionSelector.js |  3 ++-
 3 files changed, 12 insertions(+), 3 deletions(-)

__pve-guest-common__
  Fix: #2124 add zstd support

 PVE/VZDump/Common.pm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
-- 
2.20.1