[pve-devel] [PATCH storage v4 00/12] Fix: #2124 zstd

[Alwin Antreich]

Zstandard (zstd) [0] is a data compression algorithm, in addition to
gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But
by default it has one compression and one writer thread.


Here some quick tests I made on my workstation. The files where placed
on a ram disk. And with dd filled from /dev/urandom and /dev/zero.

__Compression__
file size: 1073741824 bytes
  = urandom =         = zero =
  995ms 1073766414    328ms 98192    zstd -k
  732ms 1073766414    295ms 98192    zstd -k -T4
  906ms 1073791036    562ms 4894779  lzop -k
31992ms 1073915558   5594ms 1042087  gzip -k
30832ms 1074069541   5776ms 1171491  pigz -k -p 1
 7814ms 1074069541   1567ms 1171491  pigz -k -p 4

__Decompression__
file size: 1073741824 bytes
= urandom =   = zero =
   712ms      869ms  zstd -d
   685ms      872ms  zstd -k -d -T4
   841ms     2462ms  lzop -d
  5417ms     4754ms  gzip -k -d
  1248ms     3118ms  pigz -k -d -p 1
  1236ms     2379ms  pigz -k -d -p 4


And I used the same ramdisk to move a VM onto it and run a quick
backup/restore.

__vzdump backup__
INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1
INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4
INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo
INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip
INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4

__qmrestore__
progress 100% (read 34359738368 bytes, duration 36 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4

progress 100% (read 34359738368 bytes, duration 38 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo

progress 100% (read 34359738368 bytes, duration 175 sec)
total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4


v3 -> v4:
    * fixed styling issues discovered by f.ebner (thanks)
    * incorporated tests of d.jaeger into patches
    * added fixes discovered by tests (f.ebner thanks)

v2 -> v3:
    * split archive_info into decompressor_info and archive_info
    * "compact" regex pattern is now a constant and used in
      multiple modules
    * added tests for regex matching
    * bug fix for ctime of backup files

v1 -> v2:
    * factored out the decompressor info first, as Thomas suggested
    * made the regex pattern of backup files more compact, easier to
      read (hopefully)
    * less code changes for container restores

Thanks for any comment or suggestion in advance.

[0] https://facebook.github.io/zstd/

Alwin Antreich (12):
__pve-storage__
  storage: test: split archive format/compressor
  storage: replace build-in stat with File::stat
  test: parse_volname
  test: list_volumes
  Fix: backup: ctime was from stat not file name
  test: path_to_volume_id
  Fix: path_to_volume_id returned wrong content
  Fix: add missing snippets subdir
  backup: compact regex for backup file filter
  Fix: #2124 storage: add zstd support
  test: get_subdir
  test: filesystem_path

 test/Makefile                  |   5 +-
 PVE/Diskmanage.pm              |   9 +-
 PVE/Storage.pm                 |  91 ++++--
 PVE/Storage/Plugin.pm          |  47 ++-
 test/archive_info_test.pm      | 127 ++++++++
 test/filesystem_path_test.pm   |  91 ++++++
 test/get_subdir_test.pm        |  44 +++
 test/list_volumes_test.pm      | 537 +++++++++++++++++++++++++++++++++
 test/parse_volname_test.pm     | 253 ++++++++++++++++
 test/path_to_volume_id_test.pm | 274 +++++++++++++++++
 test/run_plugin_tests.pl       |  18 ++
 11 files changed, 1440 insertions(+), 56 deletions(-)
 create mode 100644 test/archive_info_test.pm
 create mode 100644 test/filesystem_path_test.pm
 create mode 100644 test/get_subdir_test.pm
 create mode 100644 test/list_volumes_test.pm
 create mode 100644 test/parse_volname_test.pm
 create mode 100644 test/path_to_volume_id_test.pm
 create mode 100755 test/run_plugin_tests.pl


__qemu-server__
  restore: replace archive format/compression
  Fix #2124: Add support for zstd

 PVE/QemuServer.pm | 38 +++++++-------------------------------
 1 file changed, 7 insertions(+), 31 deletions(-)


__pve-container__
  Fix: #2124 add zstd

 src/PVE/LXC/Create.pm | 1 +
 1 file changed, 1 insertion(+)


__guest_common__
  Fix: #2124 add zstd support

 PVE/VZDump/Common.pm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)


__pve-manager__
  Fix #2124: Add support for zstd

 PVE/VZDump.pm                            | 11 +++++++++--
 debian/control                           |  1 +
 www/manager6/form/CompressionSelector.js |  3 ++-
 3 files changed, 12 insertions(+), 3 deletions(-)
-- 
2.20.1