[pve-devel] applied-series: [PATCH storage v5 00/17] Fix: #2124 zstd

[Fabian Grünbichler]

with an additional commit adding '--rsyncable'.

thanks!

On April 28, 2020 3:58 pm, Alwin Antreich wrote:
> Zstandard (zstd) [0] is a data compression algorithm, in addition to
> gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But
> by default it has one compression and one writer thread.
> 
> 
> Here some quick tests I made on my workstation. The files where placed
> on a ram disk. And with dd filled from /dev/urandom and /dev/zero.
> 
> __Compression__
> file size: 1073741824 bytes
>   = urandom =         = zero =
>   995ms 1073766414    328ms 98192    zstd -k
>   732ms 1073766414    295ms 98192    zstd -k -T4
>   906ms 1073791036    562ms 4894779  lzop -k
> 31992ms 1073915558   5594ms 1042087  gzip -k
> 30832ms 1074069541   5776ms 1171491  pigz -k -p 1
>  7814ms 1074069541   1567ms 1171491  pigz -k -p 4
> 
> __Decompression__
> file size: 1073741824 bytes
> = urandom =   = zero =
>    712ms      869ms  zstd -d
>    685ms      872ms  zstd -k -d -T4
>    841ms     2462ms  lzop -d
>   5417ms     4754ms  gzip -k -d
>   1248ms     3118ms  pigz -k -d -p 1
>   1236ms     2379ms  pigz -k -d -p 4
> 
> 
> And I used the same ramdisk to move a VM onto it and run a quick
> backup/restore.
> 
> __vzdump backup__
> INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1
> INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4
> INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo
> INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip
> INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4
> 
> __qmrestore__
> progress 100% (read 34359738368 bytes, duration 36 sec)
> total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4
> 
> progress 100% (read 34359738368 bytes, duration 38 sec)
> total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo
> 
> progress 100% (read 34359738368 bytes, duration 175 sec)
> total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4
> 
> 
> v4 -> v5:
>     * fixup, use File::stat directly without overwriting CORE::stat,
>       thanks Dietmar for pointing this out
>       https://pve.proxmox.com/pipermail/pve-devel/2020-April/043134.html
>     * rebase to current master
> 
> v3 -> v4:
>     * fixed styling issues discovered by f.ebner (thanks)
>     * incorporated tests of d.jaeger into patches
>     * added fixes discovered by tests (f.ebner thanks)
> 
> v2 -> v3:
>     * split archive_info into decompressor_info and archive_info
>     * "compact" regex pattern is now a constant and used in
>       multiple modules
>     * added tests for regex matching
>     * bug fix for ctime of backup files
> 
> v1 -> v2:
>     * factored out the decompressor info first, as Thomas suggested
>     * made the regex pattern of backup files more compact, easier to
>       read (hopefully)
>     * less code changes for container restores
> 
> Thanks for any comment or suggestion in advance.
> 
> [0] https://facebook.github.io/zstd/
> 
> Alwin Antreich (17):
> __pve-storage__
>   storage: test: split archive format/compressor
>   storage: replace build-in stat with File::stat
>   test: parse_volname
>   test: list_volumes
>   Fix: backup: ctime was from stat not file name
>   test: path_to_volume_id
>   Fix: path_to_volume_id returned wrong content
>   Fix: add missing snippets subdir
>   backup: compact regex for backup file filter
>   Fix: #2124 storage: add zstd support
>   test: get_subdir
>   test: filesystem_path
> 
>  test/Makefile                  |   5 +-
>  PVE/Diskmanage.pm              |   9 +-
>  PVE/Storage.pm                 |  91 ++++--
>  PVE/Storage/Plugin.pm          |  47 ++-
>  test/archive_info_test.pm      | 127 ++++++++
>  test/filesystem_path_test.pm   |  91 ++++++
>  test/get_subdir_test.pm        |  44 +++
>  test/list_volumes_test.pm      | 537 +++++++++++++++++++++++++++++++++
>  test/parse_volname_test.pm     | 253 ++++++++++++++++
>  test/path_to_volume_id_test.pm | 274 +++++++++++++++++
>  test/run_plugin_tests.pl       |  18 ++
>  11 files changed, 1440 insertions(+), 56 deletions(-)
>  create mode 100644 test/archive_info_test.pm
>  create mode 100644 test/filesystem_path_test.pm
>  create mode 100644 test/get_subdir_test.pm
>  create mode 100644 test/list_volumes_test.pm
>  create mode 100644 test/parse_volname_test.pm
>  create mode 100644 test/path_to_volume_id_test.pm
>  create mode 100755 test/run_plugin_tests.pl
> 
> 
> __guest_common__
>   Fix: #2124 add zstd support
> 
>  PVE/VZDump/Common.pm | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> 
> __qemu-server__
>   restore: replace archive format/compression
>   Fix #2124: Add support for zstd
> 
>  PVE/QemuServer.pm | 38 +++++++-------------------------------
>  1 file changed, 7 insertions(+), 31 deletions(-)
> 
> 
> __pve-container__
>   Fix: #2124 add zstd
> 
>  src/PVE/LXC/Create.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> 
> __pve-manager__
>   Fix #2124: Add support for zstd
> 
>  PVE/VZDump.pm                            | 11 +++++++++--
>  debian/control                           |  1 +
>  www/manager6/form/CompressionSelector.js |  3 ++-
>  3 files changed, 12 insertions(+), 3 deletions(-)
> -- 
> 2.20.1
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
>