[pve-devel] [PATCH v3 0/13] Fix: #2124 zstd

[Fabian Ebner]

Hi,
series looks mostly good to me. Some comments on individual patches. 
Backup/restore seems to work, also still with the other compression 
formats. The tests for path_to_volume_id actually uncover a bug and a 
missing feature in the implementation in Storage.pm, which should be fixed.

For the patches that won't change (much):
Reviewed-By: Fabian Ebner <f.ebner at proxmox.com>
Tested-By: Fabian Ebner <f.ebner at proxmox.com>

For a potential follow-up, I feel like the parsing of the backup 
filename might need its own method, rather than have all those pattern 
matchings in different places.

On 08.04.20 12:25, Alwin Antreich wrote:
> Zstandard (zstd) [0] is a data compression algorithm, in addition to
> gzip, lzo for our backup/restore. It can utilize multiple core CPUs. But
> by default it has one compression and one writer thread.
> 
> 
> Here some quick tests I made on my workstation. The files where placed
> on a ram disk. And with dd filled from /dev/urandom and /dev/zero.
> 
> __Compression__
> file size: 1073741824 bytes
>    = urandom =         = zero =
>    995ms 1073766414    328ms 98192    zstd -k
>    732ms 1073766414    295ms 98192    zstd -k -T4
>    906ms 1073791036    562ms 4894779  lzop -k
> 31992ms 1073915558   5594ms 1042087  gzip -k
> 30832ms 1074069541   5776ms 1171491  pigz -k -p 1
>   7814ms 1074069541   1567ms 1171491  pigz -k -p 4
> 
> __Decompression__
> file size: 1073741824 bytes
> = urandom =   = zero =
>     712ms      869ms  zstd -d
>     685ms      872ms  zstd -k -d -T4
>     841ms     2462ms  lzop -d
>    5417ms     4754ms  gzip -k -d
>    1248ms     3118ms  pigz -k -d -p 1
>    1236ms     2379ms  pigz -k -d -p 4
> 
> 
> And I used the same ramdisk to move a VM onto it and run a quick
> backup/restore.
> 
> __vzdump backup__
> INFO: transferred 34359 MB in 69 seconds (497 MB/s) zstd -T1
> INFO: transferred 34359 MB in 37 seconds (928 MB/s) zstd -T4
> INFO: transferred 34359 MB in 51 seconds (673 MB/s) lzo
> INFO: transferred 34359 MB in 1083 seconds (31 MB/s) gzip
> INFO: transferred 34359 MB in 241 seconds (142 MB/s) pigz -n 4
> 
> __qmrestore__
> progress 100% (read 34359738368 bytes, duration 36 sec)
> total bytes read 34359738368, sparse bytes 8005484544 (23.3%) zstd -d -T4
> 
> progress 100% (read 34359738368 bytes, duration 38 sec)
> total bytes read 34359738368, sparse bytes 8005484544 (23.3%) lzo
> 
> progress 100% (read 34359738368 bytes, duration 175 sec)
> total bytes read 34359738368, sparse bytes 8005484544 (23.3%) pigz -n 4
> 
> 
> v2 -> v3:
>      * split archive_info into decompressor_info and archive_info
>      * "compact" regex pattern is now a constant and used in
>        multiple modules
>      * added tests for regex matching
>      * bug fix for ctime of backup files
> 
> v1 -> v2:
>      * factored out the decompressor info first, as Thomas suggested
>      * made the regex pattern of backup files more compact, easier to
>        read (hopefully)
>      * less code changes for container restores
> 
> Thanks for any comment or suggestion in advance.
> 
> [0] https://facebook.github.io/zstd/
> 
> Alwin Antreich (13):
> __pve-storage__
>    storage: test: split archive format/compressor
>    test: parse_volname
>    test: list_volumes
>    Fix: backup: ctime taken from stat not file name
>    test: path_to_volume_id
>    backup: more compact regex for backup file filter
>    Fix: #2124 storage: add zstd support
> 
>   test/Makefile                  |   5 +-
>   PVE/Storage.pm                 |  85 ++++++---
>   PVE/Storage/Plugin.pm          |   9 +-
>   test/run_parser_tests.pl       |  17 ++
>   test/test_archive_info.pm      |  57 ++++++
>   test/test_list_volumes.pm      | 313 +++++++++++++++++++++++++++++++++
>   test/test_parse_volname.pm     |  98 +++++++++++
>   test/test_path_to_volume_id.pm | 104 +++++++++++
>   8 files changed, 661 insertions(+), 27 deletions(-)
>   create mode 100755 test/run_parser_tests.pl
>   create mode 100644 test/test_archive_info.pm
>   create mode 100644 test/test_list_volumes.pm
>   create mode 100644 test/test_parse_volname.pm
>   create mode 100644 test/test_path_to_volume_id.pm
> 
> __pve-container__
>    Fix: #2124 add zstd
> 
>   src/PVE/LXC/Create.pm | 1 +
>   1 file changed, 1 insertion(+)
> 
> __qemu-server__
>    restore: replace archive format/compression regex
>    Fix #2124: Add support for zstd
> 
>   PVE/QemuServer.pm | 38 +++++++-------------------------------
>   1 file changed, 7 insertions(+), 31 deletions(-)
> 
> __pve-manager__
>    Fix #2124: Add support for zstd
> 
>   PVE/VZDump.pm                            | 11 +++++++++--
>   debian/control                           |  1 +
>   www/manager6/form/CompressionSelector.js |  3 ++-
>   3 files changed, 12 insertions(+), 3 deletions(-)
> 
> __pve-guest-common__
>    Fix: #2124 add zstd support
> 
>   PVE/VZDump/Common.pm | 9 ++++++++-
>   1 file changed, 8 insertions(+), 1 deletion(-)
>