[pbs-devel] [PATCH v2 proxmox-backup 0/9] drop catalog encoding for split pxar archives

[Christian Ebner]

The catalog stores metadata needed for fast lookups via e.g. the file
browser for single file restore.
Since it is now possible to create snapshots with split pxar archives,
the same metadata can be accessed in a reasonable performat way also via
the metadata archive, effectively making the catalog unneeded duplicate
data for this case.

This patch series therefore allows to lookup directory entries where
needed via the metadata archive instead of the catalog and conditionally
drops the catalog encoding for snapshots using split archive encoding.

The patches adapt not only the `catalog` api endpoint, used by the
Proxmox Backup WebUI to access the snapshots contents but also the
proxomox-file-restore, which is used by the Proxmox Virtual Environments
PBS storage plugin to access and restore contents.

Tested that regular backups can still be inspected and restored and that
the same functionality is still available for split archive backups,
without now having a catalog encoded.

Functionally was tested with the following path series applied to PVE:
https://lists.proxmox.com/pipermail/pve-devel/2024-May/064024.html

Please note: For directories with may entries a noticeable performance
drop is present, which might be improved upon by optimizing decoding.

Changes since version 1:
- Do not base64 encode archive name
- Split off file restore archive name listing

The rest of the comments will be addressed as followups as discussed
off-list.

Christian Ebner (9):
  api: datastore: factor out path decoding for catalog
  api: datastore: move reusable code out of thread
  client: tools: add helper to lookup `ArchiveEntry`s via pxar
  api: datastore: conditional lookup for catalog endpoint
  api: datastore: add optional archive-name to file-restore
  file-restore: never list ppxar as archive
  file-restore: fallback to mpxar if catalog not present
  www: content: lookup via metadata archive instead of catalog
  client: backup: conditionally write catalog for file level backups

 pbs-client/src/pxar_backup_stream.rs |  11 ++-
 pbs-client/src/tools/mod.rs          |  84 +++++++++++++++++++
 proxmox-backup-client/src/main.rs    |  21 +++--
 proxmox-file-restore/src/main.rs     |  72 ++++++++++++----
 src/api2/admin/datastore.rs          | 121 ++++++++++++++++++---------
 www/datastore/Content.js             |   3 +
 6 files changed, 243 insertions(+), 69 deletions(-)

-- 
2.39.2

>From 8dffaab3ce2c20b0c376d6c0dc8b3d561499fa24 Mon Sep 17 00:00:00 2001
From: Christian Ebner <c.ebner at proxmox.com>
Date: Fri, 7 Jun 2024 11:25:09 +0200

Christian Ebner (8):
  api: datastore: factor out path decoding for catalog
  api: datastore: move reusable code out of thread
  client: tools: add helper to lookup `ArchiveEntry`s via pxar
  api: datastore: conditional lookup for catalog endpoint
  api: datastore: add optional archive-name to file-restore
  file-restore: fallback to mpxar if catalog not present
  www: content: lookup via metadata archive instead of catalog
  client: backup: conditionally write catalog for file level backups

 pbs-client/src/pxar_backup_stream.rs |  11 ++-
 pbs-client/src/tools/mod.rs          |  84 +++++++++++++++++
 proxmox-backup-client/src/main.rs    |  21 +++--
 proxmox-file-restore/src/main.rs     |  72 +++++++++++----
 src/api2/admin/datastore.rs          | 129 ++++++++++++++++++---------
 www/datastore/Content.js             |   3 +
 6 files changed, 251 insertions(+), 69 deletions(-)

-- 
2.39.2