[pbs-devel] applied: [PATCH proxmox-backup] garbage collection: fix: account for created/deleted index files

[Fabian Grünbichler]

thanks!

On April 14, 2025 11:24 am, Christian Ebner wrote:
> Since commit 74361da8 ("garbage collection: generate index file list
> via datastore iterators") not only snapshots present at the start of
> the garbage collection run are considered for marking, but also newly
> added ones. Take these into account by adapting the total index file
> counter used for the progress output.
> 
> Further, correctly take into account also index files which have been
> pruned during GC, therefore present in the list of still to process
> index files but never encountered by the datastore iterators. These
> would otherwise be interpreted incorrectly as strange paths and logged
> accordingly, causing confusion as reported in the community forum [0].
> 
> Fixes: https://forum.proxmox.com/threads/164968/
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/datastore.rs | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index fbd9eddba..aa38e2ac1 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -1129,7 +1129,7 @@ impl DataStore {
>          // the detected index files not following the iterators logic.
>  
>          let mut unprocessed_index_list = self.list_index_files()?;
> -        let index_count = unprocessed_index_list.len();
> +        let mut index_count = unprocessed_index_list.len();
>  
>          let mut chunk_lru_cache = LruCache::new(cache_capacity);
>          let mut processed_index_files = 0;
> @@ -1156,7 +1156,10 @@ impl DataStore {
>  
>                          let index = match self.open_index_reader(&path)? {
>                              Some(index) => index,
> -                            None => continue,
> +                            None => {
> +                                unprocessed_index_list.remove(&path);
> +                                continue;
> +                            }
>                          };
>                          self.index_mark_used_chunks(
>                              index,
> @@ -1166,7 +1169,10 @@ impl DataStore {
>                              worker,
>                          )?;
>  
> -                        unprocessed_index_list.remove(&path);
> +                        if !unprocessed_index_list.remove(&path) {
> +                            info!("Encountered new index file '{path:?}', increment total index file count");
> +                            index_count += 1;
> +                        }
>  
>                          let percentage = (processed_index_files + 1) * 100 / index_count;
>                          if percentage > last_percentage {
> @@ -1182,18 +1188,22 @@ impl DataStore {
>              }
>          }
>  
> -        let strange_paths_count = unprocessed_index_list.len();
> -        if strange_paths_count > 0 {
> -            warn!("found {strange_paths_count} index files outside of expected directory scheme");
> -        }
> +        let mut strange_paths_count = unprocessed_index_list.len();
>          for path in unprocessed_index_list {
>              let index = match self.open_index_reader(&path)? {
>                  Some(index) => index,
> -                None => continue,
> +                None => {
> +                    // do not count vanished (pruned) backup snapshots as strange paths.
> +                    strange_paths_count -= 1;
> +                    continue;
> +                }
>              };
>              self.index_mark_used_chunks(index, &path, &mut chunk_lru_cache, status, worker)?;
>              warn!("Marked chunks for unexpected index file at '{path:?}'");
>          }
> +        if strange_paths_count > 0 {
> +            warn!("Found {strange_paths_count} index files outside of expected directory scheme");
> +        }
>  
>          Ok(())
>      }
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
>