[PVE-User] Backup recommendation

[Amand Tihon]

Hi all,

Disclaimer: I understand that my question is not strictly related to Proxmox. 
Feel free to ignore if you think it's too much off-topic, or reply privately.

Would anyone have recommendations for good backup + DRP setup and strategy in 
a Proxmox cluster? The snapshot functionnality readily available through 
Proxmox is of great help for DRP, but makes things pretty complex when you 
only need to restore some files. Also, it doesn't really account for files or 
database integrity, unless pre- and post-backup scripts can be started on the 
guest.

FYI, we are not using any kind of virtualization yet. Our current backups are 
done using DataProtector and we use ISO images created with MondoRescue for 
DRP.

Just to get things started, here's our current idea:

- Install a bare-metal server with large local disks as a "backup server"
- Hook a tape library to that server
- Configure weekly (or even daily?) snapshots of the VM. Retention TBD.
- Setup some kind of rsync of each guest to the backup server (only rsync, not
  a hardlinks-based history rsync thing). Pre-, post-backup scripts go here.
- During the day, take a classic tape backup of the backup server: weekly full
  with daily differential.

Advantages I see:
- Ability to "close" the system during rsync backups, for data integrity,
- Have a full copy of the guests' filesystems with history on tape,
- The backup to tape is "offline", from the production point of view
- Tapes can be stored outside of the server room (safe from flooding or fire)

Disadvantages:
- Cost, since we more or less duplicate the backup infrastructure (server AND
  tapes), but reliable backups and DRP are absolutely mandatory.
- We don't know yet how rsync will perform on a filesystem with hundreds of
  thousands files...

Is there any software you would suggest ? Can this solution be improved ? 
Any comment would be greatly appreciated.

Thanks in advance,

Amand Tihon.