[pbs-devel] [PATCH proxmox-backup v2 1/1] docs: added section on ransomware

Fri Nov 25 11:40:00 CET 2022

some notes in-line (sorry if somewhat pedantic at times). also thanks to
stefan hanreich for helping me out.

generally: you use "Proxmox Backup Server" a lot, maybe try to remove
some occurrence or shorten them to PBS.

ps: sorry if you got this twice, forgot to hit reply-all ^^'
pps: sorry for the mess, this should be properly formatted now hopefully...

On 11/24/22 15:29, Noel Ullreich wrote:
> Added a section on ransomware. This includes a bulletpoint in the
> main features section and a section in the backup storage section.
> The latter section lists mitigation resources in pbs as well as best
> practices.
> 
> Updated capitalization to be consistent in main features. Imo, since
> these are bulletpoints and not headings, they should be in lowercase
> 
> Signed-off-by: Noel Ullreich <n.ullreich at proxmox.com>
> ---
> 
>  changes since v1:
>  * squashed multiple commits into one
>  * added link in main features bulletpoint to the ransomware section
>  * restructured parts of the ransomware section
>  * fixed technical errors regarding reading checksum
>  * fixed my gitconfig ;)
> 
>  docs/introduction.rst | 14 +++++----
>  docs/storage.rst      | 70 +++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 79 insertions(+), 5 deletions(-)
> 
> diff --git a/docs/introduction.rst b/docs/introduction.rst
> index 369e7e29..e6598171 100644
> --- a/docs/introduction.rst
> +++ b/docs/introduction.rst
> @@ -58,10 +58,10 @@ Main Features
>  :Incremental backups: Changes between backups are typically low. Reading and
>     sending only the delta reduces the storage and network impact of backups.
>  
> -:Data Integrity: The built-in `SHA-256`_ checksum algorithm ensures accuracy and
> +:Data integrity: The built-in `SHA-256`_ checksum algorithm ensures accuracy and
>     consistency in your backups.
>  
> -:Remote Sync: It is possible to efficiently synchronize data to remote
> +:Remote sync: It is possible to efficiently synchronize data to remote
>     sites. Only deltas containing new data are transferred.
>  
>  :Compression: The ultra-fast Zstandard_ compression is able to compress
> @@ -76,16 +76,20 @@ Main Features
>     provides extensive support for backing up to tape and managing tape
>     libraries.
>  
> +:Ransomware protection: :ref:`Protect your critical data from ransomware attacks <ransomware_protection>` with

this line does not properly wrap at 80 columns

> +   Proxmox Backup Server's fine-grained access control, data integrity
> +   verification, and off-site backup through remote sync and tape backup.
> +
>  :Web interface: Manage the Proxmox Backup Server with the integrated, web-based
>     user interface.
>  
> -:Open Source: No secrets. Proxmox Backup Server is free and open-source
> +:Open source: No secrets. Proxmox Backup Server is free and open-source
>     software. The source code is licensed under AGPL, v3.
>  
> -:No Limits: Proxmox Backup Server has no artificial limits for backup storage or
> +:No limits: Proxmox Backup Server has no artificial limits for backup storage or
>     backup-clients.
>  
> -:Enterprise Support: Proxmox Server Solutions GmbH offers enterprise support in
> +:Enterprise support: Proxmox Server Solutions GmbH offers enterprise support in
>     the form of `Proxmox Backup Server Subscription Plans
>     <https://www.proxmox.com/en/proxmox-backup-server/pricing>`_. Users at every
>     subscription level get access to the Proxmox Backup :ref:`Enterprise
> diff --git a/docs/storage.rst b/docs/storage.rst
> index c4e44c72..00c5e519 100644
> --- a/docs/storage.rst
> +++ b/docs/storage.rst
> @@ -374,3 +374,73 @@ with a comma, like this:
>  .. code-block:: console
>  
>    # proxmox-backup-manager datastore update <storename> --tuning 'sync-level=filesystem,chunk-order=none'
> +
> +.. _ransomware_protection:
> +
> +Ransomware Protection
> +---------------------
> +
> +Prevention by Proxmox Backup Server
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +`Ransomware <https://en.wikipedia.org/wiki/Ransomware>`_ is a type of malware
> +that encrypts files until a ransom is paid. Proxmox Backup Server includes
> +features to mitigate ransomware attacks by offering easy restoration from backups.
> +
> +As a best practice, you should keep multiple backups, including outside of your
> +network and on different media. Proxmox Backup Server provides the tools to do 
> +both.

this either wraps weirdly here or is missing a new line

> +It is possible to create :ref:`remote sync jobs <backup_remote>`; by setting up
> +an Proxmox Backup Server instance off-site and, from there, pulling a datastore.

a Proxmox Backup Server. Personal preference would be something like:

By setting up a remote Proxmox Backup Server you can take advantage of
the sync job feature and create off-site copies of your backups.

> +This is recommended since offsite Proxmox Backup Server instances will not be 

comma: recommended, since

> +infected by the ransomware in your local network.

imo more accurate "are less likely to be infected". there still must be
some kind of network connection between the two syncing instances
obviously and depending on how that is done, the off-site nature of the
secondary PBS may not actually mitigate that much.

> +It it also possible to create :ref:`tape backups <tape_backup>` as a second 

It is

> +storage medium. This way you get an additional copy of your data which can easily
> +be moved off-site.
> +
> +Proxmox Backup Server does not rewrite data for existing blocks. This means that
> +a compromised Proxmox VE host, or any other compromised system using
> +the client to back up data, cannot corrupt existing backups.
> +
> +Furthermore, comprehensive :ref:`user management <user_mgmt>` is offered in

-in +by

> +Proxmox Backup Server. By limiting a sync user's or an access token's right to
> +only write backups, not delete them, compromised Proxmox VEs cannot delete

compromised clients? since you could afaik also use the
proxmox-backup-client to do that (or the api).

> +existing backups. Following this best practice, backup pruning should be done
> +by the Proxmox Backup Server using prune jobs.
> +
> +Proxmox Backup Servers can still be compromised, even when taking precautions.
> +In case of a compromised Proxmox Backup server instance, encrypted data on the

Server not server. also maybe try to use Proxmox Backup Server less in
general. that's a lot of repetition here. Maybe:

While your Proxmox Backup Server can still be compromised, it is not
possible to accidentally restore an encrypted backup and cause further
problems this way. If a ransomware encrypts part of a backup, the
SHA-256 checksums of the backups will not match the previously recorded
ones anymore. Hence, restoring the backup will fail.

> +Proxmox Backup Server can no longer be verified, since the SHA-256 checksum of 
> +the chunks can no longer be read. This should alert you that your backups are
> +corrupted.
> +
> +To detect ransomware inside a compromised guest, it is recommended to frequently
> +test restoring and booting backups. Make sure to restore to a new guest and
> +not to overwrite your current guest. In the case of many backed-up guests, it is
> +recommended to automate this restore testing or, if this is not possible, to
> +restore random samples from the backups.
> +
> +

Not sure about this paragraph, since it will probably be noticeable very
soon due to the server malfunctioning anyway. I would go about this from
a slightly different angle maybe:

In order to be able to react quickly in case of a ransomware attack, it
is recommended to regularly test restoring from your backups. Restoring
many guests at once can be cumbersome, which is why it is advisable to
automate this task and verify that your automated process works. Making
backups is only one part of the equation, being able to restore them is
equally as important. Verifying that your backup and restore process
works ensures that you are able to react quickly in case of an emergency
and keeps disruption of your services to a minimum.

Something like that maybe? this is obviously just a draft and could be
fleshed out more..

> +
> +Other Prevention Methods and Best Practices
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +It is recommended to take additional security measures, apart form the ones offered

typo: form -> from

> +by Proxmox Backup Server. These recommendations include, but are not limited to: 
> +
> +* Keeping the firmware and software up-to-date to patch exploits and
> +  vulnerabilities (such as
> +  `spectre <https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)>`_ or
> +  `meltdown <https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)>`_).

maybe too nitpicky: you aren't wrong here, but afaik neither spectre or
meltdown could actually be used directly to carry out ransomware
attacks. maybe EternalBlue (yes windows based) would be a better
example. microsoft published a patch ~1 month before it was published
and exploited for WannaCry.

if you want to stick with spectre and meltdown: i think you need to
capitalize them.

> +* Following safe and secure network practices, for example using logging and
> +  monitoring tools and setting up VLANs.
> +* Making plenty of backups using the
> +  `3-2-1 rule <https://en.wikipedia.org/wiki/Backup#Storage>`_: creating
> +  3 backups on 2 storage media, of which 1 copy is kept off-site.
> +* Retaining backups for a few months. Proxmox Backup Server allows for flexible
> +  backup retention, since some ransomware might only be encrypted weeks after

maybe:

[..] retention. Since some ransomware might lay dormant a couple of days
or weeks before starting to encrypt data, it is possible that all
remaining backups are already compromised. Thus, it is important to keep
at least a few older backups.

> +  infecting your system or you might only notice an infection a few weeks later.
> +
> +For more information on how to avoid ransomware attacks and what to do in case
> +of a ransomware infection, see 
> +`Cisa <https://www.cisa.gov/stopransomware/ransomware-guide>`_.

- see Cisa + consult the guide by CISA