[pbs-devel] [PATCH docs 2/2] Restructure docs (more first level headings)
Dylan Whyte
d.whyte at proxmox.com
Fri Oct 2 16:12:57 CEST 2020
This removes the "Backup Management" first level heading in the docs,
and either uses the sub headings contained within it as first level
headings, or groups previous sections logically under new headings.
The administration-guide.rst file is also removed. Its contents are
instead separated into various files, that relate to their respective
first level heading.
Signed-off-by: Dylan Whyte <d.whyte at proxmox.com>
---
docs/administration-guide.rst | 1480 ---------------------------------
docs/backup-client.rst | 712 ++++++++++++++++
docs/command-line-tools.rst | 2 +-
docs/index.rst | 11 +-
docs/maintenance.rst | 13 +
docs/managing-remotes.rst | 80 ++
docs/network-management.rst | 91 ++
docs/pve-integration.rst | 49 ++
docs/pxar-tool.rst | 5 +
docs/pxar/description.rst | 2 +-
docs/storage.rst | 244 ++++++
docs/terminology.rst | 118 +++
docs/user-management.rst | 186 +++++
13 files changed, 1510 insertions(+), 1483 deletions(-)
delete mode 100644 docs/administration-guide.rst
create mode 100644 docs/backup-client.rst
create mode 100644 docs/maintenance.rst
create mode 100644 docs/managing-remotes.rst
create mode 100644 docs/network-management.rst
create mode 100644 docs/pve-integration.rst
create mode 100644 docs/pxar-tool.rst
create mode 100644 docs/storage.rst
create mode 100644 docs/terminology.rst
create mode 100644 docs/user-management.rst
diff --git a/docs/administration-guide.rst b/docs/administration-guide.rst
deleted file mode 100644
index 9769f3a5..00000000
--- a/docs/administration-guide.rst
+++ /dev/null
@@ -1,1480 +0,0 @@
-Backup Management
-=================
-
-.. The administration guide.
- .. todo:: either add a bit more explanation or remove the previous sentence
-
-Terminology
------------
-
-Backup Content
-~~~~~~~~~~~~~~
-
-When doing deduplication, there are different strategies to get
-optimal results in terms of performance and/or deduplication rates.
-Depending on the type of data, it can be split into *fixed* or *variable*
-sized chunks.
-
-Fixed sized chunking requires minimal CPU power, and is used to
-backup virtual machine images.
-
-Variable sized chunking needs more CPU power, but is essential to get
-good deduplication rates for file archives.
-
-The Proxmox Backup Server supports both strategies.
-
-
-Image Archives: ``<name>.img``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-This is used for virtual machine images and other large binary
-data. Content is split into fixed-sized chunks.
-
-
-File Archives: ``<name>.pxar``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-.. see https://moinakg.wordpress.com/2013/06/22/high-performance-content-defined-chunking/
-
-A file archive stores a full directory tree. Content is stored using
-the :ref:`pxar-format`, split into variable-sized chunks. The format
-is optimized to achieve good deduplication rates.
-
-
-Binary Data (BLOBs)
-^^^^^^^^^^^^^^^^^^^
-
-This type is used to store smaller (< 16MB) binary data such as
-configuration files. Larger files should be stored as image archive.
-
-.. caution:: Please do not store all files as BLOBs. Instead, use the
- file archive to store whole directory trees.
-
-
-Catalog File: ``catalog.pcat1``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The catalog file is an index for file archives. It contains
-the list of files and is used to speed up search operations.
-
-
-The Manifest: ``index.json``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The manifest contains the list of all backup files, their
-sizes and checksums. It is used to verify the consistency of a
-backup.
-
-
-Backup Type
-~~~~~~~~~~~
-
-The backup server groups backups by *type*, where *type* is one of:
-
-``vm``
- This type is used for :term:`virtual machine`\ s. Typically
- consists of the virtual machine's configuration file and an image archive
- for each disk.
-
-``ct``
- This type is used for :term:`container`\ s. Consists of the container's
- configuration and a single file archive for the filesystem content.
-
-``host``
- This type is used for backups created from within the backed up machine.
- Typically this would be a physical host but could also be a virtual machine
- or container. Such backups may contain file and image archives, there are no restrictions in this regard.
-
-
-Backup ID
-~~~~~~~~~
-
-A unique ID. Usually the virtual machine or container ID. ``host``
-type backups normally use the hostname.
-
-
-Backup Time
-~~~~~~~~~~~
-
-The time when the backup was made.
-
-
-Backup Group
-~~~~~~~~~~~~
-
-The tuple ``<type>/<ID>`` is called a backup group. Such a group
-may contain one or more backup snapshots.
-
-
-Backup Snapshot
-~~~~~~~~~~~~~~~
-
-The triplet ``<type>/<ID>/<time>`` is called a backup snapshot. It
-uniquely identifies a specific backup within a datastore.
-
-.. code-block:: console
- :caption: Backup Snapshot Examples
-
- vm/104/2019-10-09T08:01:06Z
- host/elsa/2019-11-08T09:48:14Z
-
-As you can see, the time format is RFC3399_ with Coordinated
-Universal Time (UTC_, identified by the trailing *Z*).
-
-Backup Server Management
-------------------------
-
-The command line tool to configure and manage the backup server is called
-:command:`proxmox-backup-manager`.
-
-.. _datastore_intro:
-
-:term:`DataStore`
-~~~~~~~~~~~~~~~~~
-
-A datastore refers to a location at which backups are stored. The current
-implementation uses a directory inside a standard unix file system (``ext4``,
-``xfs`` or ``zfs``) to store the backup data.
-
-Datastores are identified by a simple *ID*. You can configure this
-when setting up the datastore. The configuration information for datastores
-is stored in the file ``/etc/proxmox-backup/datastore.cfg``.
-
-.. note:: The `File Layout`_ requires the file system to support at least *65538*
- subdirectories per directory. That number comes from the 2\ :sup:`16`
- pre-created chunk namespace directories, and the ``.`` and ``..`` default
- directory entries. This requirement excludes certain filesystems and
- filesystem configuration from being supported for a datastore. For example,
- ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
-
-Disk Management
-~~~~~~~~~~~~~~~
-
-.. image:: images/screenshots/pbs-gui-disks.png
- :align: right
- :alt: List of disks
-
-Proxmox Backup Server comes with a set of disk utilities, which are
-accessed using the ``disk`` subcommand. This subcommand allows you to initialize
-disks, create various filesystems, and get information about the disks.
-
-To view the disks connected to the system, navigate to **Administration ->
-Disks** in the web interface or use the ``list`` subcommand of
-``disk``:
-
-.. code-block:: console
-
- # proxmox-backup-manager disk list
- ┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
- │ name │ used │ gpt │ disk-type │ size │ model │ wearout │ status │
- ╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
- │ sda │ lvm │ 1 │ hdd │ 34359738368 │ QEMU_HARDDISK │ - │ passed │
- ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
- │ sdb │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
- ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
- │ sdc │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
- └──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘
-
-To initialize a disk with a new GPT, use the ``initialize`` subcommand:
-
-.. code-block:: console
-
- # proxmox-backup-manager disk initialize sdX
-
-.. image:: images/screenshots/pbs-gui-disks-dir-create.png
- :align: right
- :alt: Create a directory
-
-You can create an ``ext4`` or ``xfs`` filesystem on a disk using ``fs
-create``, or by navigating to **Administration -> Disks -> Directory** in the
-web interface and creating one from there. The following command creates an
-``ext4`` filesystem and passes the ``--add-datastore`` parameter, in order to
-automatically create a datastore on the disk (in this case ``sdd``). This will
-create a datastore at the location ``/mnt/datastore/store1``:
-
-.. code-block:: console
-
- # proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true
-
-.. image:: images/screenshots/pbs-gui-disks-zfs-create.png
- :align: right
- :alt: Create ZFS
-
-You can also create a ``zpool`` with various raid levels from **Administration
--> Disks -> Zpool** in the web interface, or by using ``zpool create``. The command
-below creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and
-mounts it on the root directory (default):
-
-.. code-block:: console
-
- # proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror
-
-.. note:: You can also pass the ``--add-datastore`` parameter here, to automatically
- create a datastore from the disk.
-
-You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
-filesystems and zpools respectively.
-
-Proxmox Backup Server uses the package smartmontools. This is a set of tools
-used to monitor and control the S.M.A.R.T. system for local hard disks. If a
-disk supports S.M.A.R.T. capability, and you have this enabled, you can
-display S.M.A.R.T. attributes from the web interface or by using the command:
-
-.. code-block:: console
-
- # proxmox-backup-manager disk smart-attributes sdX
-
-.. note:: This functionality may also be accessed directly through the use of
- the ``smartctl`` command, which comes as part of the smartmontools package
- (see ``man smartctl`` for more details).
-
-
-Datastore Configuration
-~~~~~~~~~~~~~~~~~~~~~~~
-
-.. image:: images/screenshots/pbs-gui-datastore.png
- :align: right
- :alt: Datastore Overview
-
-You can configure multiple datastores. Minimum one datastore needs to be
-configured. The datastore is identified by a simple *name* and points to a
-directory on the filesystem. Each datastore also has associated retention
-settings of how many backup snapshots for each interval of ``hourly``,
-``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
-number of backups to keep in that store. :ref:`Pruning <pruning>` and
-:ref:`garbage collection <garbage-collection>` can also be configured to run
-periodically based on a configured schedule (see :ref:`calendar-events`) per datastore.
-
-Creating a Datastore
-^^^^^^^^^^^^^^^^^^^^
-.. image:: images/screenshots/pbs-gui-datastore-create-general.png
- :align: right
- :alt: Create a datastore
-
-You can create a new datastore from the web GUI, by navigating to **Datastore** in
-the menu tree and clicking **Create**. Here:
-
-* *Name* refers to the name of the datastore
-* *Backing Path* is the path to the directory upon which you want to create the
- datastore
-* *GC Schedule* refers to the time and intervals at which garbage collection
- runs
-* *Prune Schedule* refers to the frequency at which pruning takes place
-* *Prune Options* set the amount of backups which you would like to keep (see :ref:`Pruning <pruning>`).
-
-Alternatively you can create a new datastore from the command line. The
-following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1`
-
-.. code-block:: console
-
- # proxmox-backup-manager datastore create store1 /backup/disk1/store1
-
-Managing Datastores
-^^^^^^^^^^^^^^^^^^^
-
-To list existing datastores from the command line run:
-
-.. code-block:: console
-
- # proxmox-backup-manager datastore list
- ┌────────┬──────────────────────┬─────────────────────────────┐
- │ name │ path │ comment │
- ╞════════╪══════════════════════╪═════════════════════════════╡
- │ store1 │ /backup/disk1/store1 │ This is my default storage. │
- └────────┴──────────────────────┴─────────────────────────────┘
-
-You can change the garbage collection and prune settings of a datastore, by
-editing the datastore from the GUI or by using the ``update`` subcommand. For
-example, the below command changes the garbage collection schedule using the
-``update`` subcommand and prints the properties of the datastore with the
-``show`` subcommand:
-
-.. code-block:: console
-
- # proxmox-backup-manager datastore update store1 --gc-schedule 'Tue 04:27'
- # proxmox-backup-manager datastore show store1
- ┌────────────────┬─────────────────────────────┐
- │ Name │ Value │
- ╞════════════════╪═════════════════════════════╡
- │ name │ store1 │
- ├────────────────┼─────────────────────────────┤
- │ path │ /backup/disk1/store1 │
- ├────────────────┼─────────────────────────────┤
- │ comment │ This is my default storage. │
- ├────────────────┼─────────────────────────────┤
- │ gc-schedule │ Tue 04:27 │
- ├────────────────┼─────────────────────────────┤
- │ keep-last │ 7 │
- ├────────────────┼─────────────────────────────┤
- │ prune-schedule │ daily │
- └────────────────┴─────────────────────────────┘
-
-Finally, it is possible to remove the datastore configuration:
-
-.. code-block:: console
-
- # proxmox-backup-manager datastore remove store1
-
-.. note:: The above command removes only the datastore configuration. It does
- not delete any data from the underlying directory.
-
-
-File Layout
-^^^^^^^^^^^
-
-After creating a datastore, the following default layout will appear:
-
-.. code-block:: console
-
- # ls -arilh /backup/disk1/store1
- 276493 -rw-r--r-- 1 backup backup 0 Jul 8 12:35 .lock
- 276490 drwxr-x--- 1 backup backup 1064960 Jul 8 12:35 .chunks
-
-`.lock` is an empty file used for process locking.
-
-The `.chunks` directory contains folders, starting from `0000` and taking hexadecimal values until `ffff`. These
-directories will store the chunked data after a backup operation has been executed.
-
-.. code-block:: console
-
- # ls -arilh /backup/disk1/store1/.chunks
- 545824 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 ffff
- 545823 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffe
- 415621 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffd
- 415620 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffc
- 353187 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffb
- 344995 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffa
- 144079 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff9
- 144078 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff8
- 144077 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff7
- ...
- 403180 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000c
- 403179 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000b
- 403177 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000a
- 402530 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0009
- 402513 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0008
- 402509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0007
- 276509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0006
- 276508 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0005
- 276507 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0004
- 276501 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0003
- 276499 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0002
- 276498 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0001
- 276494 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0000
- 276489 drwxr-xr-x 3 backup backup 4.0K Jul 8 12:35 ..
- 276490 drwxr-x--- 1 backup backup 1.1M Jul 8 12:35 .
-
-.. _user_mgmt:
-
-User Management
-~~~~~~~~~~~~~~~
-
-.. image:: images/screenshots/pbs-gui-user-management.png
- :align: right
- :alt: User management
-
-Proxmox Backup Server supports several authentication realms, and you need to
-choose the realm when you add a new user. Possible realms are:
-
-:pam: Linux PAM standard authentication. Use this if you want to
- authenticate as Linux system user (Users need to exist on the
- system).
-
-:pbs: Proxmox Backup Server realm. This type stores hashed passwords in
- ``/etc/proxmox-backup/shadow.json``.
-
-After installation, there is a single user ``root at pam``, which
-corresponds to the Unix superuser. User configuration information is stored in the file
-``/etc/proxmox-backup/user.cfg``. You can use the
-``proxmox-backup-manager`` command line tool to list or manipulate
-users:
-
-.. code-block:: console
-
- # proxmox-backup-manager user list
- ┌─────────────┬────────┬────────┬───────────┬──────────┬────────────────┬────────────────────┐
- │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
- ╞═════════════╪════════╪════════╪═══════════╪══════════╪════════════════╪════════════════════╡
- │ root at pam │ 1 │ │ │ │ │ Superuser │
- └─────────────┴────────┴────────┴───────────┴──────────┴────────────────┴────────────────────┘
-
-.. image:: images/screenshots/pbs-gui-user-management-add-user.png
- :align: right
- :alt: Add a new user
-
-The superuser has full administration rights on everything, so you
-normally want to add other users with less privileges. You can create a new
-user with the ``user create`` subcommand or through the web interface, under
-**Configuration -> User Management**. The ``create`` subcommand lets you specify
-many options like ``--email`` or ``--password``. You can update or change any
-user properties using the ``update`` subcommand later (**Edit** in the GUI):
-
-
-.. code-block:: console
-
- # proxmox-backup-manager user create john at pbs --email john at example.com
- # proxmox-backup-manager user update john at pbs --firstname John --lastname Smith
- # proxmox-backup-manager user update john at pbs --comment "An example user."
-
-.. todo:: Mention how to set password without passing plaintext password as cli argument.
-
-
-The resulting user list looks like this:
-
-.. code-block:: console
-
- # proxmox-backup-manager user list
- ┌──────────┬────────┬────────┬───────────┬──────────┬──────────────────┬──────────────────┐
- │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
- ╞══════════╪════════╪════════╪═══════════╪══════════╪══════════════════╪══════════════════╡
- │ john at pbs │ 1 │ │ John │ Smith │ john at example.com │ An example user. │
- ├──────────┼────────┼────────┼───────────┼──────────┼──────────────────┼──────────────────┤
- │ root at pam │ 1 │ │ │ │ │ Superuser │
- └──────────┴────────┴────────┴───────────┴──────────┴──────────────────┴──────────────────┘
-
-Newly created users do not have any permissions. Please read the next
-section to learn how to set access permissions.
-
-If you want to disable a user account, you can do that by setting ``--enable`` to ``0``
-
-.. code-block:: console
-
- # proxmox-backup-manager user update john at pbs --enable 0
-
-Or completely remove the user with:
-
-.. code-block:: console
-
- # proxmox-backup-manager user remove john at pbs
-
-
-.. _user_acl:
-
-Access Control
-~~~~~~~~~~~~~~
-
-By default new users do not have any permission. Instead you need to
-specify what is allowed and what is not. You can do this by assigning
-roles to users on specific objects like datastores or remotes. The
-following roles exist:
-
-**NoAccess**
- Disable Access - nothing is allowed.
-
-**Admin**
- Can do anything.
-
-**Audit**
- Can view things, but is not allowed to change settings.
-
-**DatastoreAdmin**
- Can do anything on datastores.
-
-**DatastoreAudit**
- Can view datastore settings and list content. But
- is not allowed to read the actual data.
-
-**DatastoreReader**
- Can Inspect datastore content and can do restores.
-
-**DatastoreBackup**
- Can backup and restore owned backups.
-
-**DatastorePowerUser**
- Can backup, restore, and prune owned backups.
-
-**RemoteAdmin**
- Can do anything on remotes.
-
-**RemoteAudit**
- Can view remote settings.
-
-**RemoteSyncOperator**
- Is allowed to read data from a remote.
-
-.. image:: images/screenshots/pbs-gui-permissions-add.png
- :align: right
- :alt: Add permissions for user
-
-Access permission information is stored in ``/etc/proxmox-backup/acl.cfg``. The
-file contains 5 fields, separated using a colon (':') as a delimiter. A typical
-entry takes the form:
-
-``acl:1:/datastore:john at pbs:DatastoreBackup``
-
-The data represented in each field is as follows:
-
-#. ``acl`` identifier
-#. A ``1`` or ``0``, representing whether propagation is enabled or disabled,
- respectively
-#. The object on which the permission is set. This can be a specific object
- (single datastore, remote, etc.) or a top level object, which with
- propagation enabled, represents all children of the object also.
-#. The user for which the permission is set
-#. The role being set
-
-You can manage datastore permissions from **Configuration -> Permissions** in the
-web interface. Likewise, you can use the ``acl`` subcommand to manage and
-monitor user permissions from the command line. For example, the command below
-will add the user ``john at pbs`` as a **DatastoreAdmin** for the datastore
-``store1``, located at ``/backup/disk1/store1``:
-
-.. code-block:: console
-
- # proxmox-backup-manager acl update /datastore/store1 DatastoreAdmin --userid john at pbs
-
-You can monitor the roles of each user using the following command:
-
-.. code-block:: console
-
- # proxmox-backup-manager acl list
- ┌──────────┬──────────────────┬───────────┬────────────────┐
- │ ugid │ path │ propagate │ roleid │
- ╞══════════╪══════════════════╪═══════════╪════════════════╡
- │ john at pbs │ /datastore/disk1 │ 1 │ DatastoreAdmin │
- └──────────┴──────────────────┴───────────┴────────────────┘
-
-A single user can be assigned multiple permission sets for different datastores.
-
-.. Note::
- Naming convention is important here. For datastores on the host,
- you must use the convention ``/datastore/{storename}``. For example, to set
- permissions for a datastore mounted at ``/mnt/backup/disk4/store2``, you would use
- ``/datastore/store2`` for the path. For remote stores, use the convention
- ``/remote/{remote}/{storename}``, where ``{remote}`` signifies the name of the
- remote (see `Remote` below) and ``{storename}`` is the name of the datastore on
- the remote.
-
-Network Management
-~~~~~~~~~~~~~~~~~~
-
-Proxmox Backup Server provides both a web interface and a command line tool for
-network configuration. You can find the configuration options in the web
-interface under the **Network Interfaces** section of the **Configuration** menu
-tree item. The command line tool is accessed via the ``network`` subcommand.
-These interfaces allow you to carry out some basic network management tasks,
-such as adding, configuring, and removing network interfaces.
-
-.. note:: Any changes made to the network configuration are not
- applied, until you click on **Apply Configuration** or enter the ``network
- reload`` command. This allows you to make many changes at once. It also allows
- you to ensure that your changes are correct before applying them, as making a
- mistake here can render the server inaccessible over the network.
-
-To get a list of available interfaces, use the following command:
-
-.. code-block:: console
-
- # proxmox-backup-manager network list
- ┌───────┬────────┬───────────┬────────┬─────────────┬──────────────┬──────────────┐
- │ name │ type │ autostart │ method │ address │ gateway │ ports/slaves │
- ╞═══════╪════════╪═══════════╪════════╪═════════════╪══════════════╪══════════════╡
- │ bond0 │ bond │ 1 │ static │ x.x.x.x/x │ x.x.x.x │ ens18 ens19 │
- ├───────┼────────┼───────────┼────────┼─────────────┼──────────────┼──────────────┤
- │ ens18 │ eth │ 1 │ manual │ │ │ │
- ├───────┼────────┼───────────┼────────┼─────────────┼──────────────┼──────────────┤
- │ ens19 │ eth │ 1 │ manual │ │ │ │
- └───────┴────────┴───────────┴────────┴─────────────┴──────────────┴──────────────┘
-
-.. image:: images/screenshots/pbs-gui-network-create-bond.png
- :align: right
- :alt: Add a network interface
-
-To add a new network interface, use the ``create`` subcommand with the relevant
-parameters. For example, you may want to set up a bond, for the purpose of
-network redundancy. The following command shows a template for creating the bond shown
-in the list above:
-
-.. code-block:: console
-
- # proxmox-backup-manager network create bond0 --type bond --bond_mode active-backup --slaves ens18,ens19 --autostart true --cidr x.x.x.x/x --gateway x.x.x.x
-
-You can make changes to the configuration of a network interface with the
-``update`` subcommand:
-
-.. code-block:: console
-
- # proxmox-backup-manager network update bond0 --cidr y.y.y.y/y
-
-You can also remove a network interface:
-
-.. code-block:: console
-
- # proxmox-backup-manager network remove bond0
-
-The pending changes for the network configuration file will appear at the bottom of the
-web interface. You can also view these changes, by using the command:
-
-.. code-block:: console
-
- # proxmox-backup-manager network changes
-
-If you would like to cancel all changes at this point, you can either click on
-the **Revert** button or use the following command:
-
-.. code-block:: console
-
- # proxmox-backup-manager network revert
-
-If you are happy with the changes and would like to write them into the
-configuration file, select **Apply Configuration**. The corresponding command
-is:
-
-.. code-block:: console
-
- # proxmox-backup-manager network reload
-
-.. note:: This command and corresponding GUI button rely on the ``ifreload``
- command, from the package ``ifupdown2``. This package is included within the
- Proxmox Backup Server installation, however, you may have to install it yourself,
- if you have installed Proxmox Backup Server on top of Debian or Proxmox VE.
-
-You can also configure DNS settings, from the **DNS** section
-of **Configuration** or by using the ``dns`` subcommand of
-``proxmox-backup-manager``.
-
-.. _backup_remote:
-
-:term:`Remote`
-~~~~~~~~~~~~~~
-
-A remote refers to a separate Proxmox Backup Server installation and a user on that
-installation, from which you can `sync` datastores to a local datastore with a
-`Sync Job`. You can configure remotes in the web interface, under **Configuration
--> Remotes**. Alternatively, you can use the ``remote`` subcommand. The
-configuration information for remotes is stored in the file
-``/etc/proxmox-backup/remote.cfg``.
-
-.. image:: images/screenshots/pbs-gui-remote-add.png
- :align: right
- :alt: Add a remote
-
-To add a remote, you need its hostname or ip, a userid and password on the
-remote, and its certificate fingerprint. To get the fingerprint, use the
-``proxmox-backup-manager cert info`` command on the remote, or navigate to
-**Dashboard** in the remote's web interface and select **Show Fingerprint**.
-
-.. code-block:: console
-
- # proxmox-backup-manager cert info |grep Fingerprint
- Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
-
-Using the information specified above, you can add a remote from the **Remotes**
-configuration panel, or by using the command:
-
-.. code-block:: console
-
- # proxmox-backup-manager remote create pbs2 --host pbs2.mydomain.example --userid sync at pam --password 'SECRET' --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
-
-Use the ``list``, ``show``, ``update``, ``remove`` subcommands of
-``proxmox-backup-manager remote`` to manage your remotes:
-
-.. code-block:: console
-
- # proxmox-backup-manager remote update pbs2 --host pbs2.example
- # proxmox-backup-manager remote list
- ┌──────┬──────────────┬──────────┬───────────────────────────────────────────┬─────────┐
- │ name │ host │ userid │ fingerprint │ comment │
- ╞══════╪══════════════╪══════════╪═══════════════════════════════════════════╪═════════╡
- │ pbs2 │ pbs2.example │ sync at pam │64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe │ │
- └──────┴──────────────┴──────────┴───────────────────────────────────────────┴─────────┘
- # proxmox-backup-manager remote remove pbs2
-
-.. _syncjobs:
-
-Sync Jobs
-~~~~~~~~~
-
-.. image:: images/screenshots/pbs-gui-syncjob-add.png
- :align: right
- :alt: Add a Sync Job
-
-Sync jobs are configured to pull the contents of a datastore on a **Remote** to
-a local datastore. You can manage sync jobs under **Configuration -> Sync Jobs**
-in the web interface, or using the ``proxmox-backup-manager sync-job`` command.
-The configuration information for sync jobs is stored at
-``/etc/proxmox-backup/sync.cfg``. To create a new sync job, click the add button
-in the GUI, or use the ``create`` subcommand. After creating a sync job, you can
-either start it manually on the GUI or provide it with a schedule (see
-:ref:`calendar-events`) to run regularly.
-
-.. code-block:: console
-
- # proxmox-backup-manager sync-job create pbs2-local --remote pbs2 --remote-store local --store local --schedule 'Wed 02:30'
- # proxmox-backup-manager sync-job update pbs2-local --comment 'offsite'
- # proxmox-backup-manager sync-job list
- ┌────────────┬───────┬────────┬──────────────┬───────────┬─────────┐
- │ id │ store │ remote │ remote-store │ schedule │ comment │
- ╞════════════╪═══════╪════════╪══════════════╪═══════════╪═════════╡
- │ pbs2-local │ local │ pbs2 │ local │ Wed 02:30 │ offsite │
- └────────────┴───────┴────────┴──────────────┴───────────┴─────────┘
- # proxmox-backup-manager sync-job remove pbs2-local
-
-
-Garbage Collection
-~~~~~~~~~~~~~~~~~~
-You can monitor and run :ref:`garbage collection <garbage-collection>` on the
-Proxmox Backup Server using the ``garbage-collection`` subcommand of
-``proxmox-backup-manager``. You can use the ``start`` subcommand to manually start garbage
-collection on an entire datastore and the ``status`` subcommand to see
-attributes relating to the :ref:`garbage collection <garbage-collection>`.
-
-
-Backup Client usage
--------------------
-
-The command line client is called :command:`proxmox-backup-client`.
-
-
-Repository Locations
-~~~~~~~~~~~~~~~~~~~~
-
-The client uses the following notation to specify a datastore repository
-on the backup server.
-
- [[username@]server[:port]:]datastore
-
-The default value for ``username`` ist ``root at pam``. If no server is specified,
-the default is the local host (``localhost``).
-
-You can specify a port if your backup server is only reachable on a different
-port (e.g. with NAT and port forwarding).
-
-Note that if the server is an IPv6 address, you have to write it with
-square brackets (e.g. [fe80::01]).
-
-You can pass the repository with the ``--repository`` command
-line option, or by setting the ``PBS_REPOSITORY`` environment
-variable.
-
-Here some examples of valid repositories and the real values
-
-================================ ============ ================== ===========
-Example User Host:Port Datastore
-================================ ============ ================== ===========
-mydatastore ``root at pam`` localhost:8007 mydatastore
-myhostname:mydatastore ``root at pam`` myhostname:8007 mydatastore
-user at pbs@myhostname:mydatastore ``user at pbs`` myhostname:8007 mydatastore
-192.168.55.55:1234:mydatastore ``root at pam`` 192.168.55.55:1234 mydatastore
-[ff80::51]:mydatastore ``root at pam`` [ff80::51]:8007 mydatastore
-[ff80::51]:1234:mydatastore ``root at pam`` [ff80::51]:1234 mydatastore
-================================ ============ ================== ===========
-
-Environment Variables
-~~~~~~~~~~~~~~~~~~~~~
-
-``PBS_REPOSITORY``
- The default backup repository.
-
-``PBS_PASSWORD``
- When set, this value is used for the password required for the
- backup server.
-
-``PBS_ENCRYPTION_PASSWORD``
- When set, this value is used to access the secret encryption key (if
- protected by password).
-
-``PBS_FINGERPRINT`` When set, this value is used to verify the server
- certificate (only used if the system CA certificates cannot
- validate the certificate).
-
-
-Output Format
-~~~~~~~~~~~~~
-
-Most commands support the ``--output-format`` parameter. It accepts
-the following values:
-
-:``text``: Text format (default). Structured data is rendered as a table.
-
-:``json``: JSON (single line).
-
-:``json-pretty``: JSON (multiple lines, nicely formatted).
-
-
-Please use the following environment variables to modify output behavior:
-
-``PROXMOX_OUTPUT_FORMAT``
- Defines the default output format.
-
-``PROXMOX_OUTPUT_NO_BORDER``
- If set (to any value), do not render table borders.
-
-``PROXMOX_OUTPUT_NO_HEADER``
- If set (to any value), do not render table headers.
-
-.. note:: The ``text`` format is designed to be human readable, and
- not meant to be parsed by automation tools. Please use the ``json``
- format if you need to process the output.
-
-
-.. _creating-backups:
-
-Creating Backups
-~~~~~~~~~~~~~~~~
-
-This section explains how to create a backup from within the machine. This can
-be a physical host, a virtual machine, or a container. Such backups may contain file
-and image archives. There are no restrictions in this case.
-
-.. note:: If you want to backup virtual machines or containers on Proxmox VE, see :ref:`pve-integration`.
-
-For the following example you need to have a backup server set up, working
-credentials and need to know the repository name.
-In the following examples we use ``backup-server:store1``.
-
-.. code-block:: console
-
- # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1
- Starting backup: host/elsa/2019-12-03T09:35:01Z
- Client name: elsa
- skip mount point: "/boot/efi"
- skip mount point: "/dev"
- skip mount point: "/run"
- skip mount point: "/sys"
- Uploaded 12129 chunks in 87 seconds (564 MB/s).
- End Time: 2019-12-03T10:36:29+01:00
-
-This will prompt you for a password and then uploads a file archive named
-``root.pxar`` containing all the files in the ``/`` directory.
-
-.. Caution:: Please note that the proxmox-backup-client does not
- automatically include mount points. Instead, you will see a short
- ``skip mount point`` notice for each of them. The idea is to
- create a separate file archive for each mounted disk. You can
- explicitly include them using the ``--include-dev`` option
- (i.e. ``--include-dev /boot/efi``). You can use this option
- multiple times for each mount point that should be included.
-
-The ``--repository`` option can get quite long and is used by all
-commands. You can avoid having to enter this value by setting the
-environment variable ``PBS_REPOSITORY``. Note that if you would like this to remain set
-over multiple sessions, you should instead add the below line to your
-``.bashrc`` file.
-
-.. code-block:: console
-
- # export PBS_REPOSITORY=backup-server:store1
-
-After this you can execute all commands without specifying the ``--repository``
-option.
-
-One single backup is allowed to contain more than one archive. For example, if
-you want to backup two disks mounted at ``/mmt/disk1`` and ``/mnt/disk2``:
-
-.. code-block:: console
-
- # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2
-
-This creates a backup of both disks.
-
-The backup command takes a list of backup specifications, which
-include the archive name on the server, the type of the archive, and the
-archive source at the client. The format is:
-
- <archive-name>.<type>:<source-path>
-
-Common types are ``.pxar`` for file archives, and ``.img`` for block
-device images. To create a backup of a block device run the following command:
-
-.. code-block:: console
-
- # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata
-
-Excluding files/folders from a backup
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Sometimes it is desired to exclude certain files or folders from a backup archive.
-To tell the Proxmox Backup client when and how to ignore files and directories,
-place a text file called ``.pxarexclude`` in the filesystem hierarchy.
-Whenever the backup client encounters such a file in a directory, it interprets
-each line as glob match patterns for files and directories that are to be excluded
-from the backup.
-
-The file must contain a single glob pattern per line. Empty lines are ignored.
-The same is true for lines starting with ``#``, which indicates a comment.
-A ``!`` at the beginning of a line reverses the glob match pattern from an exclusion
-to an explicit inclusion. This makes it possible to exclude all entries in a
-directory except for a few single files/subdirectories.
-Lines ending in ``/`` match only on directories.
-The directory containing the ``.pxarexclude`` file is considered to be the root of
-the given patterns. It is only possible to match files in this directory and its subdirectories.
-
-``\`` is used to escape special glob characters.
-``?`` matches any single character.
-``*`` matches any character, including an empty string.
-``**`` is used to match subdirectories. It can be used to, for example, exclude
-all files ending in ``.tmp`` within the directory or subdirectories with the
-following pattern ``**/*.tmp``.
-``[...]`` matches a single character from any of the provided characters within
-the brackets. ``[!...]`` does the complementary and matches any single character
-not contained within the brackets. It is also possible to specify ranges with two
-characters separated by ``-``. For example, ``[a-z]`` matches any lowercase
-alphabetic character and ``[0-9]`` matches any one single digit.
-
-The order of the glob match patterns defines whether a file is included or
-excluded, that is to say later entries override previous ones.
-This is also true for match patterns encountered deeper down the directory tree,
-which can override a previous exclusion.
-Be aware that excluded directories will **not** be read by the backup client.
-Thus, a ``.pxarexclude`` file in an excluded subdirectory will have no effect.
-``.pxarexclude`` files are treated as regular files and will be included in the
-backup archive.
-
-For example, consider the following directory structure:
-
-.. code-block:: console
-
- # ls -aR folder
- folder/:
- . .. .pxarexclude subfolder0 subfolder1
-
- folder/subfolder0:
- . .. file0 file1 file2 file3 .pxarexclude
-
- folder/subfolder1:
- . .. file0 file1 file2 file3
-
-The different ``.pxarexclude`` files contain the following:
-
-.. code-block:: console
-
- # cat folder/.pxarexclude
- /subfolder0/file1
- /subfolder1/*
- !/subfolder1/file2
-
-.. code-block:: console
-
- # cat folder/subfolder0/.pxarexclude
- file3
-
-This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of
-``subfolder1`` except ``file2``.
-
-Restoring this backup will result in:
-
-.. code-block:: console
-
- ls -aR restored
- restored/:
- . .. .pxarexclude subfolder0 subfolder1
-
- restored/subfolder0:
- . .. file0 file2 .pxarexclude
-
- restored/subfolder1:
- . .. file2
-
-Encryption
-~~~~~~~~~~
-
-Proxmox Backup supports client-side encryption with AES-256 in GCM_
-mode. To set this up, you first need to create an encryption key:
-
-.. code-block:: console
-
- # proxmox-backup-client key create my-backup.key
- Encryption Key Password: **************
-
-The key is password protected by default. If you do not need this
-extra protection, you can also create it without a password:
-
-.. code-block:: console
-
- # proxmox-backup-client key create /path/to/my-backup.key --kdf none
-
-Having created this key, it is now possible to create an encrypted backup, by
-passing the ``--keyfile`` parameter, with the path to the key file.
-
-.. code-block:: console
-
- # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key
- Password: *********
- Encryption Key Password: **************
- ...
-
-.. Note:: If you do not specify the name of the backup key, the key will be
- created in the default location
- ``~/.config/proxmox-backup/encryption-key.json``. ``proxmox-backup-client``
- will also search this location by default, in case the ``--keyfile``
- parameter is not specified.
-
-You can avoid entering the passwords by setting the environment
-variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``.
-
-Using a master key to store and recover encryption keys
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-You can also use ``proxmox-backup-client key`` to create an RSA public/private
-key pair, which can be used to store an encrypted version of the symmetric
-backup encryption key alongside each backup and recover it later.
-
-To set up a master key:
-
-1. Create an encryption key for the backup:
-
- .. code-block:: console
-
- # proxmox-backup-client key create
- creating default key at: "~/.config/proxmox-backup/encryption-key.json"
- Encryption Key Password: **********
- ...
-
- The resulting file will be saved to ``~/.config/proxmox-backup/encryption-key.json``.
-
-2. Create an RSA public/private key pair:
-
- .. code-block:: console
-
- # proxmox-backup-client key create-master-key
- Master Key Password: *********
- ...
-
- This will create two files in your current directory, ``master-public.pem``
- and ``master-private.pem``.
-
-3. Import the newly created ``master-public.pem`` public certificate, so that
- ``proxmox-backup-client`` can find and use it upon backup.
-
- .. code-block:: console
-
- # proxmox-backup-client key import-master-pubkey /path/to/master-public.pem
- Imported public master key to "~/.config/proxmox-backup/master-public.pem"
-
-4. With all these files in place, run a backup job:
-
- .. code-block:: console
-
- # proxmox-backup-client backup etc.pxar:/etc
-
- The key will be stored in your backup, under the name ``rsa-encrypted.key``.
-
- .. Note:: The ``--keyfile`` parameter can be excluded, if the encryption key
- is in the default path. If you specified another path upon creation, you
- must pass the ``--keyfile`` parameter.
-
-5. To test that everything worked, you can restore the key from the backup:
-
- .. code-block:: console
-
- # proxmox-backup-client restore /path/to/backup/ rsa-encrypted.key /path/to/target
-
- .. Note:: You should not need an encryption key to extract this file. However, if
- a key exists at the default location
- (``~/.config/proxmox-backup/encryption-key.json``) the program will prompt
- you for an encryption key password. Simply moving ``encryption-key.json``
- out of this directory will fix this issue.
-
-6. Then, use the previously generated master key to decrypt the file:
-
- .. code-block:: console
-
- # openssl rsautl -decrypt -inkey master-private.pem -in rsa-encrypted.key -out /path/to/target
- Enter pass phrase for ./master-private.pem: *********
-
-7. The target file will now contain the encryption key information in plain
- text. The success of this can be confirmed by passing the resulting ``json``
- file, with the ``--keyfile`` parameter, when decrypting files from the backup.
-
-.. warning:: Without their key, backed up files will be inaccessible. Thus, you should
- keep keys ordered and in a place that is separate from the contents being
- backed up. It can happen, for example, that you back up an entire system, using
- a key on that system. If the system then becomes inaccessable for any reason
- and needs to be restored, this will not be possible as the encryption key will be
- lost along with the broken system. In preparation for the worst case scenario,
- you should consider keeping a paper copy of this key locked away in
- a safe place.
-
-Restoring Data
-~~~~~~~~~~~~~~
-
-The regular creation of backups is a necessary step to avoiding data
-loss. More importantly, however, is the restoration. It is good practice to perform
-periodic recovery tests to ensure that you can access the data in
-case of problems.
-
-First, you need to find the snapshot which you want to restore. The snapshot
-command provides a list of all the snapshots on the server:
-
-.. code-block:: console
-
- # proxmox-backup-client snapshots
- ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐
- │ snapshot │ size │ files │
- ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡
- │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │
- ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
- │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │
- ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
- ...
-
-You can inspect the catalog to find specific files.
-
-.. code-block:: console
-
- # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z
- ...
- d "./root.pxar.didx/etc/cifs-utils"
- l "./root.pxar.didx/etc/cifs-utils/idmap-plugin"
- d "./root.pxar.didx/etc/console-setup"
- ...
-
-The restore command lets you restore a single archive from the
-backup.
-
-.. code-block:: console
-
- # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/
-
-To get the contents of any archive, you can restore the ``index.json`` file in the
-repository to the target path '-'. This will dump the contents to the standard output.
-
-.. code-block:: console
-
- # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json -
-
-
-Interactive Restores
-^^^^^^^^^^^^^^^^^^^^
-
-If you only want to restore a few individual files, it is often easier
-to use the interactive recovery shell.
-
-.. code-block:: console
-
- # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar
- Starting interactive shell
- pxar:/ > ls
- bin boot dev etc home lib lib32
- ...
-
-The interactive recovery shell is a minimalistic command line interface that
-utilizes the metadata stored in the catalog to quickly list, navigate and
-search files in a file archive.
-To restore files, you can select them individually or match them with a glob
-pattern.
-
-Using the catalog for navigation reduces the overhead considerably because only
-the catalog needs to be downloaded and, optionally, decrypted.
-The actual chunks are only accessed if the metadata in the catalog is not enough
-or for the actual restore.
-
-Similar to common UNIX shells ``cd`` and ``ls`` are the commands used to change
-working directory and list directory contents in the archive.
-``pwd`` shows the full path of the current working directory with respect to the
-archive root.
-
-Being able to quickly search the contents of the archive is a commmonly needed feature.
-That's where the catalog is most valuable.
-For example:
-
-.. code-block:: console
-
- pxar:/ > find etc/**/*.txt --select
- "/etc/X11/rgb.txt"
- pxar:/ > list-selected
- etc/**/*.txt
- pxar:/ > restore-selected /target/path
- ...
-
-This will find and print all files ending in ``.txt`` located in ``etc/`` or a
-subdirectory and add the corresponding pattern to the list for subsequent restores.
-``list-selected`` shows these patterns and ``restore-selected`` finally restores
-all files in the archive matching the patterns to ``/target/path`` on the local
-host. This will scan the whole archive.
-
-With ``restore /target/path`` you can restore the sub-archive given by the current
-working directory to the local target path ``/target/path`` on your host.
-By additionally passing a glob pattern with ``--pattern <glob>``, the restore is
-further limited to files matching the pattern.
-For example:
-
-.. code-block:: console
-
- pxar:/ > cd /etc/
- pxar:/etc/ > restore /target/ --pattern **/*.conf
- ...
-
-The above will scan trough all the directories below ``/etc`` and restore all
-files ending in ``.conf``.
-
-.. todo:: Explain interactive restore in more detail
-
-Mounting of Archives via FUSE
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The :term:`FUSE` implementation for the pxar archive allows you to mount a
-file archive as a read-only filesystem to a mountpoint on your host.
-
-.. code-block:: console
-
- # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt/mountpoint
- # ls /mnt/mountpoint
- bin dev home lib32 libx32 media opt root sbin sys usr
- boot etc lib lib64 lost+found mnt proc run srv tmp var
-
-This allows you to access the full contents of the archive in a seamless manner.
-
-.. note:: As the FUSE connection needs to fetch and decrypt chunks from the
- backup server's datastore, this can cause some additional network and CPU
- load on your host, depending on the operations you perform on the mounted
- filesystem.
-
-To unmount the filesystem use the ``umount`` command on the mountpoint:
-
-.. code-block:: console
-
- # umount /mnt/mountpoint
-
-Login and Logout
-~~~~~~~~~~~~~~~~
-
-The client tool prompts you to enter the logon password as soon as you
-want to access the backup server. The server checks your credentials
-and responds with a ticket that is valid for two hours. The client
-tool automatically stores that ticket and uses it for further requests
-to this server.
-
-You can also manually trigger this login/logout using the login and
-logout commands:
-
-.. code-block:: console
-
- # proxmox-backup-client login
- Password: **********
-
-To remove the ticket, issue a logout:
-
-.. code-block:: console
-
- # proxmox-backup-client logout
-
-
-.. _pruning:
-
-Pruning and Removing Backups
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You can manually delete a backup snapshot using the ``forget``
-command:
-
-.. code-block:: console
-
- # proxmox-backup-client forget <snapshot>
-
-
-.. caution:: This command removes all archives in this backup
- snapshot. They will be inaccessible and unrecoverable.
-
-
-Although manual removal is sometimes required, the ``prune``
-command is normally used to systematically delete older backups. Prune lets
-you specify which backup snapshots you want to keep. The
-following retention options are available:
-
-``--keep-last <N>``
- Keep the last ``<N>`` backup snapshots.
-
-``--keep-hourly <N>``
- Keep backups for the last ``<N>`` hours. If there is more than one
- backup for a single hour, only the latest is kept.
-
-``--keep-daily <N>``
- Keep backups for the last ``<N>`` days. If there is more than one
- backup for a single day, only the latest is kept.
-
-``--keep-weekly <N>``
- Keep backups for the last ``<N>`` weeks. If there is more than one
- backup for a single week, only the latest is kept.
-
- .. note:: Weeks start on Monday and end on Sunday. The software
- uses the `ISO week date`_ system and handles weeks at
- the end of the year correctly.
-
-``--keep-monthly <N>``
- Keep backups for the last ``<N>`` months. If there is more than one
- backup for a single month, only the latest is kept.
-
-``--keep-yearly <N>``
- Keep backups for the last ``<N>`` years. If there is more than one
- backup for a single year, only the latest is kept.
-
-The retention options are processed in the order given above. Each option
-only covers backups within its time period. The next option does not take care
-of already covered backups. It will only consider older backups.
-
-Unfinished and incomplete backups will be removed by the prune command unless
-they are newer than the last successful backup. In this case, the last failed
-backup is retained.
-
-.. code-block:: console
-
- # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3
-
-
-You can use the ``--dry-run`` option to test your settings. This only
-shows the list of existing snapshots and what actions prune would take.
-
-.. code-block:: console
-
- # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3
- ┌────────────────────────────────┬──────┐
- │ snapshot │ keep │
- ╞════════════════════════════════╪══════╡
- │ host/elsa/2019-12-04T13:20:37Z │ 1 │
- ├────────────────────────────────┼──────┤
- │ host/elsa/2019-12-03T09:35:01Z │ 0 │
- ├────────────────────────────────┼──────┤
- │ host/elsa/2019-11-22T11:54:47Z │ 1 │
- ├────────────────────────────────┼──────┤
- │ host/elsa/2019-11-21T12:36:25Z │ 0 │
- ├────────────────────────────────┼──────┤
- │ host/elsa/2019-11-10T10:42:20Z │ 1 │
- └────────────────────────────────┴──────┘
-
-.. note:: Neither the ``prune`` command nor the ``forget`` command free space
- in the chunk-store. The chunk-store still contains the data blocks. To free
- space you need to perform :ref:`garbage-collection`.
-
-
-.. _garbage-collection:
-
-Garbage Collection
-~~~~~~~~~~~~~~~~~~
-
-The ``prune`` command removes only the backup index files, not the data
-from the datastore. This task is left to the garbage collection
-command. It is recommended to carry out garbage collection on a regular basis.
-
-The garbage collection works in two phases. In the first phase, all
-data blocks that are still in use are marked. In the second phase,
-unused data blocks are removed.
-
-.. note:: This command needs to read all existing backup index files
- and touches the complete chunk-store. This can take a long time
- depending on the number of chunks and the speed of the underlying
- disks.
-
-.. note:: The garbage collection will only remove chunks that haven't been used
- for at least one day (exactly 24h 5m). This grace period is necessary because
- chunks in use are marked by touching the chunk which updates the ``atime``
- (access time) property. Filesystems are mounted with the ``relatime`` option
- by default. This results in a better performance by only updating the
- ``atime`` property if the last access has been at least 24 hours ago. The
- downside is, that touching a chunk within these 24 hours will not always
- update its ``atime`` property.
-
- Chunks in the grace period will be logged at the end of the garbage
- collection task as *Pending removals*.
-
-.. code-block:: console
-
- # proxmox-backup-client garbage-collect
- starting garbage collection on store store2
- Start GC phase1 (mark used chunks)
- Start GC phase2 (sweep unused chunks)
- percentage done: 1, chunk count: 219
- percentage done: 2, chunk count: 453
- ...
- percentage done: 99, chunk count: 21188
- Removed bytes: 411368505
- Removed chunks: 203
- Original data bytes: 327160886391
- Disk bytes: 52767414743 (16 %)
- Disk chunks: 21221
- Average chunk size: 2486565
- TASK OK
-
-
-.. todo:: howto run garbage-collection at regular intervalls (cron)
-
-Benchmarking
-~~~~~~~~~~~~
-The backup client also comes with a benchmarking tool. This tool measures
-various metrics relating to compression and encryption speeds. You can run a
-benchmark using the ``benchmark`` subcommand of ``proxmox-backup-client``:
-
-.. code-block:: console
-
- # proxmox-backup-client benchmark
- Uploaded 656 chunks in 5 seconds.
- Time per request: 7659 microseconds.
- TLS speed: 547.60 MB/s
- SHA256 speed: 585.76 MB/s
- Compression speed: 1923.96 MB/s
- Decompress speed: 7885.24 MB/s
- AES256/GCM speed: 3974.03 MB/s
- ┌───────────────────────────────────┬─────────────────────┐
- │ Name │ Value │
- ╞═══════════════════════════════════╪═════════════════════╡
- │ TLS (maximal backup upload speed) │ 547.60 MB/s (93%) │
- ├───────────────────────────────────┼─────────────────────┤
- │ SHA256 checksum computation speed │ 585.76 MB/s (28%) │
- ├───────────────────────────────────┼─────────────────────┤
- │ ZStd level 1 compression speed │ 1923.96 MB/s (89%) │
- ├───────────────────────────────────┼─────────────────────┤
- │ ZStd level 1 decompression speed │ 7885.24 MB/s (98%) │
- ├───────────────────────────────────┼─────────────────────┤
- │ AES256 GCM encryption speed │ 3974.03 MB/s (104%) │
- └───────────────────────────────────┴─────────────────────┘
-
-.. note:: The percentages given in the output table correspond to a
- comparison against a Ryzen 7 2700X. The TLS test connects to the
- local host, so there is no network involved.
-
-You can also pass the ``--output-format`` parameter to output stats in ``json``,
-rather than the default table format.
-
-.. _pve-integration:
-
-`Proxmox VE`_ integration
--------------------------
-
-You need to define a new storage with type 'pbs' on your `Proxmox VE`_
-node. The following example uses ``store2`` as storage name, and
-assumes the server address is ``localhost``, and you want to connect
-as ``user1 at pbs``.
-
-.. code-block:: console
-
- # pvesm add pbs store2 --server localhost --datastore store2
- # pvesm set store2 --username user1 at pbs --password <secret>
-
-.. note:: If you would rather not pass your password as plain text, you can pass
- the ``--password`` parameter, without any arguments. This will cause the
- program to prompt you for a password upon entering the command.
-
-If your backup server uses a self signed certificate, you need to add
-the certificate fingerprint to the configuration. You can get the
-fingerprint by running the following command on the backup server:
-
-.. code-block:: console
-
- # proxmox-backup-manager cert info | grep Fingerprint
- Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
-
-Please add that fingerprint to your configuration to establish a trust
-relationship:
-
-.. code-block:: console
-
- # pvesm set store2 --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
-
-After that you should be able to see storage status with:
-
-.. code-block:: console
-
- # pvesm status --storage store2
- Name Type Status Total Used Available %
- store2 pbs active 3905109820 1336687816 2568422004 34.23%
-
-Having added the PBS datastore to `Proxmox VE`_, you can backup VMs and
-containers in the same way you would for any other storage device within the
-environment (see `PVE Admin Guide: Backup and Restore
-<https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_vzdump>`_.
-
-
-.. include:: command-line-tools.rst
-
-.. include:: services.rst
diff --git a/docs/backup-client.rst b/docs/backup-client.rst
new file mode 100644
index 00000000..afa56fd8
--- /dev/null
+++ b/docs/backup-client.rst
@@ -0,0 +1,712 @@
+Backup Client Usage
+===================
+
+The command line client is called :command:`proxmox-backup-client`.
+
+
+Repository Locations
+--------------------
+
+The client uses the following notation to specify a datastore repository
+on the backup server.
+
+ [[username@]server[:port]:]datastore
+
+The default value for ``username`` ist ``root at pam``. If no server is specified,
+the default is the local host (``localhost``).
+
+You can specify a port if your backup server is only reachable on a different
+port (e.g. with NAT and port forwarding).
+
+Note that if the server is an IPv6 address, you have to write it with
+square brackets (e.g. [fe80::01]).
+
+You can pass the repository with the ``--repository`` command
+line option, or by setting the ``PBS_REPOSITORY`` environment
+variable.
+
+Here some examples of valid repositories and the real values
+
+================================ ============ ================== ===========
+Example User Host:Port Datastore
+================================ ============ ================== ===========
+mydatastore ``root at pam`` localhost:8007 mydatastore
+myhostname:mydatastore ``root at pam`` myhostname:8007 mydatastore
+user at pbs@myhostname:mydatastore ``user at pbs`` myhostname:8007 mydatastore
+192.168.55.55:1234:mydatastore ``root at pam`` 192.168.55.55:1234 mydatastore
+[ff80::51]:mydatastore ``root at pam`` [ff80::51]:8007 mydatastore
+[ff80::51]:1234:mydatastore ``root at pam`` [ff80::51]:1234 mydatastore
+================================ ============ ================== ===========
+
+Environment Variables
+---------------------
+
+``PBS_REPOSITORY``
+ The default backup repository.
+
+``PBS_PASSWORD``
+ When set, this value is used for the password required for the
+ backup server.
+
+``PBS_ENCRYPTION_PASSWORD``
+ When set, this value is used to access the secret encryption key (if
+ protected by password).
+
+``PBS_FINGERPRINT`` When set, this value is used to verify the server
+ certificate (only used if the system CA certificates cannot
+ validate the certificate).
+
+
+Output Format
+-------------
+
+Most commands support the ``--output-format`` parameter. It accepts
+the following values:
+
+:``text``: Text format (default). Structured data is rendered as a table.
+
+:``json``: JSON (single line).
+
+:``json-pretty``: JSON (multiple lines, nicely formatted).
+
+
+Please use the following environment variables to modify output behavior:
+
+``PROXMOX_OUTPUT_FORMAT``
+ Defines the default output format.
+
+``PROXMOX_OUTPUT_NO_BORDER``
+ If set (to any value), do not render table borders.
+
+``PROXMOX_OUTPUT_NO_HEADER``
+ If set (to any value), do not render table headers.
+
+.. note:: The ``text`` format is designed to be human readable, and
+ not meant to be parsed by automation tools. Please use the ``json``
+ format if you need to process the output.
+
+
+.. _creating-backups:
+
+Creating Backups
+----------------
+
+This section explains how to create a backup from within the machine. This can
+be a physical host, a virtual machine, or a container. Such backups may contain file
+and image archives. There are no restrictions in this case.
+
+.. note:: If you want to backup virtual machines or containers on Proxmox VE, see :ref:`pve-integration`.
+
+For the following example you need to have a backup server set up, working
+credentials and need to know the repository name.
+In the following examples we use ``backup-server:store1``.
+
+.. code-block:: console
+
+ # proxmox-backup-client backup root.pxar:/ --repository backup-server:store1
+ Starting backup: host/elsa/2019-12-03T09:35:01Z
+ Client name: elsa
+ skip mount point: "/boot/efi"
+ skip mount point: "/dev"
+ skip mount point: "/run"
+ skip mount point: "/sys"
+ Uploaded 12129 chunks in 87 seconds (564 MB/s).
+ End Time: 2019-12-03T10:36:29+01:00
+
+This will prompt you for a password and then uploads a file archive named
+``root.pxar`` containing all the files in the ``/`` directory.
+
+.. Caution:: Please note that the proxmox-backup-client does not
+ automatically include mount points. Instead, you will see a short
+ ``skip mount point`` notice for each of them. The idea is to
+ create a separate file archive for each mounted disk. You can
+ explicitly include them using the ``--include-dev`` option
+ (i.e. ``--include-dev /boot/efi``). You can use this option
+ multiple times for each mount point that should be included.
+
+The ``--repository`` option can get quite long and is used by all
+commands. You can avoid having to enter this value by setting the
+environment variable ``PBS_REPOSITORY``. Note that if you would like this to remain set
+over multiple sessions, you should instead add the below line to your
+``.bashrc`` file.
+
+.. code-block:: console
+
+ # export PBS_REPOSITORY=backup-server:store1
+
+After this you can execute all commands without specifying the ``--repository``
+option.
+
+One single backup is allowed to contain more than one archive. For example, if
+you want to backup two disks mounted at ``/mmt/disk1`` and ``/mnt/disk2``:
+
+.. code-block:: console
+
+ # proxmox-backup-client backup disk1.pxar:/mnt/disk1 disk2.pxar:/mnt/disk2
+
+This creates a backup of both disks.
+
+The backup command takes a list of backup specifications, which
+include the archive name on the server, the type of the archive, and the
+archive source at the client. The format is:
+
+ <archive-name>.<type>:<source-path>
+
+Common types are ``.pxar`` for file archives, and ``.img`` for block
+device images. To create a backup of a block device run the following command:
+
+.. code-block:: console
+
+ # proxmox-backup-client backup mydata.img:/dev/mylvm/mydata
+
+
+Excluding files/folders from a backup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Sometimes it is desired to exclude certain files or folders from a backup archive.
+To tell the Proxmox Backup client when and how to ignore files and directories,
+place a text file called ``.pxarexclude`` in the filesystem hierarchy.
+Whenever the backup client encounters such a file in a directory, it interprets
+each line as glob match patterns for files and directories that are to be excluded
+from the backup.
+
+The file must contain a single glob pattern per line. Empty lines are ignored.
+The same is true for lines starting with ``#``, which indicates a comment.
+A ``!`` at the beginning of a line reverses the glob match pattern from an exclusion
+to an explicit inclusion. This makes it possible to exclude all entries in a
+directory except for a few single files/subdirectories.
+Lines ending in ``/`` match only on directories.
+The directory containing the ``.pxarexclude`` file is considered to be the root of
+the given patterns. It is only possible to match files in this directory and its subdirectories.
+
+``\`` is used to escape special glob characters.
+``?`` matches any single character.
+``*`` matches any character, including an empty string.
+``**`` is used to match subdirectories. It can be used to, for example, exclude
+all files ending in ``.tmp`` within the directory or subdirectories with the
+following pattern ``**/*.tmp``.
+``[...]`` matches a single character from any of the provided characters within
+the brackets. ``[!...]`` does the complementary and matches any single character
+not contained within the brackets. It is also possible to specify ranges with two
+characters separated by ``-``. For example, ``[a-z]`` matches any lowercase
+alphabetic character and ``[0-9]`` matches any one single digit.
+
+The order of the glob match patterns defines whether a file is included or
+excluded, that is to say later entries override previous ones.
+This is also true for match patterns encountered deeper down the directory tree,
+which can override a previous exclusion.
+Be aware that excluded directories will **not** be read by the backup client.
+Thus, a ``.pxarexclude`` file in an excluded subdirectory will have no effect.
+``.pxarexclude`` files are treated as regular files and will be included in the
+backup archive.
+
+For example, consider the following directory structure:
+
+.. code-block:: console
+
+ # ls -aR folder
+ folder/:
+ . .. .pxarexclude subfolder0 subfolder1
+
+ folder/subfolder0:
+ . .. file0 file1 file2 file3 .pxarexclude
+
+ folder/subfolder1:
+ . .. file0 file1 file2 file3
+
+The different ``.pxarexclude`` files contain the following:
+
+.. code-block:: console
+
+ # cat folder/.pxarexclude
+ /subfolder0/file1
+ /subfolder1/*
+ !/subfolder1/file2
+
+.. code-block:: console
+
+ # cat folder/subfolder0/.pxarexclude
+ file3
+
+This would exclude ``file1`` and ``file3`` in ``subfolder0`` and all of
+``subfolder1`` except ``file2``.
+
+Restoring this backup will result in:
+
+.. code-block:: console
+
+ ls -aR restored
+ restored/:
+ . .. .pxarexclude subfolder0 subfolder1
+
+ restored/subfolder0:
+ . .. file0 file2 .pxarexclude
+
+ restored/subfolder1:
+ . .. file2
+
+
+Encryption
+----------
+
+Proxmox Backup supports client-side encryption with AES-256 in GCM_
+mode. To set this up, you first need to create an encryption key:
+
+.. code-block:: console
+
+ # proxmox-backup-client key create my-backup.key
+ Encryption Key Password: **************
+
+The key is password protected by default. If you do not need this
+extra protection, you can also create it without a password:
+
+.. code-block:: console
+
+ # proxmox-backup-client key create /path/to/my-backup.key --kdf none
+
+Having created this key, it is now possible to create an encrypted backup, by
+passing the ``--keyfile`` parameter, with the path to the key file.
+
+.. code-block:: console
+
+ # proxmox-backup-client backup etc.pxar:/etc --keyfile /path/to/my-backup.key
+ Password: *********
+ Encryption Key Password: **************
+ ...
+
+.. Note:: If you do not specify the name of the backup key, the key will be
+ created in the default location
+ ``~/.config/proxmox-backup/encryption-key.json``. ``proxmox-backup-client``
+ will also search this location by default, in case the ``--keyfile``
+ parameter is not specified.
+
+You can avoid entering the passwords by setting the environment
+variables ``PBS_PASSWORD`` and ``PBS_ENCRYPTION_PASSWORD``.
+
+
+Using a master key to store and recover encryption keys
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can also use ``proxmox-backup-client key`` to create an RSA public/private
+key pair, which can be used to store an encrypted version of the symmetric
+backup encryption key alongside each backup and recover it later.
+
+To set up a master key:
+
+1. Create an encryption key for the backup:
+
+ .. code-block:: console
+
+ # proxmox-backup-client key create
+ creating default key at: "~/.config/proxmox-backup/encryption-key.json"
+ Encryption Key Password: **********
+ ...
+
+ The resulting file will be saved to ``~/.config/proxmox-backup/encryption-key.json``.
+
+2. Create an RSA public/private key pair:
+
+ .. code-block:: console
+
+ # proxmox-backup-client key create-master-key
+ Master Key Password: *********
+ ...
+
+ This will create two files in your current directory, ``master-public.pem``
+ and ``master-private.pem``.
+
+3. Import the newly created ``master-public.pem`` public certificate, so that
+ ``proxmox-backup-client`` can find and use it upon backup.
+
+ .. code-block:: console
+
+ # proxmox-backup-client key import-master-pubkey /path/to/master-public.pem
+ Imported public master key to "~/.config/proxmox-backup/master-public.pem"
+
+4. With all these files in place, run a backup job:
+
+ .. code-block:: console
+
+ # proxmox-backup-client backup etc.pxar:/etc
+
+ The key will be stored in your backup, under the name ``rsa-encrypted.key``.
+
+ .. Note:: The ``--keyfile`` parameter can be excluded, if the encryption key
+ is in the default path. If you specified another path upon creation, you
+ must pass the ``--keyfile`` parameter.
+
+5. To test that everything worked, you can restore the key from the backup:
+
+ .. code-block:: console
+
+ # proxmox-backup-client restore /path/to/backup/ rsa-encrypted.key /path/to/target
+
+ .. Note:: You should not need an encryption key to extract this file. However, if
+ a key exists at the default location
+ (``~/.config/proxmox-backup/encryption-key.json``) the program will prompt
+ you for an encryption key password. Simply moving ``encryption-key.json``
+ out of this directory will fix this issue.
+
+6. Then, use the previously generated master key to decrypt the file:
+
+ .. code-block:: console
+
+ # openssl rsautl -decrypt -inkey master-private.pem -in rsa-encrypted.key -out /path/to/target
+ Enter pass phrase for ./master-private.pem: *********
+
+7. The target file will now contain the encryption key information in plain
+ text. The success of this can be confirmed by passing the resulting ``json``
+ file, with the ``--keyfile`` parameter, when decrypting files from the backup.
+
+.. warning:: Without their key, backed up files will be inaccessible. Thus, you should
+ keep keys ordered and in a place that is separate from the contents being
+ backed up. It can happen, for example, that you back up an entire system, using
+ a key on that system. If the system then becomes inaccessable for any reason
+ and needs to be restored, this will not be possible as the encryption key will be
+ lost along with the broken system. In preparation for the worst case scenario,
+ you should consider keeping a paper copy of this key locked away in
+ a safe place.
+
+
+Restoring Data
+--------------
+
+The regular creation of backups is a necessary step to avoiding data
+loss. More importantly, however, is the restoration. It is good practice to perform
+periodic recovery tests to ensure that you can access the data in
+case of problems.
+
+First, you need to find the snapshot which you want to restore. The snapshot
+command provides a list of all the snapshots on the server:
+
+.. code-block:: console
+
+ # proxmox-backup-client snapshots
+ ┌────────────────────────────────┬─────────────┬────────────────────────────────────┐
+ │ snapshot │ size │ files │
+ ╞════════════════════════════════╪═════════════╪════════════════════════════════════╡
+ │ host/elsa/2019-12-03T09:30:15Z │ 51788646825 │ root.pxar catalog.pcat1 index.json │
+ ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
+ │ host/elsa/2019-12-03T09:35:01Z │ 51790622048 │ root.pxar catalog.pcat1 index.json │
+ ├────────────────────────────────┼─────────────┼────────────────────────────────────┤
+ ...
+
+You can inspect the catalog to find specific files.
+
+.. code-block:: console
+
+ # proxmox-backup-client catalog dump host/elsa/2019-12-03T09:35:01Z
+ ...
+ d "./root.pxar.didx/etc/cifs-utils"
+ l "./root.pxar.didx/etc/cifs-utils/idmap-plugin"
+ d "./root.pxar.didx/etc/console-setup"
+ ...
+
+The restore command lets you restore a single archive from the
+backup.
+
+.. code-block:: console
+
+ # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z root.pxar /target/path/
+
+To get the contents of any archive, you can restore the ``index.json`` file in the
+repository to the target path '-'. This will dump the contents to the standard output.
+
+.. code-block:: console
+
+ # proxmox-backup-client restore host/elsa/2019-12-03T09:35:01Z index.json -
+
+
+Interactive Restores
+~~~~~~~~~~~~~~~~~~~~
+
+If you only want to restore a few individual files, it is often easier
+to use the interactive recovery shell.
+
+.. code-block:: console
+
+ # proxmox-backup-client catalog shell host/elsa/2019-12-03T09:35:01Z root.pxar
+ Starting interactive shell
+ pxar:/ > ls
+ bin boot dev etc home lib lib32
+ ...
+
+The interactive recovery shell is a minimalistic command line interface that
+utilizes the metadata stored in the catalog to quickly list, navigate and
+search files in a file archive.
+To restore files, you can select them individually or match them with a glob
+pattern.
+
+Using the catalog for navigation reduces the overhead considerably because only
+the catalog needs to be downloaded and, optionally, decrypted.
+The actual chunks are only accessed if the metadata in the catalog is not enough
+or for the actual restore.
+
+Similar to common UNIX shells ``cd`` and ``ls`` are the commands used to change
+working directory and list directory contents in the archive.
+``pwd`` shows the full path of the current working directory with respect to the
+archive root.
+
+Being able to quickly search the contents of the archive is a commmonly needed feature.
+That's where the catalog is most valuable.
+For example:
+
+.. code-block:: console
+
+ pxar:/ > find etc/**/*.txt --select
+ "/etc/X11/rgb.txt"
+ pxar:/ > list-selected
+ etc/**/*.txt
+ pxar:/ > restore-selected /target/path
+ ...
+
+This will find and print all files ending in ``.txt`` located in ``etc/`` or a
+subdirectory and add the corresponding pattern to the list for subsequent restores.
+``list-selected`` shows these patterns and ``restore-selected`` finally restores
+all files in the archive matching the patterns to ``/target/path`` on the local
+host. This will scan the whole archive.
+
+With ``restore /target/path`` you can restore the sub-archive given by the current
+working directory to the local target path ``/target/path`` on your host.
+By additionally passing a glob pattern with ``--pattern <glob>``, the restore is
+further limited to files matching the pattern.
+For example:
+
+.. code-block:: console
+
+ pxar:/ > cd /etc/
+ pxar:/etc/ > restore /target/ --pattern **/*.conf
+ ...
+
+The above will scan trough all the directories below ``/etc`` and restore all
+files ending in ``.conf``.
+
+.. todo:: Explain interactive restore in more detail
+
+Mounting of Archives via FUSE
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :term:`FUSE` implementation for the pxar archive allows you to mount a
+file archive as a read-only filesystem to a mountpoint on your host.
+
+.. code-block:: console
+
+ # proxmox-backup-client mount host/backup-client/2020-01-29T11:29:22Z root.pxar /mnt/mountpoint
+ # ls /mnt/mountpoint
+ bin dev home lib32 libx32 media opt root sbin sys usr
+ boot etc lib lib64 lost+found mnt proc run srv tmp var
+
+This allows you to access the full contents of the archive in a seamless manner.
+
+.. note:: As the FUSE connection needs to fetch and decrypt chunks from the
+ backup server's datastore, this can cause some additional network and CPU
+ load on your host, depending on the operations you perform on the mounted
+ filesystem.
+
+To unmount the filesystem use the ``umount`` command on the mountpoint:
+
+.. code-block:: console
+
+ # umount /mnt/mountpoint
+
+Login and Logout
+----------------
+
+The client tool prompts you to enter the logon password as soon as you
+want to access the backup server. The server checks your credentials
+and responds with a ticket that is valid for two hours. The client
+tool automatically stores that ticket and uses it for further requests
+to this server.
+
+You can also manually trigger this login/logout using the login and
+logout commands:
+
+.. code-block:: console
+
+ # proxmox-backup-client login
+ Password: **********
+
+To remove the ticket, issue a logout:
+
+.. code-block:: console
+
+ # proxmox-backup-client logout
+
+
+.. _pruning:
+
+Pruning and Removing Backups
+----------------------------
+
+You can manually delete a backup snapshot using the ``forget``
+command:
+
+.. code-block:: console
+
+ # proxmox-backup-client forget <snapshot>
+
+
+.. caution:: This command removes all archives in this backup
+ snapshot. They will be inaccessible and unrecoverable.
+
+
+Although manual removal is sometimes required, the ``prune``
+command is normally used to systematically delete older backups. Prune lets
+you specify which backup snapshots you want to keep. The
+following retention options are available:
+
+``--keep-last <N>``
+ Keep the last ``<N>`` backup snapshots.
+
+``--keep-hourly <N>``
+ Keep backups for the last ``<N>`` hours. If there is more than one
+ backup for a single hour, only the latest is kept.
+
+``--keep-daily <N>``
+ Keep backups for the last ``<N>`` days. If there is more than one
+ backup for a single day, only the latest is kept.
+
+``--keep-weekly <N>``
+ Keep backups for the last ``<N>`` weeks. If there is more than one
+ backup for a single week, only the latest is kept.
+
+ .. note:: Weeks start on Monday and end on Sunday. The software
+ uses the `ISO week date`_ system and handles weeks at
+ the end of the year correctly.
+
+``--keep-monthly <N>``
+ Keep backups for the last ``<N>`` months. If there is more than one
+ backup for a single month, only the latest is kept.
+
+``--keep-yearly <N>``
+ Keep backups for the last ``<N>`` years. If there is more than one
+ backup for a single year, only the latest is kept.
+
+The retention options are processed in the order given above. Each option
+only covers backups within its time period. The next option does not take care
+of already covered backups. It will only consider older backups.
+
+Unfinished and incomplete backups will be removed by the prune command unless
+they are newer than the last successful backup. In this case, the last failed
+backup is retained.
+
+.. code-block:: console
+
+ # proxmox-backup-client prune <group> --keep-daily 7 --keep-weekly 4 --keep-monthly 3
+
+
+You can use the ``--dry-run`` option to test your settings. This only
+shows the list of existing snapshots and what actions prune would take.
+
+.. code-block:: console
+
+ # proxmox-backup-client prune host/elsa --dry-run --keep-daily 1 --keep-weekly 3
+ ┌────────────────────────────────┬──────┐
+ │ snapshot │ keep │
+ ╞════════════════════════════════╪══════╡
+ │ host/elsa/2019-12-04T13:20:37Z │ 1 │
+ ├────────────────────────────────┼──────┤
+ │ host/elsa/2019-12-03T09:35:01Z │ 0 │
+ ├────────────────────────────────┼──────┤
+ │ host/elsa/2019-11-22T11:54:47Z │ 1 │
+ ├────────────────────────────────┼──────┤
+ │ host/elsa/2019-11-21T12:36:25Z │ 0 │
+ ├────────────────────────────────┼──────┤
+ │ host/elsa/2019-11-10T10:42:20Z │ 1 │
+ └────────────────────────────────┴──────┘
+
+.. note:: Neither the ``prune`` command nor the ``forget`` command free space
+ in the chunk-store. The chunk-store still contains the data blocks. To free
+ space you need to perform :ref:`garbage-collection`.
+
+
+.. _garbage-collection:
+
+Garbage Collection
+------------------
+
+The ``prune`` command removes only the backup index files, not the data
+from the datastore. This task is left to the garbage collection
+command. It is recommended to carry out garbage collection on a regular basis.
+
+The garbage collection works in two phases. In the first phase, all
+data blocks that are still in use are marked. In the second phase,
+unused data blocks are removed.
+
+.. note:: This command needs to read all existing backup index files
+ and touches the complete chunk-store. This can take a long time
+ depending on the number of chunks and the speed of the underlying
+ disks.
+
+.. note:: The garbage collection will only remove chunks that haven't been used
+ for at least one day (exactly 24h 5m). This grace period is necessary because
+ chunks in use are marked by touching the chunk which updates the ``atime``
+ (access time) property. Filesystems are mounted with the ``relatime`` option
+ by default. This results in a better performance by only updating the
+ ``atime`` property if the last access has been at least 24 hours ago. The
+ downside is, that touching a chunk within these 24 hours will not always
+ update its ``atime`` property.
+
+ Chunks in the grace period will be logged at the end of the garbage
+ collection task as *Pending removals*.
+
+.. code-block:: console
+
+ # proxmox-backup-client garbage-collect
+ starting garbage collection on store store2
+ Start GC phase1 (mark used chunks)
+ Start GC phase2 (sweep unused chunks)
+ percentage done: 1, chunk count: 219
+ percentage done: 2, chunk count: 453
+ ...
+ percentage done: 99, chunk count: 21188
+ Removed bytes: 411368505
+ Removed chunks: 203
+ Original data bytes: 327160886391
+ Disk bytes: 52767414743 (16 %)
+ Disk chunks: 21221
+ Average chunk size: 2486565
+ TASK OK
+
+
+.. todo:: howto run garbage-collection at regular intervalls (cron)
+
+Benchmarking
+------------
+
+The backup client also comes with a benchmarking tool. This tool measures
+various metrics relating to compression and encryption speeds. You can run a
+benchmark using the ``benchmark`` subcommand of ``proxmox-backup-client``:
+
+.. code-block:: console
+
+ # proxmox-backup-client benchmark
+ Uploaded 656 chunks in 5 seconds.
+ Time per request: 7659 microseconds.
+ TLS speed: 547.60 MB/s
+ SHA256 speed: 585.76 MB/s
+ Compression speed: 1923.96 MB/s
+ Decompress speed: 7885.24 MB/s
+ AES256/GCM speed: 3974.03 MB/s
+ ┌───────────────────────────────────┬─────────────────────┐
+ │ Name │ Value │
+ ╞═══════════════════════════════════╪═════════════════════╡
+ │ TLS (maximal backup upload speed) │ 547.60 MB/s (93%) │
+ ├───────────────────────────────────┼─────────────────────┤
+ │ SHA256 checksum computation speed │ 585.76 MB/s (28%) │
+ ├───────────────────────────────────┼─────────────────────┤
+ │ ZStd level 1 compression speed │ 1923.96 MB/s (89%) │
+ ├───────────────────────────────────┼─────────────────────┤
+ │ ZStd level 1 decompression speed │ 7885.24 MB/s (98%) │
+ ├───────────────────────────────────┼─────────────────────┤
+ │ AES256 GCM encryption speed │ 3974.03 MB/s (104%) │
+ └───────────────────────────────────┴─────────────────────┘
+
+.. note:: The percentages given in the output table correspond to a
+ comparison against a Ryzen 7 2700X. The TLS test connects to the
+ local host, so there is no network involved.
+
+You can also pass the ``--output-format`` parameter to output stats in ``json``,
+rather than the default table format.
+
+
diff --git a/docs/command-line-tools.rst b/docs/command-line-tools.rst
index 3240c49f..9b0a1290 100644
--- a/docs/command-line-tools.rst
+++ b/docs/command-line-tools.rst
@@ -12,7 +12,7 @@ Command Line Tools
.. include:: proxmox-backup-manager/description.rst
``pxar``
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~
.. include:: pxar/description.rst
diff --git a/docs/index.rst b/docs/index.rst
index 04ecd330..fffcb4fd 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -22,7 +22,16 @@ in the section entitled "GNU Free Documentation License".
introduction.rst
installation.rst
- administration-guide.rst
+ terminology.rst
+ gui.rst
+ storage.rst
+ network-management.rst
+ user-management.rst
+ managing-remotes.rst
+ maintenance.rst
+ backup-client.rst
+ pve-integration.rst
+ pxar-tool.rst
sysadmin.rst
faq.rst
diff --git a/docs/maintenance.rst b/docs/maintenance.rst
new file mode 100644
index 00000000..9b0357b8
--- /dev/null
+++ b/docs/maintenance.rst
@@ -0,0 +1,13 @@
+Maintenance Tasks
+=================
+
+Garbage Collection
+------------------
+
+You can monitor and run :ref:`garbage collection <garbage-collection>` on the
+Proxmox Backup Server using the ``garbage-collection`` subcommand of
+``proxmox-backup-manager``. You can use the ``start`` subcommand to manually start garbage
+collection on an entire datastore and the ``status`` subcommand to see
+attributes relating to the :ref:`garbage collection <garbage-collection>`.
+
+.. todo:: Add section on verification
diff --git a/docs/managing-remotes.rst b/docs/managing-remotes.rst
new file mode 100644
index 00000000..e32eb3b4
--- /dev/null
+++ b/docs/managing-remotes.rst
@@ -0,0 +1,80 @@
+Managing Remotes
+================
+
+:term:`Remote`
+--------------
+
+A remote refers to a separate Proxmox Backup Server installation and a user on that
+installation, from which you can `sync` datastores to a local datastore with a
+`Sync Job`. You can configure remotes in the web interface, under **Configuration
+-> Remotes**. Alternatively, you can use the ``remote`` subcommand. The
+configuration information for remotes is stored in the file
+``/etc/proxmox-backup/remote.cfg``.
+
+.. image:: images/screenshots/pbs-gui-remote-add.png
+ :align: right
+ :alt: Add a remote
+
+To add a remote, you need its hostname or ip, a userid and password on the
+remote, and its certificate fingerprint. To get the fingerprint, use the
+``proxmox-backup-manager cert info`` command on the remote, or navigate to
+**Dashboard** in the remote's web interface and select **Show Fingerprint**.
+
+.. code-block:: console
+
+ # proxmox-backup-manager cert info |grep Fingerprint
+ Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
+
+Using the information specified above, you can add a remote from the **Remotes**
+configuration panel, or by using the command:
+
+.. code-block:: console
+
+ # proxmox-backup-manager remote create pbs2 --host pbs2.mydomain.example --userid sync at pam --password 'SECRET' --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
+
+Use the ``list``, ``show``, ``update``, ``remove`` subcommands of
+``proxmox-backup-manager remote`` to manage your remotes:
+
+.. code-block:: console
+
+ # proxmox-backup-manager remote update pbs2 --host pbs2.example
+ # proxmox-backup-manager remote list
+ ┌──────┬──────────────┬──────────┬───────────────────────────────────────────┬─────────┐
+ │ name │ host │ userid │ fingerprint │ comment │
+ ╞══════╪══════════════╪══════════╪═══════════════════════════════════════════╪═════════╡
+ │ pbs2 │ pbs2.example │ sync at pam │64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe │ │
+ └──────┴──────────────┴──────────┴───────────────────────────────────────────┴─────────┘
+ # proxmox-backup-manager remote remove pbs2
+
+
+.. _syncjobs:
+
+Sync Jobs
+---------
+
+.. image:: images/screenshots/pbs-gui-syncjob-add.png
+ :align: right
+ :alt: Add a Sync Job
+
+Sync jobs are configured to pull the contents of a datastore on a **Remote** to
+a local datastore. You can manage sync jobs under **Configuration -> Sync Jobs**
+in the web interface, or using the ``proxmox-backup-manager sync-job`` command.
+The configuration information for sync jobs is stored at
+``/etc/proxmox-backup/sync.cfg``. To create a new sync job, click the add button
+in the GUI, or use the ``create`` subcommand. After creating a sync job, you can
+either start it manually on the GUI or provide it with a schedule (see
+:ref:`calendar-events`) to run regularly.
+
+.. code-block:: console
+
+ # proxmox-backup-manager sync-job create pbs2-local --remote pbs2 --remote-store local --store local --schedule 'Wed 02:30'
+ # proxmox-backup-manager sync-job update pbs2-local --comment 'offsite'
+ # proxmox-backup-manager sync-job list
+ ┌────────────┬───────┬────────┬──────────────┬───────────┬─────────┐
+ │ id │ store │ remote │ remote-store │ schedule │ comment │
+ ╞════════════╪═══════╪════════╪══════════════╪═══════════╪═════════╡
+ │ pbs2-local │ local │ pbs2 │ local │ Wed 02:30 │ offsite │
+ └────────────┴───────┴────────┴──────────────┴───────────┴─────────┘
+ # proxmox-backup-manager sync-job remove pbs2-local
+
+
diff --git a/docs/network-management.rst b/docs/network-management.rst
new file mode 100644
index 00000000..0656ea5f
--- /dev/null
+++ b/docs/network-management.rst
@@ -0,0 +1,91 @@
+Network Management
+==================
+
+Proxmox Backup Server provides both a web interface and a command line tool for
+network configuration. You can find the configuration options in the web
+interface under the **Network Interfaces** section of the **Configuration** menu
+tree item. The command line tool is accessed via the ``network`` subcommand.
+These interfaces allow you to carry out some basic network management tasks,
+such as adding, configuring, and removing network interfaces.
+
+.. note:: Any changes made to the network configuration are not
+ applied, until you click on **Apply Configuration** or enter the ``network
+ reload`` command. This allows you to make many changes at once. It also allows
+ you to ensure that your changes are correct before applying them, as making a
+ mistake here can render the server inaccessible over the network.
+
+To get a list of available interfaces, use the following command:
+
+.. code-block:: console
+
+ # proxmox-backup-manager network list
+ ┌───────┬────────┬───────────┬────────┬─────────────┬──────────────┬──────────────┐
+ │ name │ type │ autostart │ method │ address │ gateway │ ports/slaves │
+ ╞═══════╪════════╪═══════════╪════════╪═════════════╪══════════════╪══════════════╡
+ │ bond0 │ bond │ 1 │ static │ x.x.x.x/x │ x.x.x.x │ ens18 ens19 │
+ ├───────┼────────┼───────────┼────────┼─────────────┼──────────────┼──────────────┤
+ │ ens18 │ eth │ 1 │ manual │ │ │ │
+ ├───────┼────────┼───────────┼────────┼─────────────┼──────────────┼──────────────┤
+ │ ens19 │ eth │ 1 │ manual │ │ │ │
+ └───────┴────────┴───────────┴────────┴─────────────┴──────────────┴──────────────┘
+
+.. image:: images/screenshots/pbs-gui-network-create-bond.png
+ :align: right
+ :alt: Add a network interface
+
+To add a new network interface, use the ``create`` subcommand with the relevant
+parameters. For example, you may want to set up a bond, for the purpose of
+network redundancy. The following command shows a template for creating the bond shown
+in the list above:
+
+.. code-block:: console
+
+ # proxmox-backup-manager network create bond0 --type bond --bond_mode active-backup --slaves ens18,ens19 --autostart true --cidr x.x.x.x/x --gateway x.x.x.x
+
+You can make changes to the configuration of a network interface with the
+``update`` subcommand:
+
+.. code-block:: console
+
+ # proxmox-backup-manager network update bond0 --cidr y.y.y.y/y
+
+You can also remove a network interface:
+
+.. code-block:: console
+
+ # proxmox-backup-manager network remove bond0
+
+The pending changes for the network configuration file will appear at the bottom of the
+web interface. You can also view these changes, by using the command:
+
+.. code-block:: console
+
+ # proxmox-backup-manager network changes
+
+If you would like to cancel all changes at this point, you can either click on
+the **Revert** button or use the following command:
+
+.. code-block:: console
+
+ # proxmox-backup-manager network revert
+
+If you are happy with the changes and would like to write them into the
+configuration file, select **Apply Configuration**. The corresponding command
+is:
+
+.. code-block:: console
+
+ # proxmox-backup-manager network reload
+
+.. note:: This command and corresponding GUI button rely on the ``ifreload``
+ command, from the package ``ifupdown2``. This package is included within the
+ Proxmox Backup Server installation, however, you may have to install it yourself,
+ if you have installed Proxmox Backup Server on top of Debian or Proxmox VE.
+
+You can also configure DNS settings, from the **DNS** section
+of **Configuration** or by using the ``dns`` subcommand of
+``proxmox-backup-manager``.
+
+.. _backup_remote:
+
+
diff --git a/docs/pve-integration.rst b/docs/pve-integration.rst
new file mode 100644
index 00000000..fc6dcdfb
--- /dev/null
+++ b/docs/pve-integration.rst
@@ -0,0 +1,49 @@
+.. _pve-integration:
+
+`Proxmox VE`_ Integration
+-------------------------
+
+You need to define a new storage with type 'pbs' on your `Proxmox VE`_
+node. The following example uses ``store2`` as storage name, and
+assumes the server address is ``localhost``, and you want to connect
+as ``user1 at pbs``.
+
+.. code-block:: console
+
+ # pvesm add pbs store2 --server localhost --datastore store2
+ # pvesm set store2 --username user1 at pbs --password <secret>
+
+.. note:: If you would rather not pass your password as plain text, you can pass
+ the ``--password`` parameter, without any arguments. This will cause the
+ program to prompt you for a password upon entering the command.
+
+If your backup server uses a self signed certificate, you need to add
+the certificate fingerprint to the configuration. You can get the
+fingerprint by running the following command on the backup server:
+
+.. code-block:: console
+
+ # proxmox-backup-manager cert info | grep Fingerprint
+ Fingerprint (sha256): 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
+
+Please add that fingerprint to your configuration to establish a trust
+relationship:
+
+.. code-block:: console
+
+ # pvesm set store2 --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
+
+After that you should be able to see storage status with:
+
+.. code-block:: console
+
+ # pvesm status --storage store2
+ Name Type Status Total Used Available %
+ store2 pbs active 3905109820 1336687816 2568422004 34.23%
+
+Having added the PBS datastore to `Proxmox VE`_, you can backup VMs and
+containers in the same way you would for any other storage device within the
+environment (see `PVE Admin Guide: Backup and Restore
+<https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_vzdump>`_.
+
+
diff --git a/docs/pxar-tool.rst b/docs/pxar-tool.rst
new file mode 100644
index 00000000..b0164ff6
--- /dev/null
+++ b/docs/pxar-tool.rst
@@ -0,0 +1,5 @@
+pxar Command Line Tool
+======================
+
+.. include:: pxar/description.rst
+
diff --git a/docs/pxar/description.rst b/docs/pxar/description.rst
index 856ffd51..18be7687 100644
--- a/docs/pxar/description.rst
+++ b/docs/pxar/description.rst
@@ -47,7 +47,7 @@ by running:
Be aware that the shell itself will try to expand all of the glob patterns before
invoking ``pxar``.
In order to avoid this, all globs have to be quoted correctly.
-
+
It is possible to pass the ``--exclude`` parameter multiple times, in order to
match more than one pattern. This allows you to use more complex
file exclusion/inclusion behavior. However, it is recommended to use
diff --git a/docs/storage.rst b/docs/storage.rst
new file mode 100644
index 00000000..3a4875c6
--- /dev/null
+++ b/docs/storage.rst
@@ -0,0 +1,244 @@
+Storage
+=======
+
+Disk Management
+---------------
+
+.. image:: images/screenshots/pbs-gui-disks.png
+ :align: right
+ :alt: List of disks
+
+Proxmox Backup Server comes with a set of disk utilities, which are
+accessed using the ``disk`` subcommand. This subcommand allows you to initialize
+disks, create various filesystems, and get information about the disks.
+
+To view the disks connected to the system, navigate to **Administration ->
+Disks** in the web interface or use the ``list`` subcommand of
+``disk``:
+
+.. code-block:: console
+
+ # proxmox-backup-manager disk list
+ ┌──────┬────────┬─────┬───────────┬─────────────┬───────────────┬─────────┬────────┐
+ │ name │ used │ gpt │ disk-type │ size │ model │ wearout │ status │
+ ╞══════╪════════╪═════╪═══════════╪═════════════╪═══════════════╪═════════╪════════╡
+ │ sda │ lvm │ 1 │ hdd │ 34359738368 │ QEMU_HARDDISK │ - │ passed │
+ ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
+ │ sdb │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
+ ├──────┼────────┼─────┼───────────┼─────────────┼───────────────┼─────────┼────────┤
+ │ sdc │ unused │ 1 │ hdd │ 68719476736 │ QEMU_HARDDISK │ - │ passed │
+ └──────┴────────┴─────┴───────────┴─────────────┴───────────────┴─────────┴────────┘
+
+To initialize a disk with a new GPT, use the ``initialize`` subcommand:
+
+.. code-block:: console
+
+ # proxmox-backup-manager disk initialize sdX
+
+.. image:: images/screenshots/pbs-gui-disks-dir-create.png
+ :align: right
+ :alt: Create a directory
+
+You can create an ``ext4`` or ``xfs`` filesystem on a disk using ``fs
+create``, or by navigating to **Administration -> Disks -> Directory** in the
+web interface and creating one from there. The following command creates an
+``ext4`` filesystem and passes the ``--add-datastore`` parameter, in order to
+automatically create a datastore on the disk (in this case ``sdd``). This will
+create a datastore at the location ``/mnt/datastore/store1``:
+
+.. code-block:: console
+
+ # proxmox-backup-manager disk fs create store1 --disk sdd --filesystem ext4 --add-datastore true
+
+.. image:: images/screenshots/pbs-gui-disks-zfs-create.png
+ :align: right
+ :alt: Create ZFS
+
+You can also create a ``zpool`` with various raid levels from **Administration
+-> Disks -> Zpool** in the web interface, or by using ``zpool create``. The command
+below creates a mirrored ``zpool`` using two disks (``sdb`` & ``sdc``) and
+mounts it on the root directory (default):
+
+.. code-block:: console
+
+ # proxmox-backup-manager disk zpool create zpool1 --devices sdb,sdc --raidlevel mirror
+
+.. note:: You can also pass the ``--add-datastore`` parameter here, to automatically
+ create a datastore from the disk.
+
+You can use ``disk fs list`` and ``disk zpool list`` to keep track of your
+filesystems and zpools respectively.
+
+Proxmox Backup Server uses the package smartmontools. This is a set of tools
+used to monitor and control the S.M.A.R.T. system for local hard disks. If a
+disk supports S.M.A.R.T. capability, and you have this enabled, you can
+display S.M.A.R.T. attributes from the web interface or by using the command:
+
+.. code-block:: console
+
+ # proxmox-backup-manager disk smart-attributes sdX
+
+.. note:: This functionality may also be accessed directly through the use of
+ the ``smartctl`` command, which comes as part of the smartmontools package
+ (see ``man smartctl`` for more details).
+
+
+.. _datastore_intro:
+
+:term:`DataStore`
+-----------------
+
+A datastore refers to a location at which backups are stored. The current
+implementation uses a directory inside a standard unix file system (``ext4``,
+``xfs`` or ``zfs``) to store the backup data.
+
+Datastores are identified by a simple *ID*. You can configure this
+when setting up the datastore. The configuration information for datastores
+is stored in the file ``/etc/proxmox-backup/datastore.cfg``.
+
+.. note:: The `File Layout`_ requires the file system to support at least *65538*
+ subdirectories per directory. That number comes from the 2\ :sup:`16`
+ pre-created chunk namespace directories, and the ``.`` and ``..`` default
+ directory entries. This requirement excludes certain filesystems and
+ filesystem configuration from being supported for a datastore. For example,
+ ``ext3`` as a whole or ``ext4`` with the ``dir_nlink`` feature manually disabled.
+
+
+Datastore Configuration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+.. image:: images/screenshots/pbs-gui-datastore.png
+ :align: right
+ :alt: Datastore Overview
+
+You can configure multiple datastores. Minimum one datastore needs to be
+configured. The datastore is identified by a simple *name* and points to a
+directory on the filesystem. Each datastore also has associated retention
+settings of how many backup snapshots for each interval of ``hourly``,
+``daily``, ``weekly``, ``monthly``, ``yearly`` as well as a time-independent
+number of backups to keep in that store. :ref:`Pruning <pruning>` and
+:ref:`garbage collection <garbage-collection>` can also be configured to run
+periodically based on a configured schedule (see :ref:`calendar-events`) per datastore.
+
+
+Creating a Datastore
+^^^^^^^^^^^^^^^^^^^^
+.. image:: images/screenshots/pbs-gui-datastore-create-general.png
+ :align: right
+ :alt: Create a datastore
+
+You can create a new datastore from the web GUI, by navigating to **Datastore** in
+the menu tree and clicking **Create**. Here:
+
+* *Name* refers to the name of the datastore
+* *Backing Path* is the path to the directory upon which you want to create the
+ datastore
+* *GC Schedule* refers to the time and intervals at which garbage collection
+ runs
+* *Prune Schedule* refers to the frequency at which pruning takes place
+* *Prune Options* set the amount of backups which you would like to keep (see :ref:`Pruning <pruning>`).
+
+Alternatively you can create a new datastore from the command line. The
+following command creates a new datastore called ``store1`` on :file:`/backup/disk1/store1`
+
+.. code-block:: console
+
+ # proxmox-backup-manager datastore create store1 /backup/disk1/store1
+
+
+Managing Datastores
+^^^^^^^^^^^^^^^^^^^
+
+To list existing datastores from the command line run:
+
+.. code-block:: console
+
+ # proxmox-backup-manager datastore list
+ ┌────────┬──────────────────────┬─────────────────────────────┐
+ │ name │ path │ comment │
+ ╞════════╪══════════════════════╪═════════════════════════════╡
+ │ store1 │ /backup/disk1/store1 │ This is my default storage. │
+ └────────┴──────────────────────┴─────────────────────────────┘
+
+You can change the garbage collection and prune settings of a datastore, by
+editing the datastore from the GUI or by using the ``update`` subcommand. For
+example, the below command changes the garbage collection schedule using the
+``update`` subcommand and prints the properties of the datastore with the
+``show`` subcommand:
+
+.. code-block:: console
+
+ # proxmox-backup-manager datastore update store1 --gc-schedule 'Tue 04:27'
+ # proxmox-backup-manager datastore show store1
+ ┌────────────────┬─────────────────────────────┐
+ │ Name │ Value │
+ ╞════════════════╪═════════════════════════════╡
+ │ name │ store1 │
+ ├────────────────┼─────────────────────────────┤
+ │ path │ /backup/disk1/store1 │
+ ├────────────────┼─────────────────────────────┤
+ │ comment │ This is my default storage. │
+ ├────────────────┼─────────────────────────────┤
+ │ gc-schedule │ Tue 04:27 │
+ ├────────────────┼─────────────────────────────┤
+ │ keep-last │ 7 │
+ ├────────────────┼─────────────────────────────┤
+ │ prune-schedule │ daily │
+ └────────────────┴─────────────────────────────┘
+
+Finally, it is possible to remove the datastore configuration:
+
+.. code-block:: console
+
+ # proxmox-backup-manager datastore remove store1
+
+.. note:: The above command removes only the datastore configuration. It does
+ not delete any data from the underlying directory.
+
+
+File Layout
+^^^^^^^^^^^
+
+After creating a datastore, the following default layout will appear:
+
+.. code-block:: console
+
+ # ls -arilh /backup/disk1/store1
+ 276493 -rw-r--r-- 1 backup backup 0 Jul 8 12:35 .lock
+ 276490 drwxr-x--- 1 backup backup 1064960 Jul 8 12:35 .chunks
+
+`.lock` is an empty file used for process locking.
+
+The `.chunks` directory contains folders, starting from `0000` and taking hexadecimal values until `ffff`. These
+directories will store the chunked data after a backup operation has been executed.
+
+.. code-block:: console
+
+ # ls -arilh /backup/disk1/store1/.chunks
+ 545824 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 ffff
+ 545823 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffe
+ 415621 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffd
+ 415620 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffc
+ 353187 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffb
+ 344995 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fffa
+ 144079 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff9
+ 144078 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff8
+ 144077 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 fff7
+ ...
+ 403180 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000c
+ 403179 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000b
+ 403177 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 000a
+ 402530 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0009
+ 402513 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0008
+ 402509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0007
+ 276509 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0006
+ 276508 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0005
+ 276507 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0004
+ 276501 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0003
+ 276499 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0002
+ 276498 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0001
+ 276494 drwxr-x--- 2 backup backup 4.0K Jul 8 12:35 0000
+ 276489 drwxr-xr-x 3 backup backup 4.0K Jul 8 12:35 ..
+ 276490 drwxr-x--- 1 backup backup 1.1M Jul 8 12:35 .
+
+
diff --git a/docs/terminology.rst b/docs/terminology.rst
new file mode 100644
index 00000000..89ec7646
--- /dev/null
+++ b/docs/terminology.rst
@@ -0,0 +1,118 @@
+Terminology
+===========
+
+Backup Content
+--------------
+
+When doing deduplication, there are different strategies to get
+optimal results in terms of performance and/or deduplication rates.
+Depending on the type of data, it can be split into *fixed* or *variable*
+sized chunks.
+
+Fixed sized chunking requires minimal CPU power, and is used to
+backup virtual machine images.
+
+Variable sized chunking needs more CPU power, but is essential to get
+good deduplication rates for file archives.
+
+The Proxmox Backup Server supports both strategies.
+
+
+Image Archives: ``<name>.img``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is used for virtual machine images and other large binary
+data. Content is split into fixed-sized chunks.
+
+
+File Archives: ``<name>.pxar``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. see https://moinakg.wordpress.com/2013/06/22/high-performance-content-defined-chunking/
+
+A file archive stores a full directory tree. Content is stored using
+the :ref:`pxar-format`, split into variable-sized chunks. The format
+is optimized to achieve good deduplication rates.
+
+
+Binary Data (BLOBs)
+~~~~~~~~~~~~~~~~~~~
+
+This type is used to store smaller (< 16MB) binary data such as
+configuration files. Larger files should be stored as image archive.
+
+.. caution:: Please do not store all files as BLOBs. Instead, use the
+ file archive to store whole directory trees.
+
+
+Catalog File: ``catalog.pcat1``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The catalog file is an index for file archives. It contains
+the list of files and is used to speed up search operations.
+
+
+The Manifest: ``index.json``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The manifest contains the list of all backup files, their
+sizes and checksums. It is used to verify the consistency of a
+backup.
+
+
+Backup Type
+-----------
+
+The backup server groups backups by *type*, where *type* is one of:
+
+``vm``
+ This type is used for :term:`virtual machine`\ s. Typically
+ consists of the virtual machine's configuration file and an image archive
+ for each disk.
+
+``ct``
+ This type is used for :term:`container`\ s. Consists of the container's
+ configuration and a single file archive for the filesystem content.
+
+``host``
+ This type is used for backups created from within the backed up machine.
+ Typically this would be a physical host but could also be a virtual machine
+ or container. Such backups may contain file and image archives, there are no restrictions in this regard.
+
+
+Backup ID
+---------
+
+A unique ID. Usually the virtual machine or container ID. ``host``
+type backups normally use the hostname.
+
+
+Backup Time
+-----------
+
+The time when the backup was made.
+
+
+Backup Group
+------------
+
+The tuple ``<type>/<ID>`` is called a backup group. Such a group
+may contain one or more backup snapshots.
+
+
+Backup Snapshot
+---------------
+
+The triplet ``<type>/<ID>/<time>`` is called a backup snapshot. It
+uniquely identifies a specific backup within a datastore.
+
+.. code-block:: console
+ :caption: Backup Snapshot Examples
+
+ vm/104/2019-10-09T08:01:06Z
+ host/elsa/2019-11-08T09:48:14Z
+
+As you can see, the time format is RFC3399_ with Coordinated
+Universal Time (UTC_, identified by the trailing *Z*).
+
+
diff --git a/docs/user-management.rst b/docs/user-management.rst
new file mode 100644
index 00000000..56ad7a71
--- /dev/null
+++ b/docs/user-management.rst
@@ -0,0 +1,186 @@
+.. _user_mgmt:
+
+User Management
+===============
+
+
+User Configuration
+------------------
+
+.. image:: images/screenshots/pbs-gui-user-management.png
+ :align: right
+ :alt: User management
+
+Proxmox Backup Server supports several authentication realms, and you need to
+choose the realm when you add a new user. Possible realms are:
+
+:pam: Linux PAM standard authentication. Use this if you want to
+ authenticate as Linux system user (Users need to exist on the
+ system).
+
+:pbs: Proxmox Backup Server realm. This type stores hashed passwords in
+ ``/etc/proxmox-backup/shadow.json``.
+
+After installation, there is a single user ``root at pam``, which
+corresponds to the Unix superuser. User configuration information is stored in the file
+``/etc/proxmox-backup/user.cfg``. You can use the
+``proxmox-backup-manager`` command line tool to list or manipulate
+users:
+
+.. code-block:: console
+
+ # proxmox-backup-manager user list
+ ┌─────────────┬────────┬────────┬───────────┬──────────┬────────────────┬────────────────────┐
+ │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
+ ╞═════════════╪════════╪════════╪═══════════╪══════════╪════════════════╪════════════════════╡
+ │ root at pam │ 1 │ │ │ │ │ Superuser │
+ └─────────────┴────────┴────────┴───────────┴──────────┴────────────────┴────────────────────┘
+
+.. image:: images/screenshots/pbs-gui-user-management-add-user.png
+ :align: right
+ :alt: Add a new user
+
+The superuser has full administration rights on everything, so you
+normally want to add other users with less privileges. You can create a new
+user with the ``user create`` subcommand or through the web interface, under
+**Configuration -> User Management**. The ``create`` subcommand lets you specify
+many options like ``--email`` or ``--password``. You can update or change any
+user properties using the ``update`` subcommand later (**Edit** in the GUI):
+
+
+.. code-block:: console
+
+ # proxmox-backup-manager user create john at pbs --email john at example.com
+ # proxmox-backup-manager user update john at pbs --firstname John --lastname Smith
+ # proxmox-backup-manager user update john at pbs --comment "An example user."
+
+.. todo:: Mention how to set password without passing plaintext password as cli argument.
+
+
+The resulting user list looks like this:
+
+.. code-block:: console
+
+ # proxmox-backup-manager user list
+ ┌──────────┬────────┬────────┬───────────┬──────────┬──────────────────┬──────────────────┐
+ │ userid │ enable │ expire │ firstname │ lastname │ email │ comment │
+ ╞══════════╪════════╪════════╪═══════════╪══════════╪══════════════════╪══════════════════╡
+ │ john at pbs │ 1 │ │ John │ Smith │ john at example.com │ An example user. │
+ ├──────────┼────────┼────────┼───────────┼──────────┼──────────────────┼──────────────────┤
+ │ root at pam │ 1 │ │ │ │ │ Superuser │
+ └──────────┴────────┴────────┴───────────┴──────────┴──────────────────┴──────────────────┘
+
+Newly created users do not have any permissions. Please read the next
+section to learn how to set access permissions.
+
+If you want to disable a user account, you can do that by setting ``--enable`` to ``0``
+
+.. code-block:: console
+
+ # proxmox-backup-manager user update john at pbs --enable 0
+
+Or completely remove the user with:
+
+.. code-block:: console
+
+ # proxmox-backup-manager user remove john at pbs
+
+
+.. _user_acl:
+
+Access Control
+--------------
+
+By default new users do not have any permission. Instead you need to
+specify what is allowed and what is not. You can do this by assigning
+roles to users on specific objects like datastores or remotes. The
+following roles exist:
+
+**NoAccess**
+ Disable Access - nothing is allowed.
+
+**Admin**
+ Can do anything.
+
+**Audit**
+ Can view things, but is not allowed to change settings.
+
+**DatastoreAdmin**
+ Can do anything on datastores.
+
+**DatastoreAudit**
+ Can view datastore settings and list content. But
+ is not allowed to read the actual data.
+
+**DatastoreReader**
+ Can Inspect datastore content and can do restores.
+
+**DatastoreBackup**
+ Can backup and restore owned backups.
+
+**DatastorePowerUser**
+ Can backup, restore, and prune owned backups.
+
+**RemoteAdmin**
+ Can do anything on remotes.
+
+**RemoteAudit**
+ Can view remote settings.
+
+**RemoteSyncOperator**
+ Is allowed to read data from a remote.
+
+.. image:: images/screenshots/pbs-gui-permissions-add.png
+ :align: right
+ :alt: Add permissions for user
+
+Access permission information is stored in ``/etc/proxmox-backup/acl.cfg``. The
+file contains 5 fields, separated using a colon (':') as a delimiter. A typical
+entry takes the form:
+
+``acl:1:/datastore:john at pbs:DatastoreBackup``
+
+The data represented in each field is as follows:
+
+#. ``acl`` identifier
+#. A ``1`` or ``0``, representing whether propagation is enabled or disabled,
+ respectively
+#. The object on which the permission is set. This can be a specific object
+ (single datastore, remote, etc.) or a top level object, which with
+ propagation enabled, represents all children of the object also.
+#. The user for which the permission is set
+#. The role being set
+
+You can manage datastore permissions from **Configuration -> Permissions** in the
+web interface. Likewise, you can use the ``acl`` subcommand to manage and
+monitor user permissions from the command line. For example, the command below
+will add the user ``john at pbs`` as a **DatastoreAdmin** for the datastore
+``store1``, located at ``/backup/disk1/store1``:
+
+.. code-block:: console
+
+ # proxmox-backup-manager acl update /datastore/store1 DatastoreAdmin --userid john at pbs
+
+You can monitor the roles of each user using the following command:
+
+.. code-block:: console
+
+ # proxmox-backup-manager acl list
+ ┌──────────┬──────────────────┬───────────┬────────────────┐
+ │ ugid │ path │ propagate │ roleid │
+ ╞══════════╪══════════════════╪═══════════╪════════════════╡
+ │ john at pbs │ /datastore/disk1 │ 1 │ DatastoreAdmin │
+ └──────────┴──────────────────┴───────────┴────────────────┘
+
+A single user can be assigned multiple permission sets for different datastores.
+
+.. Note::
+ Naming convention is important here. For datastores on the host,
+ you must use the convention ``/datastore/{storename}``. For example, to set
+ permissions for a datastore mounted at ``/mnt/backup/disk4/store2``, you would use
+ ``/datastore/store2`` for the path. For remote stores, use the convention
+ ``/remote/{remote}/{storename}``, where ``{remote}`` signifies the name of the
+ remote (see `Remote` below) and ``{storename}`` is the name of the datastore on
+ the remote.
+
+
--
2.20.1
More information about the pbs-devel
mailing list