[pve-devel] [PATCH docs 4/6] ceph: osd: revise and expand the section "Destroy OSDs"

[Max Carrara]

On Mon Feb 3, 2025 at 3:27 PM CET, Alexander Zeidler wrote:
> Existing information is slightly modified and retained.
>
> Add information:
> * Mention and link to the sections "Troubleshooting" and "Replace OSDs"
> * CLI commands (pveceph) must be executed on the affected node
> * Check in advance the "Used (%)" of OSDs to avoid blocked I/O
> * Check and wait until the OSD can be stopped safely
> * Use `pveceph stop` instead of `systemctl stop ceph-osd@<ID>.service`
> * Explain cleanup option a bit more
>
> Signed-off-by: Alexander Zeidler <a.zeidler at proxmox.com>
> ---
>  pveceph.adoc | 58 ++++++++++++++++++++++++++++------------------------
>  1 file changed, 31 insertions(+), 27 deletions(-)
>
> diff --git a/pveceph.adoc b/pveceph.adoc
> index 4e1c1e2..754c401 100644
> --- a/pveceph.adoc
> +++ b/pveceph.adoc
> @@ -502,33 +502,37 @@ ceph-volume lvm create --filestore --data /dev/sd[X] --journal /dev/sd[Y]
>  Destroy OSDs
>  ~~~~~~~~~~~~
>  
> -To remove an OSD via the GUI, first select a {PVE} node in the tree view and go
> -to the **Ceph -> OSD** panel. Then select the OSD to destroy and click the **OUT**
> -button. Once the OSD status has changed from `in` to `out`, click the **STOP**
> -button. Finally, after the status has changed from `up` to `down`, select
> -**Destroy** from the `More` drop-down menu.
> -
> -To remove an OSD via the CLI run the following commands.
> -
> -[source,bash]
> -----
> -ceph osd out <ID>
> -systemctl stop ceph-osd@<ID>.service
> -----
> -
> -NOTE: The first command instructs Ceph not to include the OSD in the data
> -distribution. The second command stops the OSD service. Until this time, no
> -data is lost.
> -
> -The following command destroys the OSD. Specify the '-cleanup' option to
> -additionally destroy the partition table.
> -
> -[source,bash]
> -----
> -pveceph osd destroy <ID>
> -----
> -
> -WARNING: The above command will destroy all data on the disk!
> +If you experience problems with an OSD or its disk, try to
> +xref:pve_ceph_mon_and_ts[troubleshoot] them first to decide if a
> +xref:pve_ceph_osd_replace[replacement] is needed.
> +
> +To destroy an OSD:
> +
> +. Either open the web interface and select any {pve} node in the tree
> +view, or open a shell on the node where the OSD to be deleted is
> +located.
> +
> +. Go to the __Ceph -> OSD__ panel (`ceph osd df tree`). If the OSD to
> +be deleted is still `up` and `in` (non-zero value at `AVAIL`), make
> +sure that all OSDs have their `Used (%)` value well below the
> +`nearfull_ratio` of default `85%`. In this way you can reduce the risk
> +from the upcoming rebalancing, which may cause OSDs to run full and
> +thereby blocking I/O on Ceph pools.
> +
> +. If the deletable OSD is not `out` yet, select the OSD and click on
> +**Out** (`ceph osd out <id>`). This will exclude it from data
> +distribution and starts a rebalance.
> +
> +. Click on **Stop**, and if a warning appears, click on **Cancel** and
> +try again shortly afterwards. When using the shell, check if it is

What kind of warning can appear in this case here, though? Is there
something that the user could perhaps miss, if they just proceed to
click on Cancel?

> +safe to stop by reading the output from `ceph osd ok-to-stop <id>`,
> +once true, run `pveceph stop --service osd.<id>` .
> +
> +. **Attention, this step removes the OSD from Ceph and deletes all

Would prefer to keep the "WARNING: " here instead of "Attention, ",
personally.

> +disk data.** To continue, first click on **More -> Destroy**. Use the
> +cleanup option to clean up the partition table and similar, enabling
> +an immediate reuse of the disk in {pve}. Finally, click on **Remove**
> +(`pveceph osd destroy <id> [--cleanup]`).
>  
>  
>  [[pve_ceph_pools]]