[PVE-User] Proxmox/Ceph problem with Broadcom SAS3008?

[Tobias Kropf]

Hi @ list

We have a new Supermirco Server with SAS HBA AOC-S3008L-L8e.

http://www.supermicro.com/products/accessories/addon/AOC-S3008L-L8e.cfm

Now the pve is in the cluster und ceph is installed... the OSD have a 
havey Latency... I removed the OSD from this server from the root 
section on CRUSH MAP:

-13 66.48984 host pve07
  44  7.26999     osd.44          up  1.00000          1.00000
  52  7.26999     osd.52          up  1.00000          1.00000
  50  7.26999     osd.50          up  1.00000          1.00000
  48  7.26999     osd.48          up  1.00000          1.00000
  40  3.62999     osd.40          up  1.00000          1.00000
  42  7.26999     osd.42          up  1.00000          1.00000
  36  3.62999     osd.36          up  1.00000          1.00000
  34  3.62999     osd.34          up  1.00000          1.00000
  46  7.26999     osd.46          up  1.00000          1.00000
  54  7.26999     osd.54          up  1.00000          1.00000
  28  0.35999     osd.28          up  1.00000          1.00000
  30  0.35999     osd.30          up  1.00000          1.00000
  32  0.35999     osd.32          up  1.00000          1.00000
  38  3.62999     osd.38          up  1.00000          1.00000
  -7 66.48984 host pve06
  33  3.62999     osd.33          up  1.00000          1.00000
  35  3.62999     osd.35          up  1.00000          1.00000
  37  3.62999     osd.37          up  1.00000          1.00000
  39  3.62999     osd.39          up  1.00000          1.00000
  41  7.26999     osd.41          up  1.00000          1.00000
  43  7.26999     osd.43          up  1.00000          1.00000
  45  7.26999     osd.45          up  1.00000          1.00000
  47  7.26999     osd.47          up  1.00000          1.00000
  49  7.26999     osd.49          up  1.00000          1.00000
  51  7.26999     osd.51          up  1.00000          1.00000
  53  7.26999     osd.53          up  1.00000          1.00000
  27  0.35999     osd.27          up  1.00000          1.00000
  29  0.35999     osd.29          up  1.00000          1.00000
  31  0.35999     osd.31          up  1.00000          1.00000
  -1 32.72995 root default
  -2  6.54999     host pve01
   0  1.64000         osd.0       up  1.00000          1.00000
   3  1.64000         osd.3       up  1.00000          1.00000
  17  1.09000         osd.17      up  1.00000          1.00000
  18  1.09000         osd.18      up  1.00000          1.00000
  19  1.09000         osd.19      up  1.00000          1.00000
  -3  6.54999     host pve02

The OSDs have nothing to do but all have high latency:


27                     0                    7
  28                     0                    8
  29                     1                   10
  30                     0                    6
  31                     0                    6
  32                     0                    8
  33                     6                  105
  34                    22                   43
  35                     0                    6
  36                    21                   48
  37                     0                   77
  38                    22                   76
  39                     1                  113
  40                    20                   37
  41                    10                  135
  42                    70                  120
  43                     3                  117
  44                    77                  131
  45                     1                  160
  46                    67                  113
  47                     1                  111
  48                    64                  109
  49                     2                  154
  50                    65                  111
  51                     1                  114
  52                    72                  107
  53                     0                  195
  54                    70                  117

Have anyone an idear?

The OSD are 8TB disks and have XFS fs...

Thanks