[PVE-User] Storage performance data

[Eneko Lacunza]

Hi all,

I have been doing some tests with two proxmox servers in preparation of 
a deployment in a client, and I thought it could be useful to share them.

Please don't take this scenario as a good implementation of Proxmox nor 
a good repeatable benchmark; it's just a setup influenced by other 
(external) factors and not a totally new setup both on client and for 
the tests (we're reusing existing hardware). Its objective was to make a 
rough comparison of disk performance.

Test scenario:

- Server A: Dell T610, 1xIntel Xeon E5520, 32 GB RAM, H200 RAID 
controller, 2x500GB SATA, 2x1TB SATA, 2xSSD Intel 320 300G
- Server B: IBM System x3400 7976, 1xIntel Xeon E5410, 12 GB RAM, IBM 
ServeRaid 8k 256 MB BBC, 2x500GB SATA, 2x1TB SATA
- Network: back-to-back 1G ethernet connection between both servers mtu 
9000, additional 1G ethernet for general network access (ssh, 
webinterface, VMs) in each server. Onboard NIC on Server A (Broadcom), 
intel pro/1000 on Server B.

Servers have working VMs running on them. There's low load and tests 
have been performed more than once and only the best is reported; 
there's plenty of RAM and CPU free on both servers. Disks used in the 
tests are not being used neither by VMs nor the base Proxmox system.

Test results:

[SERVER B]
-- 450 GB partition on 2x500 HW RAID1, XFS
CPU BOGOMIPS:      18620.04
REGEX/SECOND:      913801
HD SIZE:           442.16 GB (/dev/sdb3)
BUFFERED READS:    147.15 MB/sec
AVERAGE SEEK TIME: 12.27 ms
FSYNCS/SECOND:     4386.37
DNS EXT:           95.34 ms

-- 450 GB partition on 2x500 HW RAID1, EXT4
CPU BOGOMIPS:      18620.04
REGEX/SECOND:      894205
HD SIZE:           435.44 GB (/dev/sdb3)
BUFFERED READS:    161.24 MB/sec
AVERAGE SEEK TIME: 11.82 ms
FSYNCS/SECOND:     1797.41
DNS EXT:           229.17 ms

[SERVER A]
-- 1xSSD EXT4 (non-raid on H200 port) noop scheduler
CPU BOGOMIPS:      36176.88
REGEX/SECOND:      761723
HD SIZE:           275.08 GB (/dev/sdc)
BUFFERED READS:    211.47 MB/sec
AVERAGE SEEK TIME: 0.23 ms
FSYNCS/SECOND:     1404.39
DNS EXT:           151.66 ms

-- 1xSSD XFS (non-raid on H200 port) noop scheduler
CPU BOGOMIPS:      36176.88
REGEX/SECOND:      758984
HD SIZE:           279.32 GB (/dev/sdc)
BUFFERED READS:    174.78 MB/sec
AVERAGE SEEK TIME: 0.60 ms
FSYNCS/SECOND:     2819.07
DNS EXT:           464.62 ms

(SATA drives on Server A for completeness, not used on final tests)
- partition on H200 RAID1 2x500GB SATA, XFS
CPU BOGOMIPS:      36176.88
REGEX/SECOND:      805542
HD SIZE:           22.36 GB (/dev/sda2)
BUFFERED READS:    111.19 MB/sec
AVERAGE SEEK TIME: 6.70 ms
FSYNCS/SECOND:     117.84
DNS EXT:           519.26 ms

- partition on H200 RAID1 2x500GB SATA, EXT4
Seems I lost this data, buffered reads where around 135 MB/s and fsync/s 
was about 65.

FINAL TEST SETUP
- SERVER B is iSCSI target, exports 450GB partition on RAID 2x500GB as a lun
- SERVER A is iSCSI initiator, iSCSI uses back-to-back 1G ethernet mtu9000

[SERVER A]
-- 100GB LV on iSCSI disk, EXT4
CPU BOGOMIPS:      36176.88
REGEX/SECOND:      754021
HD SIZE:           98.43 GB (/dev/mapper/grupo_iscsi-tests)
BUFFERED READS:    102.46 MB/sec
AVERAGE SEEK TIME: 10.08 ms
FSYNCS/SECOND:     821.93
DNS EXT:           383.01 ms

-- 275GB MDRAID1 (SSD +  iSCSI) + bitmap + --write-mostly, EXT4
CPU BOGOMIPS:      36176.88
REGEX/SECOND:      799245
HD SIZE:           275.08 GB (/dev/md0)
BUFFERED READS:    206.95 MB/sec
AVERAGE SEEK TIME: 0.23 ms
FSYNCS/SECOND:     763.87
DNS EXT:           119.05 ms

-- 275GB MDRAID1 (SSD +  iSCSI) + bitmap + --write-mostly + 
--write-behind, EXT4
CPU BOGOMIPS:      36176.88
REGEX/SECOND:      780966
HD SIZE:           275.08 GB (/dev/md0)
BUFFERED READS:    207.35 MB/sec
AVERAGE SEEK TIME: 0.24 ms
FSYNCS/SECOND:     1147.99
DNS EXT:           145.67 ms

-- 275GB MDRAID1 (SSD +  iSCSI) + bitmap + --write-mostly + 
--write-behind, XFS
CPU BOGOMIPS:      36176.88
REGEX/SECOND:      790823
HD SIZE:           279.32 GB (/dev/md0)
BUFFERED READS:    152.35 MB/sec
AVERAGE SEEK TIME: 0.60 ms
FSYNCS/SECOND:     2650.10
DNS EXT:           122.49 ms


Where nothing is noted, defaults where used (ext4/xfs mount options, 
write-behind max operations, ...). EXT4 filesystems where all created 
with -m 0 (0% reserved)

Any comment or thought is appreciated :)

Cheers
Eneko

-- 
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943575997
       943493611
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es