[PVE-User] SSD caching/File System summary

[Lindsay Mathieson]

Thought I'd do a quick summary of my results - very subjective really, so take 
with a pinch of salt.

As I suspected, raw disk becnhmarks, either from within the VM (Crystal 
DiskMark) or on the host (dd, bonnie++) while interesting, arent' a very good 
guide to actual application performance improvements. In the end I ran a 
number of std application usage timings - a long build process, Eclipse 
startup and builds, UI responsiveness.

With all the different cache strategies I tried, disk bencharks gave widely 
varying results, while app benchmarking they where all very similar.

dm-Cache:
- Complicated to setup
- Fiddly and error prone to manage
- Needs custom init scripts
- No auto ignoring of bulk reads/writes (disk cloning, backups)
- I managed to destroy the underlying file system while attempting to flush 
and dismount the write cache.
- Did give good results for reads/writes

bcache
- User tools have to be compiled and installed
- can't be used with existing file systems
- No auto ignoring of bulk reads/writes (disk cloning, backups)
- Needs custom init scripts
- 3.10 kernel version is hopelessly buggy. Trashed the file system
- No tools for uninstalling. Required a hard result and then I had to use dd 
to overwrite the partition table to remove the cache store. I then blacklisted 
the module :)

EnhanceIO
- Has to be compiled and installed
- *can* be used with existing file systems
- Can be created/edited/destroyed on the fly
- No auto ignoring of bulk reads/writes (disk cloning, backups)
- persistent between reboots (udev rules)
- Good results on reads/writes
- Unfortunately when I combined it with an external ext4 journal I got data 
corruption

ZFS (zfsonlinux.org)
- Used the kernel module
- has a repo, but required kernel headers and dkms.
- Builtin support for journal and read caching using multiple SSD's
- ZFS! with all the zfs goodies, raid, striping, snapshots, backups, pool 
managment.
- Auto ignoring of bulk reads/writes (disk cloning, backups)
- good tools for management and reporting disk/cache stats and errors
- I restricted it to 1GB RAM on Hosts
- Good results on reads/writes
- No support for O_DIRECT so I had to disable glusters io cache, which is 
recommenced anyway for virtual stores.


ZFS was the clear winner - ease of management, reliability, flexibility. Its 
going to make expanding the stores so much easier in the future.

ZFS + Gluster + SSD's caches seems to be a winner for shared HA storage to me.

-- 
Lindsay
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20141115/3a8a14bf/attachment.sig>