peawee dot net

On our SAN at work, we use ZFS across it from a single head node to provide storage virtualization services, which are then exported via NFS to the rest of our machines on the network. At first, performance was a dog, especially on writing many small files to disk. This was indicative of having synchronous disk I/O, where all data is written straight to disk immediately before the writing program can go on, instead of having the data written to a cache in ram (aka asynchronous disk I/O). Synch’ed disk IO can slow down disk speeds a lot, as software accessing the filesystem has to wait for the disk writes to finish before it can go on, and there’s a lot more overhead involved with small files than with big files.

Upon some study, I went and turned off two ZFS module options in /etc/system that had more than a twofold increase in speed, especially when dealing lots of small writes.

set zfs:zil_disable=1
set zfs:zfs_noforcecache=1

These two settings made untarring the Linux source code (dozens of megabytes of small files) go from over 15 minutes for just linux/Documentation, to about 25 seconds. While on the SAN head node it’s under 15 seconds for the same operation, there’s still NFS overhead to worry about.

I also know that using these settings really annoy the ZFS developers, as they rightly should, as it mucks about with internals that effectively neuter ZFS’s very ingenious and effective data protection schemes. One of them, the noforcecache option, is mitigated by our disk controller hardware having battery backups for their on-hardware disk cache. The zil_disable is only backed by the SAN running off of UPS systems, however. Hopefully the ZFS devs can make the ZIL (the ZFS Intent Log) work correctly with NFS in the future without essentially forcing sync’ed IO.