Obsolete:Thumper benchmarks
I ran some testing on the new thumpers using filebench. I tested Solaris 10 Update 5, all current patches installed, with ZFS. The machines have 48 250GB disks, configured as two root disks, 44 data disks, and two hotspares (unused in this test). It has 2 dual-core 2.8GHz Opterons and 16GB RAM.
I first tested with a workload designed to simulate the images workload on amane. This started 2 "uploader" threads; these deleted one file, created two new files (of a random size up to 1MB), then slept for 2 seconds. Concurrently, it started 100 "webservers" threads; these opened a random file, read its entire contents in 64KB blocks, then closed it.
I prepared the benchmark by creating 25,000 files in a filesystem, totalling about 25GB. Notice this size exceeds the system's RAM.
Over a 1200-second run, the system achieved 1MB/sec write performance (expected given the low workload) and 726MB/sec read performance. On average, 725 entire file reads per second was achieved (this suggests that file size did not greatly affect the performance; probably caused by ZFS's aggressive read-ahead).
Next I ran the same benchmark with 20 creator threads (that is, 10 times the write workload) and ran for 1200 seconds again. This produced 10MB/sec writes and 900MB/sec reads (possibly because of caching? need to re-run this one.)
Then I re-ran both benchmarks with an fsync() after every file write (this is a more accurate simulation of NFS, which requires synchronous writes). This produced no apparent performance decrease in either benchmark, although significantly more disk writing was noticed in 'zpool iostat'.
I ran the benchmark again while taking a 'zfs snapshot' every 30 seconds (to simulate filesystem replication to another host). I tested with the snapshots only, and with an incremental 'zfs send' after each snapshot. This seemed to have no impact on performance.
I ran the same benchmark over NFS, with filebench running on ms2. This achieved 121MB/sec (close to the GE limit) and 120 operations/sec. It would be interesting to try this with smaller files.
I changed the test to use <= 64KB files instead of <= 1MB. Although the throughput went down (478MB/sec), file operations (open, read, close) went up to 7685/sec. Over NFS, this gave 52MB/sec, and 2.500 file ops/sec.
Comparison of RAID 10 and RAID Z
Test setup
- RAID 10
- 22 pairs of mirrored disks concatenated to one volume. Two hot spares
- RAID Z
- 9 RAID Z sets of 5 disks concatenated to one volume. One hot spare
Workload script
set $dir=/export/filebench
set $nfiles=500000
set $meandirwidth=1000
set $filesize=128k
set $nthreads=200
set $readiosize=32k
set $ncreators=8
define fileset name=bigfileset,path=$dir,size=$filesize,entries=$nfiles,dirwidth=$meandirwidth,prealloc=50,reuse
define process name=images,instances=1
{
thread name=uploads,memsize=10m,instances=$ncreators
{
flowop deletefile name=deletefile1,filesetname=bigfileset
flowop createfile name=createfile1,filesetname=bigfileset,fd=1
flowop appendfilerand name=appendfilerand1,iosize=$filesize,fd=1
flowop fsync name=fsync1,fd=1
flowop closefile name=closefile1,fd=1
flowop createfile name=createfile2,filesetname=bigfileset,fd=1
flowop appendfilerand name=appendfilerand2,iosize=$filesize,fd=1
flowop fsync name=fsync2,fd=1
flowop closefile name=closefile2,fd=1
flowop delay name=wait,value=2
}
thread name=webserver,memsize=10m,instances=$nthreads
{
flowop openfile name=openfile2,filesetname=bigfileset,opennext,fd=1
flowop readwholefile name=readfile2,fd=1,iosize=$readiosize
flowop closefile name=closefile2,fd=1
flowop opslimit name=limit
}
}
RAID 10 Results
limit 0ops/s 0.0mb/s 0.0ms/op 0us/op-cpu closefile2 21698ops/s 0.0mb/s 0.0ms/op 5us/op-cpu readfile2 21695ops/s 2709.8mb/s 4.0ms/op 126us/op-cpu openfile2 21695ops/s 0.0mb/s 3.0ms/op 36us/op-cpu wait 3ops/s 0.0mb/s 2108.9ms/op 26us/op-cpu closefile2 0ops/s 0.0mb/s 0.0ms/op 0us/op-cpu fsync2 3ops/s 0.0mb/s 168.9ms/op 217us/op-cpu appendfilerand2 3ops/s 0.2mb/s 1.4ms/op 147us/op-cpu createfile2 3ops/s 0.0mb/s 23.0ms/op 162us/op-cpu closefile1 3ops/s 0.0mb/s 0.3ms/op 6us/op-cpu fsync1 3ops/s 0.0mb/s 214.2ms/op 176us/op-cpu appendfilerand1 3ops/s 0.2mb/s 0.6ms/op 140us/op-cpu createfile1 3ops/s 0.0mb/s 2.5ms/op 129us/op-cpu deletefile1 3ops/s 0.0mb/s 1.7ms/op 112us/op-cpu 15064: 3527.288: IO Summary: 39458334 ops 65111.8 ops/s, (21695/6 r/w) 2710.2mb/s, 173us cpu/op, 7.0ms latency 15064: 3527.288: Shutting down processes
RAID Z Results
22288: 3300.576: Per-Operation Breakdown limit 0ops/s 0.0mb/s 0.0ms/op 0us/op-cpu closefile2 22927ops/s 0.0mb/s 0.0ms/op 4us/op-cpu readfile2 22924ops/s 2860.0mb/s 3.7ms/op 122us/op-cpu openfile2 22924ops/s 0.0mb/s 2.5ms/op 36us/op-cpu wait 3ops/s 0.0mb/s 2088.8ms/op 26us/op-cpu closefile2 0ops/s 0.0mb/s 0.0ms/op 0us/op-cpu fsync2 3ops/s 0.0mb/s 191.0ms/op 325us/op-cpu appendfilerand2 3ops/s 0.2mb/s 1.3ms/op 141us/op-cpu createfile2 3ops/s 0.0mb/s 15.9ms/op 154us/op-cpu closefile1 3ops/s 0.0mb/s 0.0ms/op 5us/op-cpu fsync1 3ops/s 0.0mb/s 213.8ms/op 278us/op-cpu appendfilerand1 3ops/s 0.2mb/s 0.7ms/op 137us/op-cpu createfile1 3ops/s 0.0mb/s 2.3ms/op 123us/op-cpu deletefile1 3ops/s 0.0mb/s 2.1ms/op 125us/op-cpu 22288: 3300.576: IO Summary: 41692903 ops 68800.3 ops/s, (22924/6 r/w) 2860.4mb/s, 165us cpu/op, 6.4ms latency
Summary
RAIDZ seems to be a little bit faster than RAID10. With RAID10, 4.9 TB of diskspace are available, with RAIDZ there are 8 TB of diskspace available. All in all, RAIDZ seems to be preferable.