• Example RAID setups
• Benchmarks
  • fio results on ZFS 24 HDD
    • Stream write
    • Stream read
    • Random IO
    • Capacity

Example RAID setups

Our pilot 24-disk system could for example be organized as follows:

(here M x (N + R) also written as MxNrR means: M stripes of N data disks plus R reserve disks (mirror or parity))

setup	abbreviation	optimized for
1 x (21 + 3)	1x21r3	capacity
4 x (5 + 1)	4x5r1	capacity, stream read
8 x (2 + 1)	8x2r1	stream write, random read
4 x (4 + 2)	4x4r2	stream write, stream read
12 x (1 + 1)	12x1r1	stream write, stream read, random read

To calculate the usable space simply multiply the number of data disks by the capacity of a single HDD, e.g. for a 4 x (5 + 1) setup (4x5r1), one has 4*5=20 data disks and thus 240TByte usable capacity - assuming 12TByte disks. In reality, due to differences of calculating/displaying values, base1000 versus base1024 the real net capacity may be different by a few percent.

Benchmarks

The usual candidates for testing:

• fio
• iozone
• bonnie++
• io500

fio results on ZFS 24 HDD

Stream write

Results are in GByte/s

↓setup ; files(parallelism)→	12	24	48	96	192	384	768	1536	3072	6144
12x1r1	1.50 (±0.04)	1.48 (±0.04)	1.50 (±0.03)	1.50 (±0.04)	1.49 (±0.04)	1.49 (±0.03)	1.50 (±0.05)	1.49 (±0.04)	1.49 (±0.06)	1.25 (±0.08)
1x21r3	0.79 (±0.04)	0.79 (±0.05)	0.78 (±0.05)	0.79 (±0.05)	0.80 (±0.07)	0.79 (±0.06)	0.80 (±0.05)	0.80 (±0.07)	0.79 (±0.06)	0.65 (±0.05)
2x10r2	1.13 (±0.07)	1.09 (±0.06)	1.11 (±0.09)	1.09 (±0.10)	1.10 (±0.09)	1.09 (±0.10)	1.11 (±0.08)	1.11 (±0.09)	1.08 (±0.10)	0.89 (±0.07)
3x6r2	1.33 (±0.08)	1.28 (±0.11)	1.30 (±0.08)	1.30 (±0.07)	1.30 (±0.07)	1.31 (±0.08)	1.31 (±0.08)	1.29 (±0.08)	1.30 (±0.11)	1.05 (±0.10)
4x4r2	1.54 (±0.09)	1.52 (±0.11)	1.54 (±0.10)	1.51 (±0.12)	1.53 (±0.11)	1.54 (±0.10)	1.53 (±0.09)	1.53 (±0.10)	1.52 (±0.13)	1.24 (±0.12)
4x5r1	1.42 (±0.10)	1.38 (±0.08)	1.37 (±0.09)	1.39 (±0.08)	1.38 (±0.08)	1.39 (±0.10)	1.41 (±0.10)	1.36 (±0.09)	1.40 (±0.14)	1.13 (±0.12)
6x3r1	1.47 (±0.11)	1.44 (±0.10)	1.44 (±0.10)	1.45 (±0.10)	1.41 (±0.10)	1.44 (±0.09)	1.42 (±0.10)	1.44 (±0.10)	1.41 (±0.18)	1.19 (±0.11)
8x2r1	1.85 (±0.05)	1.82 (±0.07)	1.82 (±0.06)	1.84 (±0.04)	1.83 (±0.06)	1.83 (±0.05)	1.85 (±0.05)	1.85 (±0.05)	1.82 (±0.09)	1.39 (±0.11)

Stream read

Results are in GByte/s

↓setup ; files(parallelism)→	12	24	48	96	192	384	768
12x1r1	2.67 (±0.18)	2.66 (±0.17)	2.68 (±0.28)	2.64 (±0.36)	2.45 (±0.50)	1.75 (±1.86)	1.52 (±0.48)
1x21r3	1.92 (±0.17)	1.91 (±0.16)	1.87 (±0.18)	1.90 (±0.18)	1.88 (±0.24)	1.65 (±0.61)	0.33 (±0.46)
2x10r2	1.93 (±0.20)	1.95 (±0.18)	1.96 (±0.18)	1.95 (±0.20)	1.93 (±0.25)	1.91 (±0.64)	0.16 (±0.22)
3x6r2	1.90 (±0.25)	1.90 (±0.24)	1.91 (±0.22)	1.91 (±0.22)	1.88 (±0.27)	1.86 (±0.60)	2.09 (±1.11)
4x4r2	2.44 (±0.44)	2.45 (±0.39)	2.48 (±0.42)	2.46 (±0.50)	2.58 (±0.86)	2.75 (±0.97)	3.09 (±1.54)
4x5r1	2.07 (±0.22)	2.07 (±0.23)	2.07 (±0.23)	2.07 (±0.25)	2.03 (±0.31)	2.01 (±0.66)	0.87 (±0.75)
6x3r1	1.63 (±0.24)	1.63 (±0.25)	1.62 (±0.24)	1.64 (±0.23)	1.63 (±0.24)	1.58 (±0.42)	1.72 (±0.86)
8x2r1	1.70 (±0.17)	1.70 (±0.17)	1.70 (±0.17)	1.69 (±0.18)	1.68 (±0.18)	1.66 (±0.43)	1.76 (±0.95)

Random IO

Results are I/O per second

↓setup ; files(parallelism)→	12	24	48	96	192	384	768
12x1r1	2016 (±2580)	1596 (±36)	1656 (±108)	1680 (±168)	1596 (±36)	1572 (±72)	1584 (±24)
1x21r3	876 (±6288)	252 (±24)	264 (±12)	252 (±24)	264 (±12)	264 (±12)	264 (±12)
2x10r2	528 (±804)	444 (±36)	516 (±72)	456 (±12)	456 (±24)	456 (±12)	456 (±12)
3x6r2	732 (±1176)	612 (±24)	672 (±96)	624 (±24)	612 (±24)	612 (±24)	612 (±24)
4x4r2	1008 (±1428)	744 (±48)	852 (±96)	768 (±24)	756 (±36)	768 (±24)	768 (±24)
4x5r1	768 (±252)	684 (±24)	768 (±48)	696 (±24)	684 (±48)	684 (±24)	696 (±24)
6x3r1	1188 (±2124)	900 (±24)	996 (±132)	864 (±72)	900 (±24)	900 (±24)	888 (±24)
8x2r1	1308 (±1908)	1080 (±24)	1152 (±108)	1104 (±96)	1080 (±36)	1056 (±72)	1080 (±24)

Capacity

setup	available capacity [TB]
12x1r1	127
1x21r3	203
2x10r2	193
3x6r2	180
4x4r2	169
4x5r1	203
6x3r1	184
8x2r1	169