Example RAID setups
Our pilot 24-disk system could for example be organized as follows:
(here M x (N + R) also written as MxNrR means: M stripes of N data disks plus R reserve disks (mirror or parity))
| setup | abbreviation | optimized for |
|---|---|---|
| 1 x (21 + 3) | 1x21r3 | capacity |
| 4 x (5 + 1) | 4x5r1 | capacity, stream read |
| 8 x (2 + 1) | 8x2r1 | stream write, random read |
| 4 x (4 + 2) | 4x4r2 | stream write, stream read |
| 12 x (1 + 1) | 12x1r1 | stream write, stream read, random read |
To calculate the usable space simply multiply the number of data disks by the capacity of a single HDD, e.g. for a 4 x (5 + 1) setup (4x5r1), one has 4*5=20 data disks and thus 240TByte usable capacity - assuming 12TByte disks. In reality, due to differences of calculating/displaying values, base1000 versus base1024 the real net capacity may be different by a few percent.
Benchmarks
The usual candidates for testing:
fio results on ZFS 24 HDD
Stream write
Results are in GByte/s
| ↓setup ; files(parallelism)→ | 12 | 24 | 48 | 96 | 192 | 384 | 768 | 1536 | 3072 | 6144 |
|---|---|---|---|---|---|---|---|---|---|---|
| 12x1r1 | 1.50 (±0.04) | 1.48 (±0.04) | 1.50 (±0.03) | 1.50 (±0.04) | 1.49 (±0.04) | 1.49 (±0.03) | 1.50 (±0.05) | 1.49 (±0.04) | 1.49 (±0.06) | 1.25 (±0.08) |
| 1x21r3 | 0.79 (±0.04) | 0.79 (±0.05) | 0.78 (±0.05) | 0.79 (±0.05) | 0.80 (±0.07) | 0.79 (±0.06) | 0.80 (±0.05) | 0.80 (±0.07) | 0.79 (±0.06) | 0.65 (±0.05) |
| 2x10r2 | 1.13 (±0.07) | 1.09 (±0.06) | 1.11 (±0.09) | 1.09 (±0.10) | 1.10 (±0.09) | 1.09 (±0.10) | 1.11 (±0.08) | 1.11 (±0.09) | 1.08 (±0.10) | 0.89 (±0.07) |
| 3x6r2 | 1.33 (±0.08) | 1.28 (±0.11) | 1.30 (±0.08) | 1.30 (±0.07) | 1.30 (±0.07) | 1.31 (±0.08) | 1.31 (±0.08) | 1.29 (±0.08) | 1.30 (±0.11) | 1.05 (±0.10) |
| 4x4r2 | 1.54 (±0.09) | 1.52 (±0.11) | 1.54 (±0.10) | 1.51 (±0.12) | 1.53 (±0.11) | 1.54 (±0.10) | 1.53 (±0.09) | 1.53 (±0.10) | 1.52 (±0.13) | 1.24 (±0.12) |
| 4x5r1 | 1.42 (±0.10) | 1.38 (±0.08) | 1.37 (±0.09) | 1.39 (±0.08) | 1.38 (±0.08) | 1.39 (±0.10) | 1.41 (±0.10) | 1.36 (±0.09) | 1.40 (±0.14) | 1.13 (±0.12) |
| 6x3r1 | 1.47 (±0.11) | 1.44 (±0.10) | 1.44 (±0.10) | 1.45 (±0.10) | 1.41 (±0.10) | 1.44 (±0.09) | 1.42 (±0.10) | 1.44 (±0.10) | 1.41 (±0.18) | 1.19 (±0.11) |
| 8x2r1 | 1.85 (±0.05) | 1.82 (±0.07) | 1.82 (±0.06) | 1.84 (±0.04) | 1.83 (±0.06) | 1.83 (±0.05) | 1.85 (±0.05) | 1.85 (±0.05) | 1.82 (±0.09) | 1.39 (±0.11) |
Stream read
Results are in GByte/s
| ↓setup ; files(parallelism)→ | 12 | 24 | 48 | 96 | 192 | 384 | 768 |
|---|---|---|---|---|---|---|---|
| 12x1r1 | 2.67 (±0.18) | 2.66 (±0.17) | 2.68 (±0.28) | 2.64 (±0.36) | 2.45 (±0.50) | 1.75 (±1.86) | 1.52 (±0.48) |
| 1x21r3 | 1.92 (±0.17) | 1.91 (±0.16) | 1.87 (±0.18) | 1.90 (±0.18) | 1.88 (±0.24) | 1.65 (±0.61) | 0.33 (±0.46) |
| 2x10r2 | 1.93 (±0.20) | 1.95 (±0.18) | 1.96 (±0.18) | 1.95 (±0.20) | 1.93 (±0.25) | 1.91 (±0.64) | 0.16 (±0.22) |
| 3x6r2 | 1.90 (±0.25) | 1.90 (±0.24) | 1.91 (±0.22) | 1.91 (±0.22) | 1.88 (±0.27) | 1.86 (±0.60) | 2.09 (±1.11) |
| 4x4r2 | 2.44 (±0.44) | 2.45 (±0.39) | 2.48 (±0.42) | 2.46 (±0.50) | 2.58 (±0.86) | 2.75 (±0.97) | 3.09 (±1.54) |
| 4x5r1 | 2.07 (±0.22) | 2.07 (±0.23) | 2.07 (±0.23) | 2.07 (±0.25) | 2.03 (±0.31) | 2.01 (±0.66) | 0.87 (±0.75) |
| 6x3r1 | 1.63 (±0.24) | 1.63 (±0.25) | 1.62 (±0.24) | 1.64 (±0.23) | 1.63 (±0.24) | 1.58 (±0.42) | 1.72 (±0.86) |
| 8x2r1 | 1.70 (±0.17) | 1.70 (±0.17) | 1.70 (±0.17) | 1.69 (±0.18) | 1.68 (±0.18) | 1.66 (±0.43) | 1.76 (±0.95) |
Random IO
Results are I/O per second
| ↓setup ; files(parallelism)→ | 12 | 24 | 48 | 96 | 192 | 384 | 768 |
|---|---|---|---|---|---|---|---|
| 12x1r1 | 2016 (±2580) | 1596 (±36) | 1656 (±108) | 1680 (±168) | 1596 (±36) | 1572 (±72) | 1584 (±24) |
| 1x21r3 | 876 (±6288) | 252 (±24) | 264 (±12) | 252 (±24) | 264 (±12) | 264 (±12) | 264 (±12) |
| 2x10r2 | 528 (±804) | 444 (±36) | 516 (±72) | 456 (±12) | 456 (±24) | 456 (±12) | 456 (±12) |
| 3x6r2 | 732 (±1176) | 612 (±24) | 672 (±96) | 624 (±24) | 612 (±24) | 612 (±24) | 612 (±24) |
| 4x4r2 | 1008 (±1428) | 744 (±48) | 852 (±96) | 768 (±24) | 756 (±36) | 768 (±24) | 768 (±24) |
| 4x5r1 | 768 (±252) | 684 (±24) | 768 (±48) | 696 (±24) | 684 (±48) | 684 (±24) | 696 (±24) |
| 6x3r1 | 1188 (±2124) | 900 (±24) | 996 (±132) | 864 (±72) | 900 (±24) | 900 (±24) | 888 (±24) |
| 8x2r1 | 1308 (±1908) | 1080 (±24) | 1152 (±108) | 1104 (±96) | 1080 (±36) | 1056 (±72) | 1080 (±24) |
Capacity
| setup | available capacity [TB] |
|---|---|
| 12x1r1 | 127 |
| 1x21r3 | 203 |
| 2x10r2 | 193 |
| 3x6r2 | 180 |
| 4x4r2 | 169 |
| 4x5r1 | 203 |
| 6x3r1 | 184 |
| 8x2r1 | 169 |