I'm performing some FIO random read 4k I/O benchmarks on a ZFS file system. So since I didn't trust the numbers I got, I wanted to know how many of the IOPs I got were due to cache hits rather than disk hits.
This is why I wrote a small shell script called archhitratio.
Sample output:
IOPs: 133 | ARC cache hit ratio: 48.00 % | Hitrate: 64 / Missrate: 69
IOPs: 131 | ARC cache hit ratio: 48.00 % | Hitrate: 63 / Missrate: 68
IOPs: 136 | ARC cache hit ratio: 49.00 % | Hitrate: 67 / Missrate: 69
IOPs: 128 | ARC cache hit ratio: 46.00 % | Hitrate: 59 / Missrate: 69
IOPs: 127 | ARC cache hit ratio: 46.00 % | Hitrate: 59 / Missrate: 68
IOPs: 135 | ARC cache hit ratio: 48.00 % | Hitrate: 65 / Missrate: 70
IOPs: 127 | ARC cache hit ratio: 45.00 % | Hitrate: 58 / Missrate: 69
IOPs: 125 | ARC cache hit ratio: 44.00 % | Hitrate: 56 / Missrate: 69
IOPs: 128 | ARC cache hit ratio: 46.00 % | Hitrate: 60 / Missrate: 68
In this example, I'm performing a random read test on a 16GB data set. This host has 16 GB RAM and 6 GB of this dataset was already in memory from previous FIO runs. This is why we see a ~45% hit ratio.
This is a more interesting result:
IOPs: 1404 | ARC cache hit ratio: 90.0 % | Hitrate: 1331 / Missrate: 73
IOPs: 1425 | ARC cache hit ratio: 90.0 % | Hitrate: 1350 / Missrate: 75
IOPs: 1395 | ARC cache hit ratio: 90.0 % | Hitrate: 1323 / Missrate: 72
IOPs: 1740 | ARC cache hit ratio: 90.0 % | Hitrate: 1664 / Missrate: 76
IOPs: 1351 | ARC cache hit ratio: 90.0 % | Hitrate: 1277 / Missrate: 74
IOPs: 1613 | ARC cache hit ratio: 90.0 % | Hitrate: 1536 / Missrate: 77
IOPs: 1920 | ARC cache hit ratio: 90.0 % | Hitrate: 1845 / Missrate: 75
IOPs: 1431 | ARC cache hit ratio: 90.0 % | Hitrate: 1354 / Missrate: 77
IOPs: 1675 | ARC cache hit ratio: 90.0 % | Hitrate: 1598 / Missrate: 77
IOPs: 1560 | ARC cache hit ratio: 90.0 % | Hitrate: 1484 / Missrate: 76
IOPs: 1574 | ARC cache hit ratio: 90.0 % | Hitrate: 1500 / Missrate: 74
IOPs: 2017 | ARC cache hit ratio: 90.0 % | Hitrate: 1946 / Missrate: 71
IOPs: 1696 | ARC cache hit ratio: 90.0 % | Hitrate: 1623 / Missrate: 73
IOPs: 1776 | ARC cache hit ratio: 90.0 % | Hitrate: 1702 / Missrate: 74
IOPs: 1671 | ARC cache hit ratio: 90.0 % | Hitrate: 1597 / Missrate: 74
IOPs: 1729 | ARC cache hit ratio: 90.0 % | Hitrate: 1656 / Missrate: 73
IOPs: 1902 | ARC cache hit ratio: 90.0 % | Hitrate: 1828 / Missrate: 74
IOPs: 2029 | ARC cache hit ratio: 90.0 % | Hitrate: 1956 / Missrate: 73
IOPs: 2228 | ARC cache hit ratio: 90.0 % | Hitrate: 2161 / Missrate: 67
IOPs: 2289 | ARC cache hit ratio: 90.0 % | Hitrate: 2216 / Missrate: 73
IOPs: 2385 | ARC cache hit ratio: 90.0 % | Hitrate: 2277 / Missrate: 108
IOPs: 2595 | ARC cache hit ratio: 90.0 % | Hitrate: 2524 / Missrate: 71
IOPs: 2940 | ARC cache hit ratio: 90.0 % | Hitrate: 2872 / Missrate: 68
IOPs: 2984 | ARC cache hit ratio: 90.0 % | Hitrate: 2872 / Missrate: 112
IOPs: 2622 | ARC cache hit ratio: 90.0 % | Hitrate: 2385 / Missrate: 237
IOPs: 1518 | ARC cache hit ratio: 90.0 % | Hitrate: 1461 / Missrate: 57
IOPs: 3221 | ARC cache hit ratio: 90.0 % | Hitrate: 3150 / Missrate: 71
IOPs: 3745 | ARC cache hit ratio: 90.0 % | Hitrate: 3674 / Missrate: 71
IOPs: 3363 | ARC cache hit ratio: 90.0 % | Hitrate: 3292 / Missrate: 71
IOPs: 3931 | ARC cache hit ratio: 90.0 % | Hitrate: 3856 / Missrate: 75
IOPs: 3765 | ARC cache hit ratio: 90.0 % | Hitrate: 3689 / Missrate: 76
IOPs: 4845 | ARC cache hit ratio: 90.0 % | Hitrate: 4772 / Missrate: 73
IOPs: 4422 | ARC cache hit ratio: 90.0 % | Hitrate: 4350 / Missrate: 72
IOPs: 5602 | ARC cache hit ratio: 90.0 % | Hitrate: 5531 / Missrate: 71
IOPs: 5351 | ARC cache hit ratio: 90.0 % | Hitrate: 5279 / Missrate: 72
IOPs: 6075 | ARC cache hit ratio: 90.0 % | Hitrate: 6004 / Missrate: 71
IOPs: 6586 | ARC cache hit ratio: 90.0 % | Hitrate: 6515 / Missrate: 71
IOPs: 7974 | ARC cache hit ratio: 90.0 % | Hitrate: 7907 / Missrate: 67
IOPs: 4434 | ARC cache hit ratio: 90.0 % | Hitrate: 4180 / Missrate: 254
IOPs: 9793 | ARC cache hit ratio: 90.0 % | Hitrate: 9721 / Missrate: 72
IOPs: 9395 | ARC cache hit ratio: 90.0 % | Hitrate: 9300 / Missrate: 95
IOPs: 6171 | ARC cache hit ratio: 90.0 % | Hitrate: 6089 / Missrate: 82
IOPs: 9209 | ARC cache hit ratio: 90.0 % | Hitrate: 9142 / Missrate: 67
IOPs: 14883 | ARC cache hit ratio: 90.0 % | Hitrate: 14817 / Missrate: 66
IOPs: 11304 | ARC cache hit ratio: 90.0 % | Hitrate: 11152 / Missrate: 152
IOPs: 228 | ARC cache hit ratio: 30.0 % | Hitrate: 71 / Missrate: 157
IOPs: 8321 | ARC cache hit ratio: 90.0 % | Hitrate: 8072 / Missrate: 249
IOPs: 15550 | ARC cache hit ratio: 90.0 % | Hitrate: 15450 / Missrate: 100
IOPs: 11819 | ARC cache hit ratio: 90.0 % | Hitrate: 11683 / Missrate: 136
IOPs: 28630 | ARC cache hit ratio: 90.0 % | Hitrate: 28367 / Missrate: 263
IOPs: 40484 | ARC cache hit ratio: 90.0 % | Hitrate: 40409 / Missrate: 75
IOPs: 104501 | ARC cache hit ratio: 90.0 % | Hitrate: 103982 / Missrate: 519
IOPs: 164483 | ARC cache hit ratio: 90.0 % | Hitrate: 163997 / Missrate: 486
IOPs: 229729 | ARC cache hit ratio: 90.0 % | Hitrate: 228956 / Missrate: 773
IOPs: 236479 | ARC cache hit ratio: 90.0 % | Hitrate: 235886 / Missrate: 593
IOPs: 249232 | ARC cache hit ratio: 90.0 % | Hitrate: 248836 / Missrate: 396
IOPs: 259156 | ARC cache hit ratio: 90.0 % | Hitrate: 258968 / Missrate: 188
IOPs: 276099 | ARC cache hit ratio: 90.0 % | Hitrate: 275857 / Missrate: 242
IOPs: 249382 | ARC cache hit ratio: 90.0 % | Hitrate: 249287 / Missrate: 95
What does this result mean? The RAM size is 16 GB and the test data size is only 6 GB. If you just continue performing random I/O, eventually all data will be in RAM. I believe that here, you witness the moment when all data is in RAM and the already high IOPs goes through the roof (250K IOPS). However, I cannot explain the increase of the Missrate.
Comments