1. What Home NAS Builders Should Understand About Silent Data Corruption

    Thu 23 April 2020

    Introduction

    When it comes to dealing with storage in a DIY NAS context, two important topics come up:

    1. Unrecoverable read errors (UREs) or what old people like me call 'bad sectors'
    2. Silent data corruption (data corruption unnoticed by the storage layers)

    I get a strong impression that people tend to confuse those concepts. However, they often come up when people evaluate their options when they want to buy or build their own do-it-yourself NAS.

    In this article, I want to make a clear distinction between the two and assess their risk. This may help you evaluating these risks and make an informed decision.

    Unrecoverable read errors (due to bad sectors)

    When a hard drive hits a 'bad sector', it means that it can't read the contents of that particular sector anymore.

    If the hard drive is unable to read that data even after multiple attempts, the operating system will return an Unrecoverable Read Error (URE).

    This is an example (on Linux) of a drive experiencing read errors, as pulled from /var/log/syslog (culled a bit for readability):

    sd 0:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 0:0:0:0: [sda] tag#19 Sense Key : Medium Error [current] 
    sd 0:0:0:0: [sda] tag#19 Add. Sense: Unrecovered read error
    sd 0:0:0:0: [sda] tag#19 CDB: Read(10) 28 00 02 1c 8c 00 00 00 98 00
    blk_update_request: critical medium error, dev sda, sector 35425280 op 0x0:(READ)
    sd 0:0:0:0: [sda] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
    sd 0:0:0:0: [sda] tag#16 Sense Key : Medium Error [current] 
    sd 0:0:0:0: [sda] tag#16 Add. Sense: Unrecovered read error
    sd 0:0:0:0: [sda] tag#16 CDB: Read(10) 28 00 02 1c 8d 00 00 00 88 00
    blk_update_request: critical medium error, dev sda, sector 35425536 op 0x0:(READ)
    

    If a sector cannot be read, the data stored in that sector is lost. And in my experience, if you encounter a single bad sector, soon, there will be more. So if this happens, it's time to replace the hard drive.

    We use RAID to protect against drive failure. RAID (no matter the implementation) also can deal with 'partial failure' such as a drive encountering bad sectors.

    In a RAID array, a drive encountering unrecoverable read errors is just kicked out of the array, so it doesn't 'hang' or 'stall' the entire RAID array.

    Please note that this behaviour does depend on the particular RAID solution of choice1. The point is though that bad sectors or UREs are a common event and RAID solutions can deal with them properly.

    The real problem with bad sectors (resulting in UREs) is that they can remain undiscovered until it is too late. So to uncover them in an early state, it's very important to run regular data scrubs. I've written an article specifically about this topic.

    Silent data corruption

    An unrecoverable read error means that we can't read (a portion of) a file. Although it is unfortunate - because we better have an intact backup of that file - we are also fortunate.

    Why are we fortunate?

    We are fortunate because the storage system - the hard drive and in turn the operating system - reported an error. We were able to diagnose the problem an take action.

    But it is possible that bits and bytes get mangled without your hard drive, SATA controller or operating system noticing. Somewhere, somehow, a bit is read or transmitted as a 1 where it should have been a 0.

    This is really bad, because this data corruption is undetected, it is 'silent', there is no notification.

    Because imagine what happens: the corrupted file is happily backed up by your backup software, because it's unaware that anything is wrong. And by the time you discover the data corruption, the original pristine file is no longer part of the backup (rotated out). You are left with a lot of backups of a corrupted file. We encounter dataloss.

    This is one of the scariest kinds of data loss. Because it's very difficult to detect. You'll have to constantly calculate the checksum of a file and verify it's still ok.

    And that's - although rather simplified - exactly what ZFS does (amongst many other things). ZFS uses checksums at the block-level and thus assures with every read if the data contained in the block is still valid. ZFS is one of the few file systems that has this very powerfull feature (BTRFS is another example).

    Regular RAID arrays (be it hardware-based or software-based) cannot detect silent data corruption (although it could be possible with RAID6). So it must be clear that ZFS is capable of protecting against a risk 'regular' RAID cannot cope with.

    Is silent data corruption a significant threat for home DIY NAS builders?

    Although silent data corruption is a very scary threat, from what I can tell there is no significant independant evidence that the risk of silent data corruption is so high that the average home DIY NAS builder should take this risk into account2.

    Maybe I'm wrong, but I think many people mistakenly confuse UREs or unrecoverable read errors (caused by bad sectors) with silent data corruption. And I think that's wrong, because there's nothing silent about an unrecoverable read error.

    The truth is that hard drives are in fact very reliable when it comes to silent data corruption, because they make heavy use of error detection and correction algoritms. A significant portion of the raw capacity of a hard drive is sacrificed to store redundant information to aid in detecting and correcting data corruption. According to wikipedia, hard drives used Reed-Solomon error correction in the past and more modern drives use LDPC.

    These error correction codes asure data integrity. Although 'soft' read errors may occur, there is enough additional redundant information stored on the hard drive to detect errors and even reconstruct the data (to some extend). Your hard drive handles this all by itself, it's part of normal operation.

    So this is my point: it's important to understand that there is a lot of protection against silent data corruption in a hard drive. The risk of silent data corruption is therefore small3.

    Sometimes the read data is so garbled that even the error correction codes cannot reconstruct the data as it was originally stored and that's what we then experience as an unrecoverable read error. But the disk notices! And it will report it!. This is not silent at all!

    To really create silent data corruption, something very special need to happen. And to be very clear: such events do happen. But they are very rare.

    Somehow, a bit must flip and this event is not detected by the error correction algorithm. Maybe the bit flipped in the hard drive cache memory when it was read from the drive. Maybe it flipped during transport over the SATA cable.

    But it's fun to realise that the SATA protocol also has error detection embedded in the protocol for reliable data transmission. It's error detection and correction all the way down.

    The risk that silent data corruption happens is thus very small, especially for home users.

    Again, make no mistake: the risk is real and storage solutions for larger scale storage solutions (SANs / Storage arrays) with hundreds, thousands or tens of thousands of drives do really have to take into account the risk of silent data corruption. At scale, even very small risks become a certainty.

    Enterprise storage solutions often employ their own proprietary solutions to protect against silent data corruption. Although it depend on the particular solution4, it's often part of the storage array. ZFS was revolutionary because they put the data integrity checking in the filesystem itself.

    So if you think the risk of silent data corruption is still high enough that you should protect yourself against it, I would recommend to consider using ECC memory to protect against corrupted data in memory. To be frank: I consider non-ECC memory a more likely cause of silent data corruption than the storage subsystem, which already employs all these error detection and correction algoritms. Non-ECC memory is totally unprotected.

    Anekdote: I myself run a 24-drive NAS based on ZFS and it has been rock-solid for 6 years straight.

    mynasimage

    From time to time, I do run disk 'scrubs', which can take quite some time. Although I have many terrabytes of data protected by ZFS, not a single instance of silent data corruption has been detected. And I have performed so many scrubs that I've read more than a petabyte worth of data.

    Anekdote: Somebody made a mistake and used the wrong type of cable to connect the hard drives to the HBA controller card. This caused actual silent data corruption. Because that person was running ZFS, it was detected so ZFS saved his data. This an example where ZFS did protect a person against silent data corruption.

    Evaluation

    I hope that the difference between unrecoverable read errors and silent data corruption is clear and that we should not confuse the two. They have different risk profiles associated with them.

    Furthermore, I have argued that silent data corruption is real and a serious issue at scale, and that it is that is dealt with accordingly.

    However, I've also argued that unless you are a home user running a small datacenter inside your basement, the risk of silent data corruption is so small that it is reasonable to accept the risk as a DIY NAS builder and not seek specific protection against it.

    The decision is up to you. If you want to go with ZFS and protect against silent data corruption, you should also be aware and accept the cost of ZFS. I myself have accepted that cost for my own NAS, but it's OK if you don't. If you care about silent data corruption so much, please also consider using ECC-memory.

    But in my opinion, you are not taking an unreasonable risk if you chose to go with Unraid, Snapraid, Linux kernel RAID, Windows Storage Spaces or maybe other options in the same vein. I would say that this is reasonable and up to you.

    Remember: the famous vendors of home user NAS boxes all seem to use regular Linux kernel RAID under the hood. And they seem to think that's fine.

    In the end, what really matters is a solution that suits your needs and also fits your budget and level of expertise. Can you fix problems when something goes wrong?


    1. I've noticed while testing with this particular drive that the drive was not kicked out of the array, and it just kept trying to read, grinding the Linux software RAID array to a halt. Removing the drive from the array fixed this. There is a 'failfast' option that only works with RAID1 or RAID10. 

    2. I don't want to suggest in any way that it would be wrong to take silent data corruption into account, but just to say I think it's not mandatory to really fret over it. 

    3. The most significant risk is that enterprise grade hard drives use on-board ECC cache memory, whereas consumer drives use non-ECC cache memory. So silently corrupted data in the cache memory of the drive could be a risk. 

    4. Storage vendors often choose to reformat har drives with larger sector sizes5. Those larger sectors then also incorporate additional checksum data to better protect against data corruption or unrecoverable read errors. 

    5. https://www.seagate.com/files/staticfiles/docs/pdf/whitepaper/safeguarding-data-from-corruption-technology-paper-tp621us.pdf 

    Tagged as : Storage
  2. Scrub Your NAS Hard Drives Regularly if You Care About Your Data

    Wed 22 April 2020

    Introduction

    Lots of people run a NAS at home. Maybe it's a COTS device from one of the well-known vendors1, or it's a custom build solution (DIY2) based on hardware you bought and assembled yourself.

    Buying or building a NAS is one thing, but operating it in a way that assures that you won't lose data is something else.

    Obviously, the best way to protect against dataloss, is to make regular backups. So ideally, even if the NAS would go up in flames, you would still have your data.

    Since backup storage costs money, people make tradeoffs. They may decide to take the risk and only backup a small portion of the really important data and take their chances with the rest.

    Well that is their own right. But still, it would be nice if we would reduce the risk of dataloss to a minimum.

    The risk: bad sectors

    The problem is that hard drives may develop bad sectors over time. Bad sectors are tiny portions of the drive that have become unreadable3. How small a sector may be, if any data is stored in them, it is now lost and this could cause data corruption (one or more corrupt files).

    This is the thing: those bad sectors may never be discovered until it is too late!

    With todays 14+ TB hard drives, it's easy to store vast amounts of data. Most of that data is probably not frequently accessed, especially at home.

    One or more of your hard drives may be developing bad sectors and you wouldn't even know it. How would you?

    Your data might be at risk right at this moment while you are reading this article.

    A well-known disaster scenario in which people tend to lose data is double hard drive failure where only one drive faillure can be tolerated (RAID 1 (mirror) or RAID 5, and in some scenario's RAID 10).

    In this scenario, a hard drive in their RAID array has failed and a second drive (one of the remaining good drives) has developed bad sectors. That means effectively a second drive has failed although the drive may still seem operational. Due to the bad sectors, data required to rebuild the array is lost because there is no longer any redundancy4.

    If you run a (variant of) RAID 5, you can only lose a single disk, so if a second disk fails, you lose all data5.

    The mitigation: periodic scrubbing / checking of your disks

    The only way to find out if a disk has developed bad sectors is to just read them all. Yes: all the sectors.

    Checking your hard drives for bad sectors (or other issues) is called 'data scrubbing'. If you bought a NAS from QNAP, Synology or another vendor, there is a menu which allows you to control how often and when you want to perform a data scrub.

    RAID solutions are perfectly capable of handling bad sectors. For a RAID array, it's just equivalent to a failed drive and an affected drive will be kicked out of the RAID array if bad sectors start causing read errors. The big issue we want to prevent is that multiple drives start to develop bad sectors at the same time, because that is the equivalent of multiple simultaneous drive failures, which many RAID arrays can't recover from.

    For home users I would recommend checking all hard drives once a month. I would recommend configuring the data scrub to run at night (often the default) because a scrub may impact performance in a way that can be noticeable and even inconvenient.

    Your vendor may have already configured a default schedule for data scrubs, so you may have been protected all along. If you take a look, at least you know.

    People who have built a DIY NAS have to setup and configure periodic scrubs themselves or they won't happen at all. However, that's not entirely true: I've noticed that on Ubuntu, all Linux software RAID arrays (MDADM) are checked once a month at night. So if you use Linux software RAID you may already be scrubbing.

    A drive that develops bad sectors should be replaced as soon as possible. It should no longer be trusted. The goal of scrubbing is to identify these drives as soon as possible. You don't want to get in a position that multiple drives have started developing bad sectors. You can only prevent that risk by scanning for bad sectors periodically and replacing bad drives.

    You should not be afraid about having to spend a ton of money replacing drives all the time. Bad sectors are not that common. But they are a common enough that you should check for them. There is a reason why NAS vendors offer the option to run data scrubs and recommend them6.

    You probably forgot to configure email alerting

    If a disk in your NAS would fail, how would you know? If the scrub would discover bad sectors, would you ever notice7?

    The answer may be: only when it's too late. Maybe a drive already failed and you haven't even noticed yet!

    When you've finished reading this article, it may be the right moment to take some time to check the status of your NAS and configure email alerting (or any other alerting mechanism that works for you). Make your NAS sends out a test message just to confirm it actually works!

    Closing words

    So I would like to advice you to do two things:

    1. Make sure your NAS runs a data scrub once a month
    2. Make sure your NAS is able to email alerts about failed disks or scrubs.

    These actions allow you to fix problems before they become catastrophic.

    P.S. S.M.A.R.T. monitoring

    Hard drives have a build-in monitoring system called S.M.A.R.T.

    If you have a NAS from one of the NAS vendors, they will allert on SMART monitoring information that would indicate that a drive is failing. DIY builders may have to spend time setting up this kind of monitoring manually.

    For more information about SMART I would recommend [this][this article] and this one.

    this article and also this one

    Linux users can take a look at the SMART status of their hard drives with this tool (which I made).


    1. QNAP, Synology, Netgear, Buffalo, Thecus, Western Digital, and so on. 

    2. FreeNAS, Unraid, Windows/Linux with Snapraid, OpenMediaVault, or a custom solution, and so on. 

    3. Bad sectors cause 'unrecoverable read errors' or UREs. Bad sectors have nothing to do with 'silent data corruption'. There's nothing silent about unrecoverable read errors. Hard drives report read errors back to the operating system, they won't go unnoticed. 

    4. A DIY NAS based on ZFS (FreeNAS is based on ZFS) may help mitigate the impact of such an event. ZFS can continue reading data from the remaining drives, even if bad sectors are encountered. Some files will be corrupted, but most of the data would still be readable. I think this capability is by itself not enough reason to pick a NAS based on ZFS because ZFS also has a cost involved that you need to accept too. For my large NAS I have chosen ZFS because I was prepared to 'pay the cost'. 

    5. Some people may chose to go with RAID 6 which tolerates two simultaneous drive failures but they also tend to run larger arrays with more drives, which also increases the risk of drive failure or one of the drives developing bad sectors. 

    6. Enterprise storage solutions (Even entry level storage arrays) often run patrol reads both on individual hard drives and also the RAID arrays on top of them. They are also enabled by default. 

    7. At one time I worked for a small company that ran their own (single) email server. One of the system administrators discovered totally by accident that one of the two drives in a RAID 1 had failed. It turns out we were running on a single drive for months before we discovered it, because we forgot to setup email alerting. We didn't lose data, but we came close. 

    Tagged as : Storage
  3. Understanding Storage Performance - IOPS and Latency

    Sat 21 March 2020

    Introduction

    The goal of this blogpost is to help you better understand storage performance. I want to discuss some fundamentals that are true regardless of your particular needs.

    This will help you better reason about storage and may provide a scaffolding for further learning.

    If you run your applications / workloads entirely in the cloud, this information may feel antiquated or irrelevant.

    However, since the cloud is just somebody else's compute and storage, knowledge about storage may still be relevant. Cloud providers expose storage performance metrics for you to monitor and this may help to make sense of them.

    Concepts

    I/O

    An I/O is a single read/write request. That I/O is issued to a storage medium (like a hard drive or solid state drive).

    It can be a request to read a particular file from disk. Or it can be a request to write some data to an existing file. Reading or writing a file can result in multiple I/O requests.

    I/O Request Size

    The I/O request has a size. The request can be small (like 1 Kilobyte) or large (several megabytes). Different application workloads will issue I/O operations with different request sizes. The I/O request size can impact latency and IOPS figures (two metrics we will discuss shortly).

    IOPS

    IOPS stands for I/O Operations Per Second. It is a performance metric that is used (and abused) a lot in the world of storage. It tells us how many I/O requests per second can be handled by the storage (for a particular workload).

    Warning: this metric is meaningless without a latency figure. We will discuss latency shortly.

    Bandwidth or throughput

    If you multiply the IOPS figure with the (average) I/O request size, you get the bandwidth or throughput. We state storage bandwidth mostly in Megabytes and Gigabytes per second.

    To give you an example: if we issue a workload of 1000 IOPS with a request size of 4 Kilobytes, we will get a throughput of 1000 x 4 KB = 4000 KB. This is about ~4 Megabytes per second.

    Latency

    Latency is the time it takes for the I/O request to be completed. We start our measurement from the moment the request is issued to the storage layer and stop measuring when either we get the requested data, or get confirmation that the data is stored on disk.

    Latency is the single most important metric to focus on when it comes to storage performance, under most circumstances.

    For hard drives, an average latency somewhere between 10 to 20 ms is considered acceptable (20 ms is the upper limit).

    For solid state drives, depending on the workload it should never reach higher than 1-3 ms. In most cases, workloads will experience less than 1ms latency numbers.

    IOPS and Latency

    This is a very important concept to understand. The IOPS metric is meaningless without a statement about latency. You must understand how long each I/O operation will take because latency dictates the responsiveness of individual I/O operations.

    If a storage solution can reach 10,000 IOPS but only at an average latency of 50 ms that could result in very bad application performance. If we want to hit an upper latency target of 10 ms the storage solution may only be capable of 2,000 IOPS.

    For more details on this topic I would recommend this blog and this blog.

    Access Patterns

    Sequential access

    An example of a sequential data transfer is copying a large file from one hard drive to another. A large number of sequential (often adjacent) datablocks is read from the source drive and written to another drive. Backup jobs also cause sequential access patterns.

    In practice this access pattern shows the highest possible throughput.

    Hard drives have it easy as they don't have to spend much time moving their read/write heads and can spend most time reading / writing the actual data.

    Random access

    I/O requests are issued in a seemingly random pattern to the storage media. The data could be stored all over various regions on the storage media. An example of such an access pattern is a heavy utilised database server or a virtualisation host running a lot of virtual machines (all operating simultaneously).

    Hard drives will have to spend a lot of time moving their read/write heads and can only spend little time transferring data. Both throughput and IOPS will plummet (as compared to a sequential access pattern).

    In practice, most common workloads, such as running databases or virtual machines, cause random access patterns on the storage system.

    Queue depth

    The queue depth is a number between 1 and ~128 that shows how many I/O requests are queued (in-flight) on average. Having a queue is beneficial as the requests in the queue can be submitted to the storage subsystem in an optimised manner and often in parallel. A queue improves performance at the cost of latency.

    If you have some kind of storage performance monitoring solution in place, a high queue depth could be an indication that the storage subsystem cannot handle the workload. You may also observe higher than normal latency figures. As long as latency figures are still within tolerable limits, there may be no problem.

    Storage Media Performance characteristics

    Hard drives

    Hard drives (HDDs) are mechanical devices that resemble a record player.

    They have an arm with a read/write head and the data is stored on (multiple) platters. hd01

    Hard drives have to physically move read/write heads to fulfil read/write requests. This mechanical nature makes them relatively slow as compared to solid state drives (which we will cover shortly).

    Especially random access workloads cause hard drives to spend a lot of time on moving the read/write head to the right position at the right time, so less time is available for actual data transfers.

    The most important thing to know about hard drives is that from a performance perspective (focussing on latency) higher spindle speeds reduce the average latency.

    Rotational Speed (RPM)Access Latency(ms)IOPS
    5400 17-18 50-60
    7200 12-13 75-85
    10,000 7-8 120-130
    15,000 5-6 150-180

    Because the latency of individual I/O requests is lower the drives with a higher RPM, you can issue more of such requests in the same amount of time. That's why the IOPS figure also increases.

    Latency and IOPS of an older Western Digital Velociraptor 10,000 RPM drive:

    wd01 Notice the latency and IOPS in the Queue Depth = 1 column.

    Source used to validate my own research.

    Regarding sequential throughput we can state that fairly old hard drives can sustain throughputs of 100-150 megabytes per second. More modern hard drives with higher capacities can often sustain between 200 - 270 megabytes per second.

    An important note: sequential transfer speeds are not constant and depend on the physical location of the data on the hard drive platters. As a drive fills up, throughput diminishes. Throughput can drop more than fifty percent! 1.

    So if you want to calculate how long it will take to transfer a particular (large) dataset, you need to take this into account.

    Solid State Drives

    Solid state drives (SSDs) have no moving parts, they are based on flash memory (chips). SSDs can handle I/O much faster and thus show significantly lower latency.

    ssd001

    Whereas we measure the average I/O latency of HDDs in milliseconds (a thousand of a second) we measure the latency of SSD I/O operations in microseconds (a millionth of a second).

    Because of this reduced latency per I/O request, SSDs outperform HDDs in every conceivable way. Even a cheap consumer SSD can at least sustain about 5000+ IOPS with only a 0.15 millisecond (150 microseconds) latency. That latency is about 40x better than the best latency of an enterprise 15K RPM hard drive.

    Solid state drives can often handle I/O requests in parallel. This means that larger queue depths with more I/O requests in flight can show significantly higher IOPS with a limited (but not insignificant) increase in latency.

    ssd01 The random I/O performance of an older SATA consumer SSD

    More modern enterprise SSDs show better latency and IOPS. The SATA interface seems the main bottleneck.

    ssd02 The random I/O performance of an enterprise SATA SSD

    SSDs perform better than HDDs across all relevant metrics except price in relation to capacity.

    Important note: SSDs are not well-suited for archival storage of data. Data is stored as charges in the chips and those charges can diminish over time. It's expected that even hard drives are better suited for offline archival purposes although the most suitable storage method would probably be tape.

    SSD actual performance vs advertised performance

    Many SSDs are advertised with performance figures of 80,000 - 100,000 IOPS at some decent latency. Depending on the workload, you may only observe a fraction of that performance.

    Most of those high 80K-100K IOPS figures are obtained by benchmarking with very high queue depths (16-32). The SSD benefits from such queue depths because it can handle a lot of those I/O requests in parallel.

    Please beware: if your workload doesn't fit in that pattern, you may see lower performance numbers.

    If we take a look at the chart above of the Intel SSD we may notice how the IOPS figures only start to come close to the advertised 80K+ IOPS as the queue depth increases. It's therefore important to understand the characteristics of your own workload.

    RAID

    If we group several hard drives together we can create a RAID array. A RAID array is a virtual storage device that exceeds the capacity and performance of a single hard drive. This allows storage to scale within the limits of a single computer.

    RAID is also used (or some say primarily used) to assure availability by assuring redundancy (drive failure won't cause data loss). But for this article we focus it's performance characteristics.

    SSDs can achieve impressive sequential throughput speeds, of multiple gigabytes per second. Individual hard drives can never come close to those speeds, but if you put a lot of them together in a RAID array, you can come very close. For instance, my own NAS an achieve such speeds using 24 drives.

    RAID also improves the performance of random access patterns. The hard drives in a RAID array work in tandem to service those I/O requests so a RAID array shows significantly higher IOPS than a single drive. More drives means more IOPS.

    RAID 5 with 8 x 7200 RPM drives

    The picture below shows the read IOPS performance of an 8-drive RAID 5 array of 1 TB, 7200 RPM drives. We run a benchmark of random 4K read requests.

    Notice how the IOPS increases as the queue depth increases.

    raidiops

    However, nothing is free in this world. A higher queue depth - which acts as a buffer - does increase latency.

    raidlat

    Notice how quickly the latency exceeds 20ms and quickly becomes almost unusable.

    RAID 5 with 8 x 10,000 RPM drives

    Below is the result of a similar test with 10,000 RPM hard drives. Notice how much better the IOPS and latency figures are.

    raid10kiops

    The latency looks much better:

    raid10klat

    It makes sense to put SSDs in RAID. Although they are more reliable than hard drives, they can fail. If you care about availability, RAID is inevitable. Furthermore, you can observe the same benefits as with hard drives: you pool resources together, achieving higher IOPS figures and more capacity than possible with a single SSD.

    Capacity vs. Performance

    The following is mostly focussed on hard drives although it could be true for solid state drives as well.

    We put hard drives in RAID arrays to get more IOPS than a single drive can provide. At some point - as the workload increases - we may hit the maximum number of IOPS the RAID array can sustain with an acceptable latency.

    This IOPS/Latency threshold could be reached even if we have only 50% of the storage capacity of our RAID array in use. If we use the RAID array to host virtual machines for instance, we cannot add more virtual machines because this would cause the latency to rise to unacceptable levels.

    It may feel like a lot of good storage space is going to waste, and in some sense this may be true. For this reason, it could be a wise strategy to buy smaller 10,000 RPM or 15,000 RPM drives purely for the IOPS they can provide and forgo on capacity.

    So it might be the case that you may have to order and add let's say 10 more hard drives to meet the IOPS/Latency demands while there's still plenty of space left.

    This kind of situation is less likely as SSDs have taken over the role of the performance storage layer and (larger capacity) hard drives are pushed in the role of 'online' archival storage.

    Closing words

    I hope this article has given you a better understanding of storage performance. Although it is just an introduction, it may help you to better understand the challenges of storage performance.


    1. https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics#Data_transfer_rate 

    Tagged as : storage

Page 2 / 6