Louwrentius

Update 2020: I've written another blogpost about this topic, including some benchmark examples.

When most people think about storage performance, they think about throughput. But throughput is similar to the top speed of a car. In reality, you will almost never reach the top speed of your car (unless you are living in Germany). And that's fine, because in most situations that's not so relevant.

For instance, properties like how fast your car accelerates and how well the car handles bends and corners are often more important than its top speed. And this example also holds for storage performance.

Most people know that SSDs are often way faster than regular mechanical hard drives. But it's not about the throughput of these devices. Its all about Input/Output operations per second (IOPS). If you can handle a high number of IOPS, that is great for real life application performance. But IOPS does not tell you the whole story. To be more precise: IOPS is a meaningless figure unless tied to an average latency and a certain request size (how much data is processed with the I/O). Let's first focus on IOPS and Latency and talk about the request size later.

Latency is how fast a single I/O-request is handled. This is very important, because a storage subsystem that can handle 1000 IOPS with an average latency of 10ms may get better application performance than a subsystem that can handle 5000 IOPS with an average latency of 50ms. Especially if the application is sensitive to latency, such as a database service.

This is a very important thing to understand: how IOPS and latency relate to each other. Here, the car analogy probably breaks down. We need a different one to better understand what is going on. So picture you are in a super market. This is a special supermarket, where customers (I/Os) are served by cashiers (disk) at an average speed of 10ms. If you divide one second with 10ms, we understand that this cashier can handle 100 customers per second. But only one at a time, in succession.

It is clear that although the cashier can handle 100 customers per second, he cannot handle them at the same time! So when a customer arrives at the register, and within those 10ms handling time, a second customer arrives, that customer must wait. Once the waiting customer is handled by the cashier, handling of that customer still takes just 10ms, but the overal processing time was maybe 15ms or worst case (two customers arriving at the same time) even 20ms.

So it is very important to understand that although a disk may handle individual I/Os with an average latency of 10ms, the actual latency as perceived by the application may be higher as some I/Os must wait in line.

This example also illustrates that waiting in line increases the latency for a particular I/O to be handled. So if you increase the Read I/O queue, you will notice that the average latency will increase. Longer queues will mean higher latency, but also more IOPS!!!

How is that possible? How can a disk drive suddenly do more random IOPs at the cost of latency? The trick lies in that the storage subsystem can be smart and look at the queue and then order the I/Os in such a way that the actual access pattern to disk will be more serialised. So a disk can serve more IOPS/s at the cost of an increase in average latency. Depending on the achieved latency and the performance requirements of the application layer, this can be acceptable or not.

In future blog posts I will show some performance benchmarks of a single drive to illustrate these examples.

Update 2020: Please beware of the impact of random write I/O performance.

Please note that with a modern Linux distribution, bitmaps are enabled by default. They will not help speed up a rebuild after a failed drive. But it will help resync an array that got out-of-sync due to power failure or another intermittent cause.

When a disk fails or gets kicked out of your RAID array, it often takes a lot of time to recover the array. It takes 5 hours for my own array of 20 disks to recover a single drive.

Wouldn't it be nice if that time can be reduced? Even to 5 seconds?

Although not enabled by default, you can enable so called 'bitmaps'. As I understand it, a bitmap is basically a map of your RAID array and it charts which areas need to be resynced if a drive fails.

This is great, because I have the issues that of every 30 reboots, sometimes a disk won't get recognized and the array is degraded. Adding the disk back into the array will mean that the system will be recovering for 5+ hours.

I enabled Bitmaps and after adding a missing disk back into the array, the array was recovered instantly.

Isn't that cool?

So there are two types of bitmapsL

1. internal: part of the array itself
2. external: a file residing on an external drive outside the array

The internal bitmap is integrated in the array itself. Keeping the bitmap up to date will probably affect performance of the array. However I didn't notice any performance degradation.

The external bitmap is a file that must reside on a EXT2 or EXT3 based file system that is not on top of the RAID array. So this means that you need an extra drive for this or need to use your boot drive for this. I can imagine that this solution will have less impact on the performance of the array but it is a bit more hassle to maintain.

I enabled an internal bitmap on my RAID arrays like this:

mdadm --grow /dev/md5 --bitmap=internal

This is all there is to it. You can configure an external bitmap like this:

mdadm --grow /dev/md5 --bitmap=/some/directory/somefilename

There probably will be some performance penalty involved, but it does not seem to affect sequential throughput, which is the only thing that is important for my particular case.

For most people, I would recommend configuring an internal bitmap, unless you really know why you would have to use an external bitmap.

If you want to order a new iMac with an SSD there are two problems:

1. An SSD of 256 gigabytes will cost you a lot: 600 euros.
2. The SSD is of medium quality, and does not justify the cost of 600 euros.

Here you can find information about the Toshiba SSD that Apple provides.

You can consider buying an SSD for half the price and put it inside your new iMac yourself afterwards. However, this is a hassle and you will probably void your waranty.

The solution is to buy an SSD yourself and put it in an external Firewire 800 casing. Firewire 800 provides you with about 70 megabytes per second of troughput, which is enough for most applications. It is about three times faster than USB and the latency of Firewire is lower than that of USB, thus improving responsiveness.

Thunderbolt would provide the best solution, but as of July 2011, there are no Thunderbolt products on the market.

The product I use is an external 2,5 inch Firewire 800 casing of OWC: the Mercury Elite-AL Pro mini. This casing cost me about 60 euros or 80 dollars including shipping costs. I admit that this solution is not cheap, but it does work. And you won't void your warranty and it is way cheaper than the Apple SSD.

I've put my Intel SSD inside this casing and my mac boots in about 15 seconds. After entering username and password, login is almost instantaneously.

Caveat

I encountered one problem with the OWC casing. I encountered random system freezes, related to the external OWC housing. Using the internal 1 TB hard drive, I had no problems. The cause of this issue is probably related to power not being provided to the OCW casing after the systems returns from sleep.

I resolved this issue by connecting a special USB cable to the iMac and connecting the other end to the 5V DC Power Input on the OWC casing. After that, the daily random system freezes vanished. These cables can be obtained almost everywhere.