Louwrentius - Storage

Benchmarking cheap SSDs for fun, no profit (be warned)

2023-03-26T12:00:00+02:00

The price of Solid-state drives (SSDs) has dropped significantly over the last few years. It's now possible to buy a 1TB solid-state drive for less than €60. However, at such low price points, there is a catch.

Although cheap SSDs do perform fine regarding reads, sustained write performance can be really atrocious. To demonstrate this concept, I bought a bunch of the cheapest SATA SSDs I could find - as listed below - and benchmarked them with Fio.

Model	Capacity	Price
ADATA Ultimate SU650	240 GB	€ 15,99
PNY CS900	120 GB	€ 14,56
Kingston A400	120 GB	€ 20,85
Verbatim Vi550 S3	128 GB	€ 14,99

I didn't have the budget to buy a bunch of 1TB of 2TB SSD, so these ultra-cheap, low capacity SSDs are a bit of a stand-in. I've also added a Crucial MX500 1TB (CT1000MX500SSD1) SATA¹ SSD - which I already owned - to the benchmarks to see how well those small-capacity SSDs stack up to a cheap SSD with a much larger capacity.

Understanding SSD write performance

To understand the benchmark results a bit better, we discuss some SSD concepts in this section. Feel free to skip to the actual benchmarks if you're already familiar with them.

SLC Cache

SSDs originally used single-level cell (SLC) flash memory, which can hold a single bit and is the fastest and most reliable flash memory available. Unfortunately, it's also the most expensive. To reduce cost, multi-level cell (MLC) flash was invented, which can hold two bits instead of one, at the cost of speed and longevity². This is even more so for triple-level cell (TLC) and quad-level cell (QLC) flash memory. All 'cheap' SSDs I benchmark use 3D v-nand³ TLC flash memory.

One technique to temporarily boost SSD performance is to use a (small) portion of (in our case) TLC flash memory as if it was SLC memory. This SLC memory then acts as a fast write cache⁴. When the SSD is idle, data is moved from the SLC cache to the TLC flash memory in the background. However, this process is limited by the speed of the 'slower' TLC flash memory and can take a while to complete.

While this trick with SLC memory works well for brief, intermittent write loads, sustained write loads will fill up the SLC cache and cause a significant drop in performance as the SSD is forced to write data into slower TLC memory.

DRAM cache

As flash memory has a limited lifespan and can only take a limited number of writes, a wear-leveling mechanism is used to distribute writes over all cells evenly, regardless of where data is written logically. Keeping track of this mapping between logical and physical 'locations' can be sped up with a DRAM cache (chip) as DRAM tend to be faster than flash memory. In addition, the DRAM can also be used to cache writes, improving performance. Cheap SSDs don't use DRAM cache chips to reduce cost, thus they have to update their data mapping tables in flash memory, which is slower. This can also impact (sustained) write performance. To be frank, I'm not sure how much a lack of DRAM impacts our benchmarks.

Benchmark method

Before I started benchmarking I submitted a trim command to clear each drive. Next, I performed a sequential write benchmark of the entire SSD with a block size of 1 megabyte and a queue depth of 32. The benchmark is performed on the 'raw' device, no filesystem is used. I used Fio for these benchmarks.

Benchmark results

The chart below shows write bandwith over time for all tested SSDs. Each drive has been benchmarked in full, but the data is truncated to the first 400 seconds for readability (performance didn't change). The raw Fio benchmark data can be found here (.tgz).

click for a larger image

It's funny to me that some cheap SSDs initially perform way better than the more expensive Crucial 1TB SSD⁵. As soon as their SLC cache runs out, the Crucial 1TB has the last laugh as it shows best sustained throughput, beating all cheaper drives, but the Kingston A400 comes close.

Of all the cheap SSDs only the Kingston shows the best sustained write speed at around 100 MB/s and there are no intermittent drops in performance. The ADATA, PNY and Verbatim SSDs show flakey behaviour and basically terrible sustained write performance. But make no mistake, I would not call the performance of the Kingston SSD, nor the Crucial SSD - added as a reference - 'good' by any definition of that word. Even the Kingston can't saturate gigabit Ethernet.

The bandwidth alone doesn't tell the whole story. The latency or responsiveness of the SSDs is also significantly impacted:

click for a larger image

The Crucial 1TB SSD shows best latency overall, followed by the Kingston SSD. The rest of the cheap SSDs show quite high latency spikes and very high latency overall, even when some of the spikes settle, like for the ADATA SSD. When latency is measured in seconds, things are bad.

To put things a bit in perspective, let's compare these results to a Toshiba 8 TB 7200 RPM hard drive I had lying around.

click for a larger image

The hard drive shows better write throughput and latency⁶ as compared to most of the tested SSDs. Yes, except for the initial few minutes where the cheap SSDs tend to be faster (except for the Kingston & Crucial SSDs) but how much does that matter?

As we've shown the performance of a hard drive to contrast the terrible write performance of the cheap SSDs, it's time to also compare them to a more expensive, higher-tier SSD.

click for a larger image

I've bought this Samsung SSD in 2019 for €137 euro, so that's quite a different price point. I think the graph speaks for itself, especially if you consider that this graph is not truncated, this is the full drive write.

Evaluation & conclusion

One of the funnier conclusions to draw is that it's beter to use a hard drive than to use cheap SSDs if you need to ingest a lot of data. Even the Crucial 1TB SSD could not keep up with the HDD.

A more interesting conclusion is that the 1TB SSD didn't perform that much better than the small cheaper SSDs. Or to put it differently: although the performance of the small, cheap SSDs is not representative of the larger SSD, it is still quite in the same ball park. I don't think it's a coincidence that the Kingston SSD came very close to the performance of the Crucial SSD, as it's the most 'expensive' of the cheap drives.

In the end, my intend was to demonstrate with actual benchmarks how cheap SSDs show bad sustained write performance and I think I succeeded. I hope it helps people to understand that good SSD write performance is not a given, especially for cheaper drives.

The Hacker News discussion of this blog post can be found here

Disclaimer

I'm not sponsored in any way. All mentioned products have been bought with my own money.

The graphs are created with fio-plot, a tool I've made and maintain. The benchmarks have been performed with bench-fio, a tool included with fio-plot, to automate benchmarking with Fio.

As I don't have a test system with NVMe, I had to use SATA-based SSDs. The fact that the SATA interface was not the limiting factor in any of the tests, is foreboding. ↩
As a general note, I think the vast majority of users should not worry about SSD longevity in general. Only people with high-volume write workloads should keep an eye on write endurance of SSD and buy a suitable product. ↩
instead of packing the bits really dense together in a cell horizontally, the bits are stacked vertically, saving horizontal space. This allows for higher data densities in the same footprint. ↩
Some SSDs have a static SLC cache, but others size the SLC cache in accordance to how full an SSD is. When the SSD starts to fill up, the SLC cache size is reduced. ↩
After around 45-50 minutes of testing, performance of the Crucial MX 500 also started to drop to around 40 MB/s and fluctuate up and down. Evidence. ↩
it's so funny to me that a hard drive beats an SSD on latency. ↩

An ode to the 10,000 RPM Western Digital (Veloci)Raptor

2021-10-30T12:00:00+02:00

Introduction

Back in 2004, I visited a now bankrupt Dutch computer store called MyCom¹, located at the Kinkerstraat in Amsterdam. I was there to buy a Western Digital Raptor model WD740, with 74 GB of capacity, running at 10,000 RPM.

When I bought this drive, we were still in the middle of the transition from the PATA interface to SATA². My raptor hard drive still had a molex connector because older computer power supplies didn't have SATA power connectors.

You may notice that I eventually managed to break off the plastic tab of the SATA power connector. Fortunately, I could still power the drive through the Molex connector.

A later version of the same drive came with the Molex connector disabled, as you can see below.

Why did the Raptor matter so much?

I was very eager to get this drive as it was quite a bit faster than any consumer drive on the market at that time.

This drive not only made your computer start up faster, but it made it much more responsive. At least, it really felt like that to me at the time.

The faster spinning drive wasn't so much about more throughput in MB/s - although that improved too - it was all about reduced latency.

A drive that spins faster³ can complete more I/O operations per second or IOPs⁴. It can do more work in the same amount of time, because each operation takes less time, compared to slower turning drives.

The Raptor - mostly focussed on desktop applications⁵ - brought a lot of relief for professionals and consumer enthusiasts alike. Hard disk performance, notably latency, was one of the big performance bottlenecks at the time.

For the vast majority of consumers or employees this bottleneck would start to be alleviated only well after 2010 when SSDs slowly started to become standard in new computers.

And that's mostly also the point of SSDs: their I/O operations are measured in micro seconds instead of milliseconds. It's not that throughput (MB/s) doesn't matter, but for most interactive applications, you care about latency. That's what makes an old computer feel as new when you swap out the hard drive for an SSD.

The Raptor as a boot drive

For consumers and enthusiast, the Raptor was an amazing boot drive. The 74 GB model was large enough to hold the operating system and applications. The bulk of the data would still be stored on a second hard drive either also connected through SATA or even still through PATA.

Running your computer with a Raptor for the boot drive, resulted in lower boot times and application load times. But most of all, the system felt more responsive.

And despite the 10,000 RPM speed of the platters, it wasn't that much louder than regular drives at the time.⁶.

In the video above, a Raspberry Pi4 boots from a 74 GB Raptor hard drive.

Alternatives to the raptor at that time

To put things into perspective, 10,000 RPM drives were quite common even in 2003/2004 for usage in servers. The server-oriented drives used the SCSI interface/protocol which was incompatible with the on-board IDE/SATA controllers.

Some enthusiasts - who had the means to do so - did buy both the controller⁷ and one or more SCSI 'server' drives to increase the performance of their computer. They could even get 15,000 RPM hard drives! These drives however, were extremely loud and had even less capacity.

The Raptor did perform remarkably well in almost all circumstances, especially those who mattered to consumers and consumer enthusiasts alike. Suddenly you could get SCSI/Server performance for consumer prices.

The in-depth review of the WD740 by Techreport really shows how significant the raptor was.

The Velociraptor

The Raptor eventually got replaced with the Velociraptor. The Velociraptor had a 2.5" formfactor, but it was much thicker than a regular 2.5" laptop drive. Because it spun at 10,000 RPM, the drive would get hot and thus it was mounted in an 'icepack' to disipate the generated heat. This gave the Velociraptor a 3.5" formfactor, just like the older Raptor drives.

In the video below, a Raspberry Pi4 boots from a 500 GB Velociraptor hard drive.

Benchmarking the (Veloci)raptor

Hard drives do well with sequential read/write patterns, but their performance implodes when the data access pattern becomes random. This is due to the mechanical nature of the device. That random access pattern is where 10,000 RPM outperform their slower turning siblings.

Random 4K read performance showing both IOPs and latency. This is kind of a worst-case benchmark to understand the raw I/O and latency performance of a drive.

Drive ID	Form Factor	RPM	Size (GB)	Description
ST9500423AS	2.5"	7200	500	Seagate laptop hard drive
WD740GD-75FLA1	3.5"	10,000	74	Western Digital Raptor WD740
SAMSUNG HD103UJ	3.5"	7200	1000	Samsung Spintpoint F1
WDC WD5000HHTZ	2.5" in 3.5"	10,000	500	Western Digital Velociraptor
ST2000DM008	3.5"	7200	2000	Seagate 3.5" 2TB drive
MB1000GCWCV	3.5"	7200	1000	HP Branded Seagate 1 TB drive

I've tested the drives on an IBM M1015 SATA RAID card flashed to IT mode (HBA mode, no RAID firmware). The image is generated with fio-plot, which also comes with a tool to run the fio benchmarks.

It is quite clear that both 10,000 RPM drives outperform all 7200 rpm drives, as expected.

If we compare the original 3.5" Raptor to the 2.5" Velociraptor, the performance increase is significant: 22% more IOPs and 18% lower latency. I think that performance increase is due to a combination of the higher data density, the smaller size (r/w head is faster in the spot it needs to be) and maybe better firmware.

Both the laptop and desktop Seagate drives seem to be a bit slower than they should be based on theory. The opposite is true for the HP (rebranded Seagate), which seem to perform better than expected for the capacity and rotational speed. I have no idea why that is. I can only speculate that because the HP drive came out of a server, that the fireware was tuned for server usage patterns.

Closing words

Although the performance increase of the (veloci)raptor was quite significant, it never gained wide-spread adoption. Especially when the Raptor first came to marked, its primary role was that of a boot drive because of its small capacity. You still needed a second drive for your data. So the increase in performance came at a significant extra cost.

The Raptor and Velociraptor are now obsolete. You can get a solid state drive for $20 to $40 and even those budget-oriented SSDs will outperform a (Veloci)raptor many times over.

If you are interested in more pictures and details, take a look at this article.

This article was discussed on Hacker News here.

Reddit thread about this article can be found here

Mycom, a chain store with quite a few shops in all major cities in The Netherlands, went bankrupt twice, once in 2015 and finally in 2019. ↩
We are talking about the first SATA version, with a maximum bandwidth capacity of 150 MB/s. Plenty enough for hard drives at that time. ↩
https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics ↩
https://louwrentius.com/understanding-storage-performance-iops-and-latency.html ↩
I read that WD intended the first Raptor (34 GB version) to be used in low-end servers as a cheaper alternative to SCSI drives . After the adoption of the Raptor by computer enthusiasts and professionals, it seems that Western Digital pivoted, so the next version - the 74 GB I have - was geared more towards desktop usage. That also meant that this 74 GB model got fluid bearings, making it quieter⁸. ↩
Please note that the first model, the 37 GB version, used ball bearings in stead of fluid bearings, and was reported to be significant louder. ↩
Low-end SCSI card were often used to power flatbed scanners, Iomega ZIP drives, tape drives or other peripherals, but in order to benefit from the performance of those server hard drives, you needed a SCSI controller supporting higher bandwidth and those were more expensive. ↩
The 74 GB model is actually rather quiet drive at idle. Drive activity sounds rather smooth and pleasant, no rattling. ↩

Don't be afraid of RAID

2020-05-22T12:00:00+02:00

Introduction

I sense this sentiment on the internet that RAID is dangerous, that the likelihood of your RAID array failing during a rebuild is almost a certainty, because hard drives have become so large.

I think nothing is further from the truth and I would like to dispel this myth.

Especially for home users and small businesses, RAID arrays are still a reliable and efficient way of storing a lot of data in a single place.

Perception of RAID reliability

There are many horror stories to be found on the internet about people at home losing their RAID array. These stories may have contributed to a negative attitude towards RAID in general.

You may acuse me of victim blaming, but in many cases, I do wonder if those incidents were due to user error¹, due to bad luck or actual RAID causing problems. And there is a bias in reporting: you won't hear from the countless people who have no issues.

In any case, the damage is done, but I still think (software) RAID is perfectly fine.

The myth about the Unrecoverable Read Error (URE)

I think the trouble started with this terrible article on ZDNET from 2007.

In this article, it's argued that as drives become bigger, but not more reliable, you will see more unrecoverable read errors (UREs). More capacity means more sectors, so more risk of one of them going bad.

An URE is an incident where the hard drive can't read a sector². For old people like me, that sounds like the definition of a 'bad sector'. The article argues that on average you would encounter an URE for every 12.5 TB of data read.

By the logic of the ZDNET acticle, just copying all data from a 14 TB drive would probably be impossible, because you would probably hit an URE / bad sector before you finish your copy.

This is a very big issue for RAID arrays. A RAID array rebuild consists of reading the contents of all remaining drives in their entirety³. So you are guaranteed to hit an URE during a RAID rebuild.

The good news is that you don't have to worry about any of this. Because it is not true.

Hard drives are not that unreliable in practice. On the contrary. They are remarkably reliable, I would say. Just look at the Backblaze drive statistics⁴.

The prediction of the infamous ZDNET article has not come true. The URE specification for hard drive describes a worst-case scenario and seem to be more about marketing (a way to differentiate enterprise drives from consumer drives) than about reality.

If the ZDNET article were true, I, myself, should have encountered many UREs because of the many RAID array scrubs/patrol reads that have completed acros various RAID arrays.

RAID has never stopped working and is still going strong.

Card

Scrubbing protects against the impact of bad sectors

When a drive fails in a RAID array that can only tollerate one drive failure, it's very important that all remaining drives won't encounter any read errors. Because redundancy is lost, any read errors due to bad sectors could mean that the entire array is lost or at least some files are corrupted⁵.

Every RAID array supports 'scrubbing'. It's a process where every sector of the RAID array is read, which in effect causes all sectors of all hard drives to be read.

A scrub is a process to check for bad sectors in advance. If bad sectors are found on a hard drive, the drive can be replaced so it will not cause problems during a potential future rebuild. Replacing the drive itself will cause a rebuild, but assuming the scrub didn't find any other drives with bad sectors, that rebuild will be fine.

A RAID array that doesn't undergo a regular scrub is a disaster waiting to happen. Bad sectors may be building up on one of the other drivs and when a drive actually fails, the entire array may be lost because of the undetected bad sectors on (one of) the remaining drives.

If you want to store data in a reliable way on a RAID array, you need to assure the array is scrubbed periodically. And even if you don't use RAID, I would recommend running a long SMART test once a month against every hard drive you own.

By default, a Linux software RAID array is scrubbed once a week on Ubuntu. For details, look at the contents of /etc/cron.d/mdadm.

If you use ZFS on Linux, your array is automatically scrubbed on the second Sunday of every month if you run Ubuntu.

NAS vendors like Synology or QNAP have data scrubs enabled by default. Consider the manual of your particular NAS to adjust the frequency. I would recommend to scrub at least once a month and at night.

Why is RAID 5 considered harmful?

Frankly, I wonder that too.

I notice a lot of people on the internet claiming that you should never use RAID 5 but I disagree. It all depends on the circumstances. Finding a balance between cost and risk is important.

This page dating back to 2003 advocated not to use RAID 5 but that's focused on the enterprise environment and even there I see its uses.

For small RAID arrays with five or less drives I think RAID 5 is still a great fit. Especially if you run a small 4-bay NAS it would make total sense to use RAID 5. You get a nice balance between capacity and the cost of availability.

It's not really recommended to create larger RAID 5 arrays. Compared to a single drive, a RAID array with 8 drives is 8 times more likely to experience a drive failure. You multiply the risk of a single drive failing by eight. With larger arrays, double drive failure becomes a serious risk.

This is why it's really recommended to use RAID 6 for larger RAID arrays, because RAID 6 can tollerate two simultaneous drive failures. I've used RAID 6 in the past and I use RAIDZ2 (ZFS) as the basis for my current NAS.

I also run an 8-drive RAID 5 in one of my servers that hosts not so important data that I still want to keep around and would rather not lose, but not at every cost. It's all about a balance between risk and cost. Please also read the postscript of this post, you will like it.

It is true that during a rebuild, hard drives are strained more, but unless the RAID array is also in heavy use, the load on the drive isn't that big: the data is read sequentially, which is quite easy on the drives.

RAID rebuild performance is mostly determined by the size of the drives and not by the number of drives in the RAID array⁶.

Years ago I ran a 20-drive RAID 6 based on 1 TB drives and it did a rebuild in 5 hours. Recently I tested a rebuild of 8 drives in RAID 5 (using the same drives) and it also took almost 5 hours (4H45M).

The RAID write hole

The RAID 5/6 'write hole' is often mentioned as something you should be afraid about.

Parity-based RAID like RAID 5 and RAID 6 may be affected by an issue called the 'write hole'. To (over)simplify: if a computer would experience a sudden power failure, a write to the RAID array may be interrupted. This could cause a partial write to the RAID array, leaving it in an inconsistent state.

As a side note, I would always recommend protecting your NAS with a UPS (battery backup) so your server can shut down in a clean way, before power is lost as the battery runs out.

ZFS RAIDZ is not affected by the 'write hole' issue, because it writes data to a log first before writing it to the actual array⁷.

Linux MDADM software RAID also is protected against the 'write hole' phenomenon by using a bitmap (which is enabled by default⁷).

Hardware RAID is also protected against this by using a battery backup for the cache memory. The data in the cache memory is written to disk as soon as the computer is powered back on.

Setup alerting if you care about your data

I think that a lot of RAID horror stories are due to the fact that people may never notice any problems until it is too late because they never set up any kind of alerting (by email or other).

Ideally, you would also make sure your system monitors the SMART data of your hard drives and alert when critical numbers start to rise (Reallocated Sector count and Current Pending Sector count).

This is also a moment of personal reflection. Do you run a RAID array? Did you setup alerting? Or could your RAID array be failing this very moment and you wouldn't know?

Anyway: I think a lack of proper alerting is a nice way of getting into trouble with RAID, but that's not on RAID. Any storage solution that is not monitored is just a disaster waiting to happen.

Why people choose not to use RAID

If a RAID array fails, all data is lost. Some people are not comfortable with this risk. They would rather lose the contents of some drives, but not all of them.

Solutions like Unraid and SnapRAID use one or more dedicated hard drives to store redundant (parity) data. The other hard drives are formatted with your filesystem of choice and can be accessed as normal hard drives. Altough I have no experience with this product, StableBit DrivePool seems to work in a similar manner.

If you would have six hard drives, thus five data drives and one parity disk, the loss of two drives would result in data loss, as with RAID 5. However, the data on the remaining four drives would still be intact. The data loss is limited to just one drive worth of data.

The 'all-or-nothing' risk associated with regular software RAID is thus mitigated. I myself don't think those risks aren't that large, but Unraid and snapraid are popular product and I think they are reasonable alternatives.

Mergerfs could also be an interesting option, although it only supports mirroring.

Backups are still important

Storing your data on any kind of RAID array is never a substitute for a backup.

You should still copy your data to some other storage if you want to protect your data. You may chose to only make a backup of a subset of all of the data, but at least you take an informed risk.

Evaluation

I hope I have demonstrated why RAID is still a valid and reliable option for data storage.

Feel free to share your own views in the comments.

P.S.

I ran a scrub on my 8-disk RAID 5 array (based on 2 TB drives) as I was writing this article. My servers are only powered on when I need them and while powered off, it's easy for them to miss their periodic scrub window.

So as to practice what I preach I ran a scrub. Lo and behold, one of the drives was kicked out of my Linux software RAID array. Don't you love the irony?

sd 0:0:4:0: [sde] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:4:0: [sde] tag#29 Sense Key : Medium Error [current] 
sd 0:0:4:0: [sde] tag#29 Add. Sense: Unrecovered read error
sd 0:0:4:0: [sde] tag#29 CDB: Read(10) 28 00 9f 42 9e 30 00 04 00 00
print_req_error: critical medium error, dev sde, sector 2671943216

Followed by:

md/raid:md6: Disk failure on sde, disabling device.
md/raid:md6: Operation continuing on 7 devices.

The drive was clearly kicked out because the drive encountered bad sectors. A quick check of the SMART data revealed more than 300+ sectors were already remapped, but the data stored in them could not be recovered, causing read errors.

This drive is clearly done, although it was still operational.

After swapping this defective drive with a spare replacement, I started the rebuild proces, which took four hours and twenty minutes. My RAID 5 has rebuild and is now perfectly fine.

If an event like this doesn't drive the point home that scrubs are important, I don't know what will.

Sometimes I read what hardware people use for storage and I think about this quote by John Glenn: ‘I felt exactly how you would feel if you were getting ready to launch and knew you were sitting on top of 2 million parts — all built by the lowest bidder on a government contract.’ ↩
The smallest unit of storage a drive can store, often 4K or 512 bytes for older, smaller drives. ↩
ZFS works differently, it only reads the sectors containing actual data. ↩
Those hard drive live in a datacenter with a conditioned environment, which you probably don't have at home. But as long as you keep the temperature of hard drive within limits, I don't think it matters that much. ↩
ZFS is both a RAID solution and a filesystem in one and can tell you exactly which file is affected. A nice feature. ↩
ZFS rebuilds or 'resilvers' become slower as you add more drives to a RAIDZ(2/3) VDEV, it seems. I'm not sure this is still the case with more recent ZFS versions. ↩
Both ZFS and MDADM will take a performance hit by using a log/bitmap. Both solutions support using an SSD to accelerate the log/bitmap to remove this performance hit. Most home users probably won't need this. ↩↩

What home NAS builders should understand about silent data corruption

2020-04-23T12:00:00+02:00

Introduction

When it comes to dealing with storage in a DIY NAS context, two important topics come up:

Unrecoverable read errors (UREs) or what old people like me call 'bad sectors'
Silent data corruption (data corruption unnoticed by the storage layers)

I get a strong impression that people tend to confuse those concepts. However, they often come up when people evaluate their options when they want to buy or build their own do-it-yourself NAS.

In this article, I want to make a clear distinction between the two and assess their risk. This may help you evaluating these risks and make an informed decision.

Unrecoverable read errors (due to bad sectors)

When a hard drive hits a 'bad sector', it means that it can't read the contents of that particular sector anymore.

If the hard drive is unable to read that data even after multiple attempts, the operating system will return an Unrecoverable Read Error (URE).

This is an example (on Linux) of a drive experiencing read errors, as pulled from /var/log/syslog (culled a bit for readability):

sd 0:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda] tag#19 Sense Key : Medium Error [current] 
sd 0:0:0:0: [sda] tag#19 Add. Sense: Unrecovered read error
sd 0:0:0:0: [sda] tag#19 CDB: Read(10) 28 00 02 1c 8c 00 00 00 98 00
blk_update_request: critical medium error, dev sda, sector 35425280 op 0x0:(READ)
sd 0:0:0:0: [sda] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda] tag#16 Sense Key : Medium Error [current] 
sd 0:0:0:0: [sda] tag#16 Add. Sense: Unrecovered read error
sd 0:0:0:0: [sda] tag#16 CDB: Read(10) 28 00 02 1c 8d 00 00 00 88 00
blk_update_request: critical medium error, dev sda, sector 35425536 op 0x0:(READ)

If a sector cannot be read, the data stored in that sector is lost. And in my experience, if you encounter a single bad sector, soon, there will be more. So if this happens, it's time to replace the hard drive.

We use RAID to protect against drive failure. RAID (no matter the implementation) also can deal with 'partial failure' such as a drive encountering bad sectors.

In a RAID array, a drive encountering unrecoverable read errors is just kicked out of the array, so it doesn't 'hang' or 'stall' the entire RAID array.

Please note that this behaviour does depend on the particular RAID solution of choice¹. The point is though that bad sectors or UREs are a common event and RAID solutions can deal with them properly.

The real problem with bad sectors (resulting in UREs) is that they can remain undiscovered until it is too late. So to uncover them in an early state, it's very important to run regular data scrubs. I've written an article specifically about this topic.

Silent data corruption

An unrecoverable read error means that we can't read (a portion of) a file. Although it is unfortunate - because we better have an intact backup of that file - we are also fortunate.

Why are we fortunate?

We are fortunate because the storage system - the hard drive and in turn the operating system - reported an error. We were able to diagnose the problem an take action.

But it is possible that bits and bytes get mangled without your hard drive, SATA controller or operating system noticing. Somewhere, somehow, a bit is read or transmitted as a 1 where it should have been a 0.

This is really bad, because this data corruption is undetected, it is 'silent', there is no notification.

Because imagine what happens: the corrupted file is happily backed up by your backup software, because it's unaware that anything is wrong. And by the time you discover the data corruption, the original pristine file is no longer part of the backup (rotated out). You are left with a lot of backups of a corrupted file. We encounter dataloss.

This is one of the scariest kinds of data loss. Because it's very difficult to detect. You'll have to constantly calculate the checksum of a file and verify it's still ok.

And that's - although rather simplified - exactly what ZFS does (amongst many other things). ZFS uses checksums at the block-level and thus assures with every read if the data contained in the block is still valid. ZFS is one of the few file systems that has this very powerfull feature (BTRFS is another example).

Regular RAID arrays (be it hardware-based or software-based) cannot detect silent data corruption (although it could be possible with RAID6). So it must be clear that ZFS is capable of protecting against a risk 'regular' RAID cannot cope with.

Is silent data corruption a significant threat for home DIY NAS builders?

Although silent data corruption is a very scary threat, from what I can tell there is no significant independant evidence that the risk of silent data corruption is so high that the average home DIY NAS builder should take this risk into account².

Maybe I'm wrong, but I think many people mistakenly confuse UREs or unrecoverable read errors (caused by bad sectors) with silent data corruption. And I think that's wrong, because there's nothing silent about an unrecoverable read error.

The truth is that hard drives are in fact very reliable when it comes to silent data corruption, because they make heavy use of error detection and correction algoritms. A significant portion of the raw capacity of a hard drive is sacrificed to store redundant information to aid in detecting and correcting data corruption. According to wikipedia, hard drives used Reed-Solomon error correction in the past and more modern drives use LDPC.

These error correction codes asure data integrity. Although 'soft' read errors may occur, there is enough additional redundant information stored on the hard drive to detect errors and even reconstruct the data (to some extend). Your hard drive handles this all by itself, it's part of normal operation.

So this is my point: it's important to understand that there is a lot of protection against silent data corruption in a hard drive. The risk of silent data corruption is therefore small³.

Sometimes the read data is so garbled that even the error correction codes cannot reconstruct the data as it was originally stored and that's what we then experience as an unrecoverable read error. But the disk notices! And it will report it!. This is not silent at all!

To really create silent data corruption, something very special need to happen. And to be very clear: such events do happen. But they are very rare.

Somehow, a bit must flip and this event is not detected by the error correction algorithm. Maybe the bit flipped in the hard drive cache memory when it was read from the drive. Maybe it flipped during transport over the SATA cable.

But it's fun to realise that the SATA protocol also has error detection embedded in the protocol for reliable data transmission. It's error detection and correction all the way down.

The risk that silent data corruption happens is thus very small, especially for home users.

Again, make no mistake: the risk is real and storage solutions for larger scale storage solutions (SANs / Storage arrays) with hundreds, thousands or tens of thousands of drives do really have to take into account the risk of silent data corruption. At scale, even very small risks become a certainty.

Enterprise storage solutions often employ their own proprietary solutions to protect against silent data corruption. Although it depend on the particular solution⁴, it's often part of the storage array. ZFS was revolutionary because they put the data integrity checking in the filesystem itself.

So if you think the risk of silent data corruption is still high enough that you should protect yourself against it, I would recommend to consider using ECC memory to protect against corrupted data in memory. To be frank: I consider non-ECC memory a more likely cause of silent data corruption than the storage subsystem, which already employs all these error detection and correction algoritms. Non-ECC memory is totally unprotected.

Anekdote: I myself run a 24-drive NAS based on ZFS and it has been rock-solid for 6 years straight.

From time to time, I do run disk 'scrubs', which can take quite some time. Although I have many terrabytes of data protected by ZFS, not a single instance of silent data corruption has been detected. And I have performed so many scrubs that I've read more than a petabyte worth of data.

Anekdote: Somebody made a mistake and used the wrong type of cable to connect the hard drives to the HBA controller card. This caused actual silent data corruption. Because that person was running ZFS, it was detected so ZFS saved his data. This an example where ZFS did protect a person against silent data corruption.

Evaluation

I hope that the difference between unrecoverable read errors and silent data corruption is clear and that we should not confuse the two. They have different risk profiles associated with them.

Furthermore, I have argued that silent data corruption is real and a serious issue at scale, and that it is that is dealt with accordingly.

However, I've also argued that unless you are a home user running a small datacenter inside your basement, the risk of silent data corruption is so small that it is reasonable to accept the risk as a DIY NAS builder and not seek specific protection against it.

The decision is up to you. If you want to go with ZFS and protect against silent data corruption, you should also be aware and accept the cost of ZFS. I myself have accepted that cost for my own NAS, but it's OK if you don't. If you care about silent data corruption so much, please also consider using ECC-memory.

But in my opinion, you are not taking an unreasonable risk if you chose to go with Unraid, Snapraid, Linux kernel RAID, Windows Storage Spaces or maybe other options in the same vein. I would say that this is reasonable and up to you.

Remember: the famous vendors of home user NAS boxes all seem to use regular Linux kernel RAID under the hood. And they seem to think that's fine.

In the end, what really matters is a solution that suits your needs and also fits your budget and level of expertise. Can you fix problems when something goes wrong?

I've noticed while testing with this particular drive that the drive was not kicked out of the array, and it just kept trying to read, grinding the Linux software RAID array to a halt. Removing the drive from the array fixed this. There is a 'failfast' option that only works with RAID1 or RAID10. ↩
I don't want to suggest in any way that it would be wrong to take silent data corruption into account, but just to say I think it's not mandatory to really fret over it. ↩
The most significant risk is that enterprise grade hard drives use on-board ECC cache memory, whereas consumer drives use non-ECC cache memory. So silently corrupted data in the cache memory of the drive could be a risk. ↩
Storage vendors often choose to reformat har drives with larger sector sizes⁵. Those larger sectors then also incorporate additional checksum data to better protect against data corruption or unrecoverable read errors. ↩
https://www.seagate.com/files/staticfiles/docs/pdf/whitepaper/safeguarding-data-from-corruption-technology-paper-tp621us.pdf ↩

Scrub your NAS hard drives regularly if you care about your data

2020-04-22T12:00:00+02:00

Introduction

Lots of people run a NAS at home. Maybe it's a COTS device from one of the well-known vendors¹, or it's a custom build solution (DIY²) based on hardware you bought and assembled yourself.

Buying or building a NAS is one thing, but operating it in a way that assures that you won't lose data is something else.

Obviously, the best way to protect against dataloss, is to make regular backups. So ideally, even if the NAS would go up in flames, you would still have your data.

Since backup storage costs money, people make tradeoffs. They may decide to take the risk and only backup a small portion of the really important data and take their chances with the rest.

Well that is their own right. But still, it would be nice if we would reduce the risk of dataloss to a minimum.

The risk: bad sectors

The problem is that hard drives may develop bad sectors over time. Bad sectors are tiny portions of the drive that have become unreadable³. How small a sector may be, if any data is stored in them, it is now lost and this could cause data corruption (one or more corrupt files).

This is the thing: those bad sectors may never be discovered until it is too late!

With todays 14+ TB hard drives, it's easy to store vast amounts of data. Most of that data is probably not frequently accessed, especially at home.

One or more of your hard drives may be developing bad sectors and you wouldn't even know it. How would you?

Your data might be at risk right at this moment while you are reading this article.

A well-known disaster scenario in which people tend to lose data is double hard drive failure where only one drive faillure can be tolerated (RAID 1 (mirror) or RAID 5, and in some scenario's RAID 10).

In this scenario, a hard drive in their RAID array has failed and a second drive (one of the remaining good drives) has developed bad sectors. That means effectively a second drive has failed although the drive may still seem operational. Due to the bad sectors, data required to rebuild the array is lost because there is no longer any redundancy⁴.

If you run a (variant of) RAID 5, you can only lose a single disk, so if a second disk fails, you lose all data⁵.

The mitigation: periodic scrubbing / checking of your disks

The only way to find out if a disk has developed bad sectors is to just read them all. Yes: all the sectors.

Checking your hard drives for bad sectors (or other issues) is called 'data scrubbing'. If you bought a NAS from QNAP, Synology or another vendor, there is a menu which allows you to control how often and when you want to perform a data scrub.

RAID solutions are perfectly capable of handling bad sectors. For a RAID array, it's just equivalent to a failed drive and an affected drive will be kicked out of the RAID array if bad sectors start causing read errors. The big issue we want to prevent is that multiple drives start to develop bad sectors at the same time, because that is the equivalent of multiple simultaneous drive failures, which many RAID arrays can't recover from.

For home users I would recommend checking all hard drives once a month. I would recommend configuring the data scrub to run at night (often the default) because a scrub may impact performance in a way that can be noticeable and even inconvenient.

Your vendor may have already configured a default schedule for data scrubs, so you may have been protected all along. If you take a look, at least you know.

People who have built a DIY NAS have to setup and configure periodic scrubs themselves or they won't happen at all. However, that's not entirely true: I've noticed that on Ubuntu, all Linux software RAID arrays (MDADM) are checked once a month at night. So if you use Linux software RAID you may already be scrubbing.

A drive that develops bad sectors should be replaced as soon as possible. It should no longer be trusted. The goal of scrubbing is to identify these drives as soon as possible. You don't want to get in a position that multiple drives have started developing bad sectors. You can only prevent that risk by scanning for bad sectors periodically and replacing bad drives.

You should not be afraid about having to spend a ton of money replacing drives all the time. Bad sectors are not that common. But they are a common enough that you should check for them. There is a reason why NAS vendors offer the option to run data scrubs and recommend them⁶.

You probably forgot to configure email alerting

If a disk in your NAS would fail, how would you know? If the scrub would discover bad sectors, would you ever notice⁷?

The answer may be: only when it's too late. Maybe a drive already failed and you haven't even noticed yet!

When you've finished reading this article, it may be the right moment to take some time to check the status of your NAS and configure email alerting (or any other alerting mechanism that works for you). Make your NAS sends out a test message just to confirm it actually works!

Closing words

So I would like to advice you to do two things:

Make sure your NAS runs a data scrub once a month
Make sure your NAS is able to email alerts about failed disks or scrubs.

These actions allow you to fix problems before they become catastrophic.

P.S. S.M.A.R.T. monitoring

Hard drives have a build-in monitoring system called S.M.A.R.T.

If you have a NAS from one of the NAS vendors, they will allert on SMART monitoring information that would indicate that a drive is failing. DIY builders may have to spend time setting up this kind of monitoring manually.

For more information about SMART I would recommend [this][this article] and this one.

this article and also this one

Linux users can take a look at the SMART status of their hard drives with this tool (which I made).

QNAP, Synology, Netgear, Buffalo, Thecus, Western Digital, and so on. ↩
FreeNAS, Unraid, Windows/Linux with Snapraid, OpenMediaVault, or a custom solution, and so on. ↩
Bad sectors cause 'unrecoverable read errors' or UREs. Bad sectors have nothing to do with 'silent data corruption'. There's nothing silent about unrecoverable read errors. Hard drives report read errors back to the operating system, they won't go unnoticed. ↩
A DIY NAS based on ZFS (FreeNAS is based on ZFS) may help mitigate the impact of such an event. ZFS can continue reading data from the remaining drives, even if bad sectors are encountered. Some files will be corrupted, but most of the data would still be readable. I think this capability is by itself not enough reason to pick a NAS based on ZFS because ZFS also has a cost involved that you need to accept too. For my large NAS I have chosen ZFS because I was prepared to 'pay the cost'. ↩
Some people may chose to go with RAID 6 which tolerates two simultaneous drive failures but they also tend to run larger arrays with more drives, which also increases the risk of drive failure or one of the drives developing bad sectors. ↩
Enterprise storage solutions (Even entry level storage arrays) often run patrol reads both on individual hard drives and also the RAID arrays on top of them. They are also enabled by default. ↩
At one time I worked for a small company that ran their own (single) email server. One of the system administrators discovered totally by accident that one of the two drives in a RAID 1 had failed. It turns out we were running on a single drive for months before we discovered it, because we forgot to setup email alerting. We didn't lose data, but we came close. ↩

Benchmarking storage with Fio and generating charts of the results

2020-04-21T12:00:00+02:00

Introduction

Fio is a widely-used tool for performing storage benchmarks. Fio offers a lot of options to create a storage benchmark that would best reflect your needs. Fio allows you to assess if your storage solution is up to its task and how much headroom it has.

Fio outputs .json and .log files that need further processing if you would like to make nice charts. Charts may help better communicate your test results to other people.

To make graphs of Fio benchmark data, I've created fio-plot. With fio-plot you can generate charts like:

It's very common that you want to run multiple benchmarks with different parameters to compare results. To generate the data of the charts, many benchmarks need to be run. This process needs to be automated.

Automating Fio benchmarks

I've chosen to build my own tool to automate Fio benchmarking. This tool is called bench_fio and is part of fio-plot. I'm aware that - as part of fio - a tool called genfio is provided, to generate fio job files with multiple benchmarks. It's up to you what you want to use. Bench-fio is tailored to output data in a way that aligns with fio-plot.

Bench-fio allows you to benchmark loads with different iodepths, simultaneous jobs, block sizes and other parameters. A benchmark run can consist of hundreds of tests and take many hours.

When you run bench_fio, you can expect output like this:

████████████████████████████████████████████████████
    +++ Fio Benchmark Script +++

Job template:                  fio-job-template.fio
I/O Engine:                    libaio
Number of benchmarks:          98
Estimated duration:            1:38:00
Devices to be tested:          /dev/md0
Test mode (read/write):        randrw
IOdepth to be tested:          1 2 4 8 16 32 64
NumJobs to be tested:          1 2 4 8 16 32 64
Blocksize(s) to be tested:     4k
Time per test (s):             60
Mixed workload (% Read):       75 90

████████████████████████████████████████████████████
4% |█                        | - [0:04:02, 1:35:00]-]

Bench-fio runs real-time and shows the expected remaining time. It also shows all relevant parameters that have been configured for this benchmark run. This makes it easier to spot any mis-configurations.

Notice that this benchmark consists of 98 individual tests: iodepth x NumJobs x Mixed Workload parameters (7 x 7 x 2). With a standard of 60 seconds per benchmark

This is an example of the command-line syntax: :::text ./bench_fio --target /dev/md0 -t device --mode randrw -o RAID_ARRAY --readmix 75 90

More examples can be found here.

The impact of the MDADM bitmap on RAID performance

2020-04-06T12:00:00+02:00

Introduction

I'm aware that most people with intensive storage workloads won't run those workloads on hard drives anymore, that ship has sailed a long time ago. SSDs have taken their place (or 'the cloud').

For those few left who do use hard drives in Linux software RAID setups and run workloads that generate a lot of random IOPS, this may still be relevant.

I'm not sure how much a bitmap affects MDADM software RAID arrays based on solid state drives as I have not tested them.

The purpose of the bitmap

By default, when you create a new software RAID array with MDADM, a bitmap is also configured. The purpose of the bitmap is to speed up recovery of your RAID array in case the array gets out of sync.

A bitmap won't help speed up the recovery from drive failure, but the RAID array can get out of sync due to a hard reset or power failure during write operations.

The performance impact

During some benchmarking of various RAID arrays, I noticed very bad random write IOPS performance. No matter what the test conditions were, I got the random write performance of a single drive, although the RAID array should perform better.

Then I noticed that the array was configured with a bitmap. Just for testing purposes, I removed the bitmap all together with:

mdadm --grow --bitmap=none /dev/md0

Random write IOPs figures improved immediately. This resource explains why:

If  the  word internal is given, then the bitmap is stored with the metadata
on the array, and so is replicated on all devices.

So when you write data to our RAID array, the bitmap is also constantly updated. Since that bitmap lives on each drive in the array, it's probably obvious that this really deteriorates random write IOPS.

Some examples of the performance impact

Bitmap disabled

An example of a RAID 5 array with 8 x 7200 RPM drives.

Another example with 10.000 RPM drives:

Bitmap enabled (internal)

We observe significant lower random write IOPs performance overall:

Which is also true for 10.000 RPM drives.

External bitmap

You could keep the bitmap and still get great random write IOPS by putting the bitmap on a separate SSD. Since my boot device is an SSD, I tested this option like this:

mdadm --grow --bitmap=/raidbitmap /dev/md0

I noticed excellent random write IOPS with this external bitmap, similar to running without a bitmap at all. An external bitmap has it's own risks and caveats, so make sure it really fits your needs.

Note: external bitmaps are only known to work on ext2  and  ext3. 
Storing bitmap files on other filesystems may result in serious problems.

Conclusion

For home users who build DIY NAS servers and who do run MDADM RAID arrays, I would recommend leaving the bitmap enabled. The impact on sequential file transfers is negligible and the benefit of a quick RAID resync is very obvious.

Only if you have a workload that would cause a ton of random writes on your storage server would I consider disabling the bitmap. An example of such a use case would be running virtual machines with a heavy write workload.

Update on bitmap-chunks

Based on feedback in the comments, I've performed a benchmark on a new RAID 5 array setting the --bitmap-chunk option to 128M (Default is 64M).

The results seem to be significantly worse than the default for random write IOPS performance.

Understanding Storage Performance - IOPS and Latency

2020-03-21T12:00:00+01:00

Introduction

The goal of this blogpost is to help you better understand storage performance. I want to discuss some fundamentals that are true regardless of your particular needs.

This will help you better reason about storage and may provide a scaffolding for further learning.

If you run your applications / workloads entirely in the cloud, this information may feel antiquated or irrelevant.

However, since the cloud is just somebody else's compute and storage, knowledge about storage may still be relevant. Cloud providers expose storage performance metrics for you to monitor and this may help to make sense of them.

Concepts

I/O

An I/O is a single read/write request. That I/O is issued to a storage medium (like a hard drive or solid state drive).

It can be a request to read a particular file from disk. Or it can be a request to write some data to an existing file. Reading or writing a file can result in multiple I/O requests.

I/O Request Size

The I/O request has a size. The request can be small (like 1 Kilobyte) or large (several megabytes). Different application workloads will issue I/O operations with different request sizes. The I/O request size can impact latency and IOPS figures (two metrics we will discuss shortly).

IOPS

IOPS stands for I/O Operations Per Second. It is a performance metric that is used (and abused) a lot in the world of storage. It tells us how many I/O requests per second can be handled by the storage (for a particular workload).

Warning: this metric is meaningless without a latency figure. We will discuss latency shortly.

Bandwidth or throughput

If you multiply the IOPS figure with the (average) I/O request size, you get the bandwidth or throughput. We state storage bandwidth mostly in Megabytes and Gigabytes per second.

To give you an example: if we issue a workload of 1000 IOPS with a request size of 4 Kilobytes, we will get a throughput of 1000 x 4 KB = 4000 KB. This is about ~4 Megabytes per second.

Latency

Latency is the time it takes for the I/O request to be completed. We start our measurement from the moment the request is issued to the storage layer and stop measuring when either we get the requested data, or get confirmation that the data is stored on disk.

Latency is the single most important metric to focus on when it comes to storage performance, under most circumstances.

For hard drives, an average latency somewhere between 10 to 20 ms is considered acceptable (20 ms is the upper limit).

For solid state drives, depending on the workload it should never reach higher than 1-3 ms. In most cases, workloads will experience less than 1ms latency numbers.

IOPS and Latency

This is a very important concept to understand. The IOPS metric is meaningless without a statement about latency. You must understand how long each I/O operation will take because latency dictates the responsiveness of individual I/O operations.

If a storage solution can reach 10,000 IOPS but only at an average latency of 50 ms that could result in very bad application performance. If we want to hit an upper latency target of 10 ms the storage solution may only be capable of 2,000 IOPS.

For more details on this topic I would recommend this blog and this blog.

Access Patterns

Sequential access

An example of a sequential data transfer is copying a large file from one hard drive to another. A large number of sequential (often adjacent) datablocks is read from the source drive and written to another drive. Backup jobs also cause sequential access patterns.

In practice this access pattern shows the highest possible throughput.

Hard drives have it easy as they don't have to spend much time moving their read/write heads and can spend most time reading / writing the actual data.

Random access

I/O requests are issued in a seemingly random pattern to the storage media. The data could be stored all over various regions on the storage media. An example of such an access pattern is a heavy utilised database server or a virtualisation host running a lot of virtual machines (all operating simultaneously).

Hard drives will have to spend a lot of time moving their read/write heads and can only spend little time transferring data. Both throughput and IOPS will plummet (as compared to a sequential access pattern).

In practice, most common workloads, such as running databases or virtual machines, cause random access patterns on the storage system.

Queue depth

The queue depth is a number between 1 and ~128 that shows how many I/O requests are queued (in-flight) on average. Having a queue is beneficial as the requests in the queue can be submitted to the storage subsystem in an optimised manner and often in parallel. A queue improves performance at the cost of latency.

If you have some kind of storage performance monitoring solution in place, a high queue depth could be an indication that the storage subsystem cannot handle the workload. You may also observe higher than normal latency figures. As long as latency figures are still within tolerable limits, there may be no problem.

Storage Media Performance characteristics

Hard drives

Hard drives (HDDs) are mechanical devices that resemble a record player.

They have an arm with a read/write head and the data is stored on (multiple) platters.

Hard drives have to physically move read/write heads to fulfil read/write requests. This mechanical nature makes them relatively slow as compared to solid state drives (which we will cover shortly).

Especially random access workloads cause hard drives to spend a lot of time on moving the read/write head to the right position at the right time, so less time is available for actual data transfers.

The most important thing to know about hard drives is that from a performance perspective (focussing on latency) higher spindle speeds reduce the average latency.

Rotational Speed (RPM)	Access Latency(ms)	IOPS
5400	17-18	50-60
7200	12-13	75-85
10,000	7-8	120-130
15,000	5-6	150-180

Because the latency of individual I/O requests is lower the drives with a higher RPM, you can issue more of such requests in the same amount of time. That's why the IOPS figure also increases.

Latency and IOPS of an older Western Digital Velociraptor 10,000 RPM drive:

Notice the latency and IOPS in the Queue Depth = 1 column.

Source used to validate my own research.

Regarding sequential throughput we can state that fairly old hard drives can sustain throughputs of 100-150 megabytes per second. More modern hard drives with higher capacities can often sustain between 200 - 270 megabytes per second.

An important note: sequential transfer speeds are not constant and depend on the physical location of the data on the hard drive platters. As a drive fills up, throughput diminishes. Throughput can drop more than fifty percent! ¹.

So if you want to calculate how long it will take to transfer a particular (large) dataset, you need to take this into account.

Solid State Drives

Solid state drives (SSDs) have no moving parts, they are based on flash memory (chips). SSDs can handle I/O much faster and thus show significantly lower latency.

Whereas we measure the average I/O latency of HDDs in milliseconds (a thousand of a second) we measure the latency of SSD I/O operations in microseconds (a millionth of a second).

Because of this reduced latency per I/O request, SSDs outperform HDDs in every conceivable way. Even a cheap consumer SSD can at least sustain about 5000+ IOPS with only a 0.15 millisecond (150 microseconds) latency. That latency is about 40x better than the best latency of an enterprise 15K RPM hard drive.

Solid state drives can often handle I/O requests in parallel. This means that larger queue depths with more I/O requests in flight can show significantly higher IOPS with a limited (but not insignificant) increase in latency.

The random I/O performance of an older SATA consumer SSD

More modern enterprise SSDs show better latency and IOPS. The SATA interface seems the main bottleneck.

The random I/O performance of an enterprise SATA SSD

SSDs perform better than HDDs across all relevant metrics except price in relation to capacity.

Important note: SSDs are not well-suited for archival storage of data. Data is stored as charges in the chips and those charges can diminish over time. It's expected that even hard drives are better suited for offline archival purposes although the most suitable storage method would probably be tape.

SSD actual performance vs advertised performance

Many SSDs are advertised with performance figures of 80,000 - 100,000 IOPS at some decent latency. Depending on the workload, you may only observe a fraction of that performance.

Most of those high 80K-100K IOPS figures are obtained by benchmarking with very high queue depths (16-32). The SSD benefits from such queue depths because it can handle a lot of those I/O requests in parallel.

Please beware: if your workload doesn't fit in that pattern, you may see lower performance numbers.

If we take a look at the chart above of the Intel SSD we may notice how the IOPS figures only start to come close to the advertised 80K+ IOPS as the queue depth increases. It's therefore important to understand the characteristics of your own workload.

RAID

If we group several hard drives together we can create a RAID array. A RAID array is a virtual storage device that exceeds the capacity and performance of a single hard drive. This allows storage to scale within the limits of a single computer.

RAID is also used (or some say primarily used) to assure availability by assuring redundancy (drive failure won't cause data loss). But for this article we focus it's performance characteristics.

SSDs can achieve impressive sequential throughput speeds, of multiple gigabytes per second. Individual hard drives can never come close to those speeds, but if you put a lot of them together in a RAID array, you can come very close. For instance, my own NAS an achieve such speeds using 24 drives.

RAID also improves the performance of random access patterns. The hard drives in a RAID array work in tandem to service those I/O requests so a RAID array shows significantly higher IOPS than a single drive. More drives means more IOPS.

RAID 5 with 8 x 7200 RPM drives

The picture below shows the read IOPS performance of an 8-drive RAID 5 array of 1 TB, 7200 RPM drives. We run a benchmark of random 4K read requests.

Notice how the IOPS increases as the queue depth increases.

However, nothing is free in this world. A higher queue depth - which acts as a buffer - does increase latency.

Notice how quickly the latency exceeds 20ms and quickly becomes almost unusable.

RAID 5 with 8 x 10,000 RPM drives

Below is the result of a similar test with 10,000 RPM hard drives. Notice how much better the IOPS and latency figures are.

The latency looks much better:

It makes sense to put SSDs in RAID. Although they are more reliable than hard drives, they can fail. If you care about availability, RAID is inevitable. Furthermore, you can observe the same benefits as with hard drives: you pool resources together, achieving higher IOPS figures and more capacity than possible with a single SSD.

Capacity vs. Performance

The following is mostly focussed on hard drives although it could be true for solid state drives as well.

We put hard drives in RAID arrays to get more IOPS than a single drive can provide. At some point - as the workload increases - we may hit the maximum number of IOPS the RAID array can sustain with an acceptable latency.

This IOPS/Latency threshold could be reached even if we have only 50% of the storage capacity of our RAID array in use. If we use the RAID array to host virtual machines for instance, we cannot add more virtual machines because this would cause the latency to rise to unacceptable levels.

It may feel like a lot of good storage space is going to waste, and in some sense this may be true. For this reason, it could be a wise strategy to buy smaller 10,000 RPM or 15,000 RPM drives purely for the IOPS they can provide and forgo on capacity.

So it might be the case that you may have to order and add let's say 10 more hard drives to meet the IOPS/Latency demands while there's still plenty of space left.

This kind of situation is less likely as SSDs have taken over the role of the performance storage layer and (larger capacity) hard drives are pushed in the role of 'online' archival storage.

Closing words

I hope this article has given you a better understanding of storage performance. Although it is just an introduction, it may help you to better understand the challenges of storage performance.

https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics#Data_transfer_rate ↩

Difference of behavior in SATA Solid State Drives

2020-01-29T00:00:00+01:00

Introduction

Update: I've noticed some strange behavior of SSDs when benchmarking them with FIO. After further investigation and additional testing, I've found the reason for the strange patterns in the graphs.

The 'strange' test results are due to the fact that they were obtained by connecting the SSDs to a P420I controller. As the HBA mode of this controller performs worse than the RAID mode, I used the RAID mode of this controller. Indvidual drives were put in a RAID0 volume. But it turns out that this creates a strange interaction between RAID controller and SSD.

Additional testing with an SATA 300 AHCI controller shows 'normal' patterns that look similar to the results of the INTEL SSD as compared to the other ones (Samsung and Kingston).

It seems I've made a mistake by using the P420i controller for testing. I have includes both 'bad' and 'good' results.

Regular SATA solid state drives may seem interchangeable at this point. They all show amazing IOPS and latency performance.

I have performed benchmarks on different SSDs from different vendors and it seems that they actually show very different behaviour. This behavior has come to light because I benchmarked the entire device capacity.

The benchmark - performed with FIO - puts a fifty percent read/write random 4K workload on the device. The benchmark stops when all sectors of the device have been read or written to. Furthermore, all tests are performed with a queue depth of 1.

I've made this post because I found the results interesting. At least the images show a very peculiar pattern for some SSDs. I can't explain them really, maybe you can.

This is the test I ran against the SSDs.

fio --filename=/dev/sdX --direct=1 --rw=randrw --refill_buffers
--norandommap --ioengine=libaio --bs=4k --rwmixread=50 --iodepth=1

Disclaimer

I've performed these benchmark to the best of my knowledge. The raw benchmark data is available here

It's always possible that I made a mistake, so it may be wise to run your own tests to see if you can replicate these results.

Caveat: I really don't know if these benchmark results impact real-life performance. Maybe these benchmark results show a kind of behaviour of SSDs that doesn't really matter in the end.

Benchmark Results

Intel D3-S4610

This SSD is meant for for datacenter usage. This is the test result on the P420i controller.

IOPS and Latency is consistent during the whole benchmark. It's behaviour seems predictable.

This is the test result on the AHCI controller:

Samsung 860 Pro

This SSD is meant for desktop usage. Its behavior seems quite different from the Intel SSD. I have separated the IOPS data from the Latency data to make the graphs more eligible.

This is the test result on the P420i controller.

IOPS

Latency

The best-case latency is almost four times better than the worst-case latency. Latency is thus less predictable. This impact also seems to be reflected in the IOPs numbers.

This is the test result on the AHCI controller:

Samsung PM883

This SSD is meant for datacenter usage. This is the test result on the P420i controller.

IOPS

Latency

This SSD seems to behave in a similar way as the 860 PRO.

This is the test result on the AHCI controller:

Kingston DC500M

This SSD is meant for datacenter usage. This is the test result on the P420i controller.

IOPS

latency

The behavior of this SSD seems similar to the behaviour of the Samsung SSDs but the pattern is distinct: it seems shifted as compared to the Samsung SSDs.

This is the test result on the AHCI controller:

Evaluation

** Updated evaluation**

We can conclude that the P420i RAID controller causes strange behavior not observed when we test the SSDs on a regular AHCI controller. Although this was an older SATA 300 controller, I'm making the assumption that this controller still has enough bandwidth to support a random 4K test as most tests never went beyond 50+ MB/s of throughput.

At this point, I can only say that I observe quite different behavior between the Intel SSD and the other SSDs from Samsung and Kingston. The problem is that I can't tell if this affects real-life day-to-day application performance.

It seems that although results for the Samsung and Kingston SSDs fluctuate quite a bit, it's quite possible that the fluctuations occur during a very short timespan and effectively cancel each other out.

If you have comments, ideas or suggestions, leave a comment below.

How are these images generated?

All images have been generated with fio-plot.

The github repository also contains a folder with a lot of example images.

My Ceph test cluster based on Raspberry Pi's and HP MicroServers

2019-01-27T00:00:00+01:00

Introduction

To learn more about Ceph, I've build myself a Ceph Cluster based on actual hardware. In this blogpost I'll discus the cluster in more detail and I've also included (fio) benchmark results.

This is my test Ceph cluster:

The cluster consists of the following components:

 3 x Raspberry Pi 3 Model B+ as Ceph monitors
 4 x HP MicroServer as OSD nodes (3 x Gen8 + 1 x Gen10)
 4 x 4 x 1 TB drives for storage (16 TB raw)
 3 x 1 x 250 GB SSD (750 GB raw)
 2 x 5-port Netgear switches for Ceph backend network (bonding)

Monitors: Raspberry Pi 3 Model B+

I've done some work getting Ceph compiled on a Raspberry Pi 3 Model B+ running Raspbian. I'm using three Raspberry Pi's as Ceph monitor nodes. The Pi boards don't break a sweat with this small cluster setup.

Note: Raspberry Pi's are not an ideal choice as a monitor node because Ceph Monitors write data (probably the cluster state) to disk every few seconds. This will wear out the SD card eventually.

Storage nodes: HP MicroServer

The storage nodes are based on four HP MicroServers. I really like these small boxes, they are sturdy, contain server-grade components, including ECC-memory and have room for four internal 3.5" hard drives. You can also install 2.5" hard drives or SSDs.

For more info on the Gen8 and the Gen10 click on their links.

Unfortunately the Gen8 servers are no longer made. The replacement, the Gen10 model, lacks IPMI/iLO and is also much more expensive (in Europe at least).

CPU and RAM

All HP Microservers have a dual-core CPU. The Gen8 servers have 10GB RAM and the Gen10 server has 12GB RAM. I've just added an 8GB ECC memory module to each server, the Gen10 comes with 4GB and the Gen8 came with only 2GB, which explains the difference.

Boot drive

The systems all have an (old) internal 2.5" laptop HDD connected to the internal USB 2.0 header using an USB enclosure.

Ceph OSD HDD

All servers are fitted with four (old) 1TB 7200 RPM 3.5" hard drives, so the entire cluster contains 16 x 1TB drives.

Ceph OSD SSD

There is a fifth SATA connector on the motherboard, meant for an optional optical drive, which I have no use for and wich is not included with the servers.

I use this SATA connector in the Gen8 MicroServers to attach a Crucial 250GB SSD, which is then tucked away at the top, where the optical drive would sit. So the Gen8 servers have an SSD installed which the Gen10 is lacking.

The entire cluster thus has 3 x 250GB SSDs installed.

Networking

All servers have two 1Gbit network cards on-board and a third one installed in one of the half-height PCIe slots¹.

The half-height PCIe NICs connect the Microservers to the public network. The internal gigabit NICs are configured in a bond (round-robin) and connected to two 5-port Netgear gigabit switches. This is the cluster network or the backend network Ceph uses for replicating data between the storage nodes.

You may notice that the first onboard NIC of each server is connected to the top switch and the second one is connected to the bottom switch. This is necessary because linux round-robin bonding requires either separate VLANs for each NIC or in this case separate switches.

Benchmarks

Benchmark conditions

The tests ran on a physical Ceph client based on an older dual-core CPU and 8GB of RAM. This machine was connected to the cluster with a single gigabit network card.
I've mapped RBD block devices from the HDD pool and the SSD pool on this machine for benchmarking.
All tests have been performed on the raw /dev/rbd0 device, not on any file or filesystem.
The pools use replication with a minimal copy count of 1 and a maximum of 3.
All benchmarks have been performed with FIO.
All benchmarks used random 4K reads/writes

    NAME     ID     USED        %USED     MAX AVAIL     OBJECTS
    hdd      36     1.47TiB     22.64       5.03TiB      396434
    ssd      38      200GiB     90.92       20.0GiB       51204

Benchmark SSD

Click on the images below to see a larger version.

Benchmark HDD

Benchmark evaluation

The random read performance of the hard drives seems unrealistic at higher queue depths and number of simultaneous jobs. This performance cannot be sustained purely on the basis that 16 hard drives with maybe 70 random IOPs each can only sustain 1120 random IOPs.

I cannot explain why I get these numbers. If anybody has a suggestion, feel free to comment/respond. Maybe the total of 42GB of memory across the cluster may act as some kind of cache.

Another interesting observation is that a low number of threads and a small IO queue depth results in fairly poor performance, both for SSD and HDD media.

Especially the performance of the SSD pool is poor with a low IO queue depth. A probable cause is that these SSDs are consumer-grade and don't perform well with low queue depth workloads.

I find it interesting that even over a single 1Gbit link, the SSD-backed pool is able to sustain 20K+ IOPs at higher queue depths and larger number of threads.

The small number of storage nodes and the low number of OSDs per node doesn't make this setup ideal but it does seem to perform fairly decent, considering the hardware involved.

You may notice that the Pi's are missing in the picture because this is an older picture when I was running the monitors as virtual machines on hardware not seen in the picture. ↩

Compiling Ceph on the Raspberry Pi 3B+ (armhf) using Clang/LLVM

2018-11-10T04:00:00+01:00

UPDATE 2019 / 2020

There are official ARM64 binaries of Ceph that you can run on a 64-bit version of Ubuntu 18.04.

Important: I consider this page obsolete. I will keep it up for transparency's sake

Introduction

In this blog post I'll show you how to compile Ceph Luminous for the Raspberry Pi 3B+.

If you follow the instructions below you can compile Ceph on Raspbian. A note of warning: we will compile Ceph on the Raspberry Pi itself which takes a lot of time.

Ubuntu has packages for Ceph on armhf but I was never able to get Ubuntu working properly on the Raspberry Pi 3B+. Maybe that's just me and I did something wrong. Using existing Ceph packages on Ubuntu would probably be the fastest way to get up and running on the Raspberry Pi if it works for you.

This is my test Ceph cluster:

 3 x Raspberry Pi 3B+ as Ceph monitors. 
 4 x HP Microserver as OSD nodes.
 4 x 4 x 1 TB drives for storage (16 TB raw)
 3 x 1 x 250 GB SSD (750 GB raw)
 2 x 5-port Netgear switches for Ceph backend network (bonding)

For the impatient

If you just want the packages you can download this file and you'll get a set of .deb files which you need to install on your Raspberry Pi.

SECURITY WARNING: these packages are created by me, an unknown, untrusted person on the internet. As a general rule you should not download and install these packages as they could be malicious for all you know. If you want to be safe, compile Ceph yourself.

Skip to the section about installing the packages at the end for further installation instructions.

The problem with compiling Ceph for armhf

There are no armhf packages for Ceph because if you try to compile Ceph on armhf the compiler (gcc) will run out of virtual memory (about three gigabytes).

The solution

Daniel Glaser discovered that he could actually compile Ceph on armhf by using Clang/LLVM as the C++ compiler. This compiler seems to use less memory and thus stay within the 3 GB memory boundary. This is why he and I were able to compile Ceph.

How to compile Ceph for armhf - preparation

The challenge: one gigabyte of memory

The Raspberry Pi 3B+ has only one gigabyte of memory but we need more. The only way to add memory is to use swap on disk, as far as I know.

If you use storage as a substitute for RAM memory, you need fast storage, so it's really recommended to use an external SSD drive that you connect through USB. You also may need sufficient storage, I'd recommend 20+ GB.

SD memory cards are not up to the task regarding being used as swap. You'll wear them out prematurely and performance is abysmal. You should really use an external SSD.

Preparing the external SSD

Attach the SSD drive to the Raspberry Pi with USB

The SSD will probably show up as '/dev/sda'.
mkfs.xfs /dev/sda -f ( this will erase all contents of the SSD ).
mkdir /mnt/ssd
mount /dev/sda /mnt/ssd

Creating and activating swap

cd /mnt/ssd
dd if=/dev/zero of=swap.dd bs=1M count=5000
swapon /mnt/ssd/swap.dd
swapoff /var/swap

By default, Raspbian configures a 100 MB swap file on /var/swap. In order to increase performance and protect the SD card from wearing out, please don't forget this last step to disable this swap file on the SD card.

Extra software

I would recommend installing 'htop' for real-time monitoring of cpu, memory and swap usage if you like to do so.

apt-get install htop

How to compile Ceph for armhf - building

Installing an alternative C++ compiler (Clang/LLVM)

As part of Daniel's instructions, you need to compile and install Clang/LLVM. I followed his instructions to the letter, I have not tested the Clang/LLVM packages made available through apt.

Compiling Clang/LLVM takes a lot of time. It took 8 hours to compile LLVM/Clang on a Raspberry Pi 3B+ with make -j3 to limit memory usage.

real    493m38.472s
user    1223m39.063s
sys 45m45.748s

I'll reproduce the steps from Daniel here:

apt update
apt install -y build-essential ca-certificates vim git 
apt install libcunit1-dev libcurl4-openssl-dev python-bcrypt python-tox python-coverage

cd /mnt/ssd
mkdir git && cd git
git clone https://github.com/llvm-mirror/llvm.git
cd llvm/tools
git clone https://github.com/llvm-mirror/clang.git
git clone https://github.com/llvm-mirror/lld.git
cd /tmp
mkdir llvm-build && cd llvm-build
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=ARM /mnt/ssd/git/llvm/
make -j3
make install
update-alternatives --install /usr/bin/cc cc /usr/local/bin/clang 100
update-alternatives --install /usr/bin/c++ c++ /usr/local/bin/clang++ 100
update-alternatives --install /usr/bin/cpp cpp /usr/local/bin/clang-cpp 100

You may chose to build in some other directory, maybe on the SSD itself. I'm not sure if that makes a big difference. Be carefull when using /tmp as all contents are lost after a reboot.

Obtaining Ceph

There are two options: 1. clone my Luminous fork containing the branch 'ceph-on-arm' which incorporates all the 'fixed' files that make Ceph build with Clang/LLVM.

Clone the official Ceph repo and use the luminous branche. Next, you edit all the relevant files and make the changes yourself. Here you can find a list of all the files and the changes made to them.

I would recommend to just git clone ceph like this:

cd /mnt/ssd
git clone https://github.com/louwrentius/ceph
cd ceph
git checkout ceph-on-arm
git reset --hard
git clean -dxf
git submodule update --init --recursive

Now we first need to install a lot of dependancies on the Raspberry Pi before we can build Ceph.

run ./install-deps.sh

This will take some time as a ton of packages will be installed. Once this is done we are ready to compile Ceph itself.

Building Ceph

So to understand what you are getting into: it took me about 12 hours to compile Ceph on a Raspberry Pi 3 B+

real    717m31.457s
user    1319m50.438s
sys 58m7.549s

This is the command to run:

./make-debs.sh

If you want to monitor cpu and memory usage, you can use 'htop' to do so.

If for some reason the compile proces does fail and you may have to restart compiling ceph after you made some adjustments:

(you may have to adjust the folder name to match your ceph version)

cd /tmp/release/Raspbian/WORKDIR/ceph-12.2.9-39-gd51dfb14f4
< edit the relevant files here >
dpkg-buildpackage -j3 -us -us -nc

Once this process is done, you will find a lot of .deb packages in your /tmp/release/Raspbian/WORKDIR folder.

Warning If you do use /tmp, the first thing to do is to copy all .deb files to a safe location because if you reboot your Pi, you loose 12 hours of work.

Assuming that you copied all .deb files to a folder like '/deb' you just created, this is how you install these packages:

dpkg --install *.deb
apt-get install --fix-missing
apt --fix-broken install

This is a bit ugly but it worked fine for me.

You can now just copy over all the .deb files to other Raspbery Pi's and install Ceph on them too.

Now you are done and you can run Ceph on a Raspberry Pi 3B+.

Ceph monitors may wear out the SD card

Important Running a Ceph monitor node on a Raspberry Pi is not ideal. The core issue is that the Ceph monitor process writes data every few seconds to files within /var/lib/ceph and this may wear out the SD card prematurely. The solution would be to use an external usb hard drive mounted through USB or a regular ssd which is way more resilient to writes than a regular SD card.

Understanding Ceph: open-source scalable storage

2018-08-19T04:00:00+02:00

Introduction

In this blog post I will try to explain why I believe Ceph is such an interesting storage solution. After you finished reading this blog post you should have a good high-level overview of Ceph.

I've written this blog post purely because I'm a storage enthusiast and I find Ceph interesting technology.

What is Ceph?

Ceph is a software-defined storage solution that can scale both in performance and capacity. Ceph is used to build multi-petabyte storage clusters.

For example, Cern has build a 65 Petabyte Ceph storage cluster. I hope that number grabs your attention. I think it's amazing.

The basic building block of a Ceph storage cluster is the storage node. These storage nodes are just commodity (COTS) servers containing a lot of hard drives and/or flash storage.

Example of a storage node

Ceph is meant to scale. And you scale by adding additional storage nodes. You will need multiple servers to satisfy your capacity, performance and resiliency requirements. And as you expand the cluster with extra storage nodes, capacity, performance and resiliency (if needed) will all increase at the same time.

It's that simple.

You don't need to start with petabytes of storage. You can actually start very small, with just a few storage nodes and expand as your needs increase.

I want to touch upon a technical detail because it illustrates the mindset surrounding Ceph. With Ceph, you don't even need a RAID controller anymore, a 'dumb' HBA is sufficient. This is possible because Ceph manages redundancy in software. A Ceph storage node at it's core is more like a JBOD. The hardware is simple and 'dumb', the intelligence resides all in software.

This means that the risk of hardware vendor lock-in is quite mitigated. You are not tied to any particular proprietary hardware.

What makes Ceph special?

At the heart of the Ceph storage cluster is the CRUSH algoritm, developed by Sage Weil, the co-creator of Ceph.

The CRUSH algoritm allows storage clients to calculate which storage node needs to be contacted for retrieving or storing data. The storage client can - on it's own - determine what to do with data or where to get it.

So to reiterate: given a particular state of the storage cluster, the client can calculate which storage node to contact for storage or retrieval of data.

Why is this so special?

Because there is no centralised 'registry' that keeps track of the location of data on the cluster (metadata). Such a centralised registry can become:

a performance bottleneck, preventing further expansion
a single-point-of-failure

Ceph does away with this concept of a centralised registry for data storage and retrieval. This is why Ceph can scale in capacity and performance while assuring availability.

At the core of the CRUSH algoritm is the CRUSH map. That map contains information about the storage nodes in the cluster. That map is the basis for the calculations the storage client need to perform in order to decide which storage node to contact.

This CRUSH map is distributed across the cluster from a special server: the 'monitor' node. Regardless of the size of the Ceph storage cluster, you typically need just three (3) monitor nodes for the whole cluster. Those nodes are contacted by both the storage nodes and the storage clients.

So Ceph does have some kind of centralised 'registry' but it serves a totally different purpose. It only keeps track of the state of the cluster, a task that is way easier to scale than running a 'registry' for data storage/retrieval itself.

It's important to keep in mind that the Ceph monitor node does not store or process any metadata. It only keeps track of the CRUSH map for both clients and individual storage nodes. Data always flows directly from the storage node towards the client and vice versa.

Ceph Scalability

A storage client will contact the appropriate storage node directly to store or retrieve data. There are no components in between, except for the network, which you will need to size accordingly¹.

Because there are no intermediate components or proxies that could potentially create a bottleneck, a Ceph cluster can really scale horizontally in both capacity and performance.

And while scaling storage and performance, data is protected by redundancy.

Ceph redundancy

Replication

In a nutshell, Ceph does 'network' RAID-1 (replication) or 'network' RAID-5/6 (erasure encoding). What do I mean by this? Imagine a RAID array but now also imagine that instead of the array consisting of hard drives, it consist of entire servers.

That's what Ceph does: it distributes the data across multiple storage nodes and assures that the copy of a piece of data is never stored on the same storage node.

This is what happens if a client writes two blocks of data:

Notice how a copy of the data block is always replicated to other hardware.

Ceph goes beyond the capabilities of regular RAID. You can configure more than one replica. You are not confined to RAID-1 with just one backup copy of your data². The only downside of storing more replicas is the storage cost.

You may decide that data availability is so important that you may have to sacrifice space and absorb the cost. Because at scale, a simple RAID-1 replication scheme may not sufficiently cover the risk and impact of hardware failure anymore. What if two storage nodes in the cluster die?

This example or consideration has nothing to do with Ceph, it's a reality you face when you operate at scale.

RAID-1 or the Ceph equivalent 'replication' offers the best overall performance but as with 'regular' RAID-1, it is not very storage space efficient. Especially if you need more than one replica of the data to achieve the level of redundancy you need.

This is why we used RAID-5 and RAID-6 in the past as an alternative to RAID-1 or RAID-10. Parity RAID assures redundancy but with much less storage overhead at the cost of storage performance (mostly write performance). Ceph uses 'erasure encoding' to achieve a similar result.

Erasure Encoding

With Ceph you are not confined to the limits of RAID-5/RAID-6 with just one or two 'redundant disks' (in Ceph's case storage nodes). Ceph allows you to use Erasure Encoding, a technique that let's you tell Ceph this:

"I want you to chop up my data in 8 data segments and 4 parity segments"

These segments are then scattered across the storage nodes and this allows you to lose up to four entire hosts before you hit trouble. You will have only 33% storage overhead for redundancy instead of 50% (or even more) you may face using replication, depending on how many copies you want.

This example does assume that you have at least 8 + 4 = 12 storage nodes. But any scheme will do, you could do 6 data segments + 2 parity segments (similar to RAID-6) with only 8 hosts. I think you catch the idea.

Ceph failure domains

Ceph is datacenter-aware. What do I mean by that? Well, the CRUSH map can represent your physical datacenter topology, consisting of racks, rows, rooms, floors, datacenters and so on. You can fully customise your topology.

This allows you to create very clear data storage policies that Ceph will use to assure the cluster can tollerate failures across certain boundaries.

An example of a topology:

If you want, you can lose a whole rack. Or a whole row of racks and the cluster could still be fully operational, although with reduced performance and capacity.

That much redundancy may cost so much storage that you may not want to employ it for all of your data. That's no problem. You can create multiple storage pools that each have their own protection level and thus cost.

How do you use Ceph?

Ceph at it's core is an object storage solution. Librados is the library you can include within your software project to access Ceph storage natively. There are Librados implementations for the following programming languages:

C(++)
Java
Python
PHP

Many people are looking for more traditional storage solutions, like block storage for storing virtual machines, a POSIX compliant shared file system or S3/OpenStack Swift compatible object storage.

Ceph provides all those features in addition to it's native object storage format.

I myself are mostly interested in block storage (Rados Block Device)(RBD) with the purpose of storing virtual machines. As Linux has native support for RBD, it makes total sense to use Ceph as a storage backend for OpenStack or plain KVM.

With very recent versions of Ceph, native support for iSCSI has been added to expose block storage to non-native clients like VMware or Windows. For the record, I have no personal experience with this feature (yet).

The Object Storage Daemon (OSD)

In this section we zoom in a little bit more into the technical details of Ceph.

If you read about Ceph, you read a lot about the OSD or object storage daemon. This is a service (daemon) that runs on the storage node. The OSD is the actual workhorse of Ceph, it serves the data from the hard drive or ingests it and stores it on the drive. The OSD also assures storage redunancy, by replicating data to other OSDs based on the CRUSH map.

To be precise: for every hard drive or solid state drive in the storage node, an OSD will be active. Does your storage node have 24 hard drives? Then it runs 24 OSDs.

And when a drive goes down, the OSD will go down too and the monitor nodes will redistribute an update CRUSH map so the clients are aware and know where to get the data. The OSDs also respond to this update, because redundancy is lost, they may start to replicate non-redundant data to make it redundant again (across fewer nodes).

When the drive is replaced, the cluster will 'self-heal'. This means that the new drive will be filled with data once again to make sure data is spread evenly across all drives within the cluster.

So maybe it's interesting to realise that storage clients effectively directly talk to the OSDs that in turn talk to the individual hard drives. There aren't many components between the client and the data itself.

Closing words

I hope that this blog post has helped you understand how Ceph works and why it is so interesting. If you have any questions or feedback please feel free to comment or email me.

If you have a ton of high-volume sequential data storage traffic, you should realise that a single host with a ton of drives can easily saturate 10Gbit or theoretically even 40Gbit. I'm assuming 150 MB/s per hard drive. With 36 hard drives you would face 5.4 GB/s. Even if you only would run half that speed, you would need to bond multiple 10Gbit interfaces to sustain this load. Imagine the requirements for your core network. But it really depends on your workload. You will never reach this kind of throughput with a ton of random I/O unless you are using SSDs, for instance. ↩
Please note that in production setups, it's the default to have a total of 3 instances of a data block. So that means 'the original' plus two extra copies. See also this link. Thanks to sep76 from Reddit to point out that the default is 3 instances of your data. ↩

Tracking down a faulty Storage Array Controller with ZFS

2016-12-15T12:00:00+01:00

One day, I lost two virtual machines on our DR environment after a storage vMotion.

Further investigation uncovered that any storage vMotion of a virtual machine residing on our DR storage array would corrupt the virtual machine's disks.

I could easily restore the affected virtual machines from backup and once that was done, continued my investigation.

I needed a way to quickly verifying if a virtual hard drive of a virtual machine was corrupted after a storage vMotion to understand what the pattern was.

First, I created a virtual machine based on Linux and installed ZFS. Then, I attached a second disk of about 50 gigabytes and formatted this drive with ZFS. Once I filled the drive using 'dd' to about 40 gigabytes I was ready to test.

ZFS was chosen for testing purposes because it stores hashes of all blocks of data. This makes it very simple to quickly detect any data corruption. If the hash doesn't match the hash generated from the data, you just detected corruption.

Other file systems don't store hashes and don't check for data corruption so they just trust the storage layer. It may take a while before you find out that data is corrupted.

I performed a storage vMotion of this secondary disk towards different datastores and then ran a 'zfs scrub' to track down any corruption. This worked better than expected: the scrub command would hang if the drive was corrupted by the storage vMotion. The test virtual machine required a reboot and a reformat of the secondary hard drive with ZFS as the previous file system, including data got corrupted.

After performing a storage vMotion on the drive in different directions, from different datastores to other datastores slowly a pattern emerged.

Storage vMotion corruption happened independent of the VMware ESXi host used.
a Storage vMotion never caused any issues when the disk was residing on our production storage array.
the corruption only happened when the virtual machine was stored on particular datastores on our DR storage array.

Now it got really 'interesting'. The thing is that our DR storage array has two separate storage controllers running in active-active mode. However, the LUNs are always owned by a particular controller. Although the other controller can take over from the controller who 'owns' the LUNs in case of a failure, the owner will process the I/O when everything is fine. Particular LUNs are thus handled by a particular controller.

So first I made a table where I listed the controllers and the LUNs it had ownership over, like this:

            Owner       
Controller      a               b
            LUN001          LUN002
            LUN003          LUN004
            LUN005          LUN006

Then I started to perform Storage vMotions of the ZFS disk from one LUN to the other. After performing several test, the pattern became quite obvious.

            LUN001  ->  LUN002  =   BAD
            LUN001  ->  LUN004  =   BAD
            LUN004  ->  LUN003  =   BAD
            LUN003  ->  LUN005  =   GOOD
            LUN005  ->  LUN001  =   GOOD

I continued to test some additional permutations but it became clear that only LUNs owned by controller b caused problems.

With the evidence in hand, I managed to convince our vendor support to replace storage controller b and that indeed resolved the problem. Data corruption due to a Storage vMotion never occurred after the controller was replaced.

There is no need to name/shame the vendor in this regard. The thing is that all equipment can fail and what can happen will happen. What really counts is: are you prepared?

RAID 5 is perfectly fine for home usage

2016-09-08T12:00:00+02:00

RAID 5 gets a lot of flak these days. You either run RAID 1, RAID 10 or you use RAID 6, but if you run RAID 5 you're told that you are a crazy person.

Using RAID 5 is portrayed as an unreasonable risk to the availability of your data. It is suggested that it is likely that you will lose your RAID array at some point.

That's an unfair representation of the actual risk that surrounds RAID 5. As I see it, the scare about RAID 5 is totally blown out of proportion.

I would argue that for small RAID arrays with a maximum of five to six drives, it's totally reasonable to use RAID 5 for your home NAS.

As far as I can tell, the campaign against RAID 5 mainly started with this article from zdnet.

As you know RAID 5 can tollerate a single drive failure. If a second drive dies and the first drive was not yet replaced or rebuild, you lose all contents of the array.

In the article the author argues that because drives become bigger but not more reliable, the risk of losing a second drive during a rebuild is so high that running RAID 5 is becoming risky.

You don't need a second drive failure for you to lose your data. A bad sector, also known as an Unrecoverable Read Error (URE), can also cause problems during a rebuild. Depending on the RAID implementation, you may lose some files or the entire array.

The author calculates and argues that the risk of such a bad sector or URE is so high with modern high-capacity drives, that this risk of a second drive failure during rebuild is almost unavoidable.

Most drives have a URE specification of 1 bit error in 12.5 TB of data (10^14). That number is used as an absolute, it's what drives do experience in our daily lives, but that's not true.

It's a worst-case number. You will see a read error in at-most 10^14 bits, but in practice drives are way more reliable.

I run ZFS on my 71 TB ZFS NAS and I scrub from time to time.

If that worst-case number were 'real', I would have caught some data errors by now. However, in line with my personal experience, ZFS hasn't corrected a single byte since the system came online a few years ago.

And I've performed so many scrubs that my system has read over a petabyte of data. No silent data corruption, no regular bad sectors.

It seems to me that all those risk aren't nearly as high as it seems.

I would argue that choosing RAID-5/Z in the right circumstances is reasonable. RAID-6 is clearly safer than RAID-5 as you can survive the loss of two drives instead of a single drive, but that doesn't mean that RAID-5 is unsafe.

If you are going to run a RAID 5 array, make sure you run a scrub or patrol read or whatever the name is that your RAID solution uses. A scrub is nothing more than attempt to try and read all data from disk.

Scrubbing allows detection of bad sectors in advance, so you can replace drives before they cause real problems (like failing during a rebuild).

If you keep the number of drives in a RAID-5 array low, maybe at most 5 or 6, I think for home users, who need to find a balance between cost and capacity, RAID-5 is an acceptable option.

And remember: if you care about your data, you need a backup anyway.

This topic was also discussed on reddit.

ZFS: resilver performance of various RAID schemas

2016-01-31T12:00:00+01:00

When building your own DIY home NAS, it is important that you simulate and test drive failures before you put your important data on it. It makes sense to know what to do in case a drive needs to be replaced. I also recommend putting a substantial amount of data on your NAS and see how long a resilver takes just so you know what to expect.

There are many reports of people building their own (ZFS-based) NAS who found out after a drive failure that resilvering would take days. If your chosen redundancy level for the VDEV would not protect against a second drive failure in the same VDEV (Mirror, RAID-Z) things may get scary. Especially because drives are quite bussy rebuilding data and the extra load on the remaining drives may increase the risk of a second failure.

The chosen RAID level for your VDEV, has an impact on the resilver performance. You may chose to accept lower resilver performance in exchange for additional redundancy (RAID-Z2, RAID-Z3).

I did wonder though how much those resilver times would differ between the various RAID levels. This is why I decided to run some tests to get some numbers.

Test hardware

I've used some test equipment running Debian Jessie + ZFS on Linux. The hardware is rather old and the CPU may have an impact on the results.

CPU : Intel(R) Core(TM)2 Duo CPU     E7400  @ 2.80GHz
RAM : 8 GB
HBA : HighPoint RocketRaid 2340 (each drive in a jbod)
Disk: Samsung Spinpoint F1 - 1 TB - 7200 RPM ( 12 x )

Test method

I've created a script that runs all tests automatically. This is how the script works:

Create pool + vdev(s).
Write data on pool ( XX % of pool capacity)
Replace arbitrary drive with another one.
Wait for resilver to complete.
Log resilver duration o csv file.

For each test, I fill the pool up to 25% with data before I measure resilver performance.

Caveats

The problem with the pool only being filled for 25% is that drives are fast at the start, but their performance deteriorates significantly as they fill up. This means that you cannot extrapolate the results and calculate resilver times for 50% or 75% pool usage, the numbers are likely worse than that.

I should run the test again with 50% usage to see if we can demonstrate this effect.

Beware that this test method is probably only suitable for DIY home NAS builds. Production file systems used within businesses may be way more fragmented and I've been told that this could slow down resilver times dramatically.

Test result (lower is better)

The results can only be used to demonstrate the relative resilver performance differences of the various RAID levels and disk counts per VDEV.

You should not expect the same performance results for your own NAS as the hardware probably differs significantly from my test setup.

Observations

I think the following observations can be made:

Mirrors resilver the fastest even if the number of drives involved is increased.
RAID-Z resilver performance is on-par with using mirrors when using 5 disks or less.
RAID-Zx resilver performance deteriorates as the number of drives in a VDEV increases.

I find it interesting that with smaller number of drives in a RAID-Z VDEV, rebuild performance is roughly on par with a mirror setup. If long rebuild times would scare you away from using RAID-Z, maybe it should not. There may be other reasons why you might shy away from RAID-Z, but this doesn't seem one of them.

RAID-Z2 is often very popular amongst home NAS builders, as it offers a very nice balance between capacity and redundancy. Wider RAID-Z2 VDEVs are more space efficient, but it is also clear that resilver operations take longer. Because RAID-Z2 can tollerate the loss of two drives, I think longer resilver times seem like a reasonable tradeoff.

It is clear that as you put more disks in a single RAID-Zx VDEV, rebuild times increase. This can be used as an argument to keep the number of drives per VDEV 'reasonable' or to switch to RAID-Z3.

25% vs 50% pool usage

To me, there's nothing special to see here. The resilver times are on average slightly worse than double the 25% resilver durations. As disks performance start to deteriorate as they fill up (inner tracks are shorter/slower) sequential performance drops. So this is why I would explain the results are slightly worse than perfect linear scaling.

Final words

I hope this benchmark is of interest to anyone and more importantly, you can run your own by using the aforementioned script. If you ever want to run your own benchmarks, expect the script to run for days. Leave a comment if you have questions or remarks about these test results or the way testing is done.

The 'hidden' cost of using ZFS for your home NAS

2016-01-02T12:00:00+01:00

Introduction

Update December 2023: In June, it was announced that iXsystems would sponsor implementing the VDEV expansion feature. A new pr has been created for this effort. The feature was merged into the code base, but may not be available to the general public before the end of 2024.

**Update August 2025: ** ZFS RAIDZ VDEV expansion has been released since OpenZFS release 2.3.0

Please note that an important limitation discussed in this article - the inability to expand VDEVs - has been lifted.

Many home NAS builders consider using ZFS for their file system. But there is a caveat with ZFS that people should be aware of.

Although ZFS is free software, implementing ZFS is not free. The key issue is that expanding capacity with ZFS is more expensive compared to legacy RAID solutions.

With ZFS, you either have to buy all storage you expect to need upfront, or you will be wasting a few hard drives on redundancy you don't need.

This fact is often overlooked, but it's very important to take it in consideration when planning a NAS build.

Other software RAID solutions like Linux MDADM lets you grow an existing RAID array with one disk at a time. This is also true for many hardware-based RAID solutions¹. This is ideal for home users because you can expand on a per-need basis.

ZFS does not allow this!

To understand why using ZFS may cost you extra money, we will dig a little bit into ZFS itself.

Quick recap of ZFS

The schema below illustrates the architecture of ZFS. There are a few things you should take away from it.

The main takeaway of this picture is that your ZFS pool and thus your file system is based on one or more VDEVs. And those VDEVs contain the actual hard drives.

Fault-tolerance or redundancy is addressed within a VDEV. A VDEV is either a mirror (RAID-1), RAIDZ (RAID-5) or RAIDZ2 (RAID-6)².

So it's important to understand that a ZFS pool itself is not fault-tolerant. If you lose a single VDEV within a pool, you lose the whole pool. You lose the pool, all data is lost.

You can't add hard drives to a VDEV

Now it's very important to understand that you cannot add hard drives to a VDEV.

This is the key limitation of ZFS as seen from the perspective of home NAS builders.

To expand the storage capacity of your pool, you need to add extra VDEVs. And because each VDEV needs to take care of its own redundancy, you also need to buy extra drives for parity.

I will quickly add that there is a way out: replace every hard drive in the VDEV, one by one, with a higher capacity hard drive. You will have to 'rebuild' or 'resilver' the VDEV after each replacement, but it will work, although it's a bit cumbersome and quite expensive.

So back to the topic at hand: what does this limitation mean in real life? I'll give an example.

Let's say you plan on building a small NAS with a capacity of four drives. Please don't create a three-drive RAID-Z thinking you can just add the fourth drive when you need to, because that's not possible.

In this example, you would be better off buying the fourth drive upfront and create a four-drive RAID-Z. This is an example where you are forced to buy the extra space you don't need yet upfront because expanding is otherwise not possible.

You could have expanded your pool with another VDEV consisting of a minimum of three drives (if you run RAID-Z) but the chassis has only room for one extra drive so that doesn't work.

Planning your ZFS Build with the VDEV limitation in mind

Many home NAS builders use RAID-6 (RAID-Z2) for their builds, because of the extra redundancy. This makes sense because a double drive failure is not something unheard of, especially during rebuilds where all drives are being taxed quite heavily for many hours.

I personally would recommend running RAID-Z2 over RAID-Z1 if you go over five to six drives and to spend the extra money on the additional hard drive it requires. Actually, With RAID-Z2 or RAID-6, I think it's perfectly reasonable to run a single VDEV at home with up to 12 drives³.

With RAID-Z2 however, the 'ZFS tax' is even more clearly visible. By having to add an additional VDEV, you will also lose two drives due to parity overhead.

Please note that the 'yellow' drives mark the parity/redundancy overhead. It does not mark where parity data lives (it's striped across all drives).

Let's illustrate the above picture with an example. Your NAS chassis can hold a maximum of twelve drives. You start out with six drives in a RAID-Z2. At some point you want to expand. The cheapest option is to expand with another RAID-Z2 consisting of four drives (minimum size of a RAID-Z2 VDEV).

With a cost of $150 per hard drive⁴, expanding the capacity of your pool will cost you $600 instead of $150 (single drive) and $300 dollar of the $600 (50%) is wasted on redundancy you don't really need.

Furthermore, you can no longer expand your pool, so the remaining two drive slots are 'wasted'⁵. You end up with a maximum of ten drives.

In this example, to make use of the drive capacity of your NAS chassis, you should expand with another six hard drives. That would cost you $900 and $300 of that $900 (33%) is wasted on redundancy. This is illustrated above.

Storage-wise it's more efficient to expand with six drives instead of four. But it will cost you another $300 to expand, paying for storage you may not immediately need.

But both options aren't that efficient. Because you end up using four drives for parity where two would - in my view - be sufficient.

So, if you want to get the most capacity out of that chassis, and the most space per dollar, your only option is to buy all twelve drives upfront and create a single RAID-Z2 consisting of twelve drives.

Buying all drives upfront is expensive and you may only benefit from that extra space years down the road.

Summary

So I hope this example clearly illustrates the issue at hand. With ZFS, you either need to buy all storage upfront or you will lose hard drives to redundancy you don't need, reducing the maximum storage capacity of your NAS.

You have to decide what your needs are. ZFS is an awesome file system that offers you way better data integrity protection than other file system + RAID solution combination.

But implementing ZFS has a certain 'cost'. You must decide if ZFS is worth it for you.

Update April 2023: It has been fairly quiet since the announcement of RAIDZ expansion. The Github PR about this feature is rather stale and people are wondering what the status is and what the plans are. Meanwhile, FreeBSD has announced In February 2023 that they suspect to integrate RAIDZ expansion by Q3.

Update June 2021| It seems that RAIDZ expansion is now being worked on. It will probably be available somewhere around August 2022.

I have written a blogpost about this new feature. The bad news is that adding drives to an existing vdev may accrue some overhead, but the good news is that this overhead can be recovered.

Update October 2017 | Please note that RAIDZ expansion is under development.

RAIDZ expansion (most requested ZFS feature ever?) is coming, courtesy of @freebsdfndation. Sneak preview at OpenZFS DevSummit!
— OpenZFS (@OpenZFS) October 19, 2017

Addressing some feedback

I found out that my article was discussed on a vodcast of BSDNOW.

This article also got some attention on hacker news.

To me, some of the feedback is not 'wrong' but feels rather disingenuous or not relevant for the intended audience of this article. I have provided the links so you can make up your own mind.

This article has a particular user group in mind so you really should think about how much their needs align with yours.

You are steering people away from ZFS

No I don't and this is not my intention. I run ZFS myself on two servers. I do feel that sometimes the downsides of ZFS are wiped under the rug and we should be very open and clear about them towards people seeking advice.

Use mirrors not RAID-Z(2/3)!

Doesn't make much sense to me for home NAS builders.

Using mirrors is wasting space

Advising people to use mirrors instead of RAID-Z(2/3) I do find a little bit disingenuous. Because you are throwing away 50% of your disk capacity. With RAIDZ you 'lose' 33% for three drives, 25% for four drives. If we look at RAIDZ2, we would 'lose' 33% for six drives, 25% for eight drives and only 20% for ten drives.

In the end, you are waisting multiple drives worth of storage capacity depending on the number of drives in your pool.

Adding mirrors with larger drives

As time goes by, larger disks become cheaper. So it could make sense to expand your pool with mirrors based on bigger drives than the original drives you started out on. The size of your pool would increase. However, it's still only 50% space efficient.

Random I/O performance is better

Using mirrors is running RAID 10. Yes you can expand your pool with two drives at a time, and you gain better random I/O performance. However, the large majority of home NAS builders don't care about random I/O performance. You just care if you can saturate gigabit and have one big pool of storage. In that case, you don't need the random IOPs.

If you run some VMs from your storage that require high storage performance, it's an entirely different matter. But I expect that most DIY NAS builders just want some storage to put a ton of data on and nothing more.

RAIDZ2 is more reliable than using mirrors

The redundancy of RAIDZ2 beats using mirrors. (If during a rebuild the surviving member of a mirror fails (the one disk in the pool that is taxed the most during rebuild) you lose your pool. With RAIDZ2 any second drive can fail and you are still OK.

There is only one 'upside' regarding mirrors that is discussed in the next section.

Mirror rebuild times are better

The only upside of using mirrors is that in the event a disk has failed and the new disk is being 'resilvered' it is reported that those rebuilds tend to be faster than if you use RAID-Z(2/3). I think this is no different from legacy RAID, where the main difference with ZFS is that ZFS only rebuilds actual data, not the entire disk.

ZFS rebuilds are faster

This is indeed a benefit of ZFS. The question is how relevant it is for you.

ZFS only rebuilds data. Legacy RAID just rebuilds every 'bit' on a drive. The latter takes longer than the former. So with legacy RAID, rebuild times depend on the size of a single drive, not on the number of drives in the array, no matter how much data you have stored on your array.

My old 18 TB server was based on a single twenty-drive RAID 6 using MDADM. It took 5 hours to rebuild a 1 TB drive. If you would have used 4 TB drives, it would have taken 20 hours if I'm allowed to extrapolate. With ZFS - if you would have been using only 50% of capacity - those rebuild times would have been half of this.

Personally with RAID6 or with RAIDZ2, rebuild times aren't that a big of a deal as you can lose a second drive and still be safe.

Just replace existing drives with bigger ones!

I did briefly touch this option in the article above. I will address it again. The problem with this approach is twofold. First, you can't expand storage capacity as you need it. You need to replace all existing drives with larger ones.

The procedure itself is also a bit cumbersome and time intensive. You need to replace each drive one by one. And every time, you need to 'resilver' your VDEV. Only when all drives have been replaced you will be able to grow the size of your pool.

If you are OK with this approach - and people have used it - it is a way to work around the 'ZFS-tax'.

Not using ZFS is putting your data at great risk!

The BSDNOW podcasts seems to agree with me that if you want true data safety, this 'ZFS-tax' is just the price you have to pay. Either you go with mirrors or you accept the extra parity redundancy.

It is not my goal to steer you away from ZFS. The above is true. ZFS offers something no other (stable) file system currently offers to home NAS builders. But at a cost.

The thing is that I find it perfectly reasonable for home NAS users to just buy a Synology, QNAP or some ready-made NAS from another quality brand. That's what the majority of people do and I think it's a reasonable option. I don't think you are taking crazy risks if you would do so.

If you do build your own home NAS, it's reasonable to accept the 'risk' of using Windows with storage spaces or hardware RAID. Or using Linux with MDADM or hardware RAID. I would say: ZFS is clearly technically the better option, but those 'legacy' options are not so bad that you are taking unreasonable risks with your data.

So using ZFS is the better option, it's up to you and your particular needs and circumstances to decide if using ZFS is worth it for you.

I believe hardware-based RAID is 100% dead, especially with SSDs but historically speaking hardare RAID allowed for flexible expansion. ↩
It can even use tripple parity (RAID-Z3) but I doubt many of you will ever need that. ↩
For my own 71 TB storage NAS I decided at that time to run with an eighteen-disk VDEV plus a six-disk VDEV. Not standard, but I decided that I accept the risk. ↩
Just an example for illustration purposes. ↩
Expanding with a VDEV consisting of a mirrored pair is technically possible but it breaks the RAID-Z2 redundancy. It doesn't make much sense to me. ↩

ZFS performance on HP Proliant Microserver Gen8 G1610T

2015-08-14T12:00:00+02:00

I think the HP Proliant Microserver Gen8 is a very interesting little box if you want to build your own ZFS-based NAS. The benchmarks I've performed seem to confirm this.

The Microserver Gen8 has nice features such as:

iLO (KVM over IP with dedicated network interface)
support for ECC memory
2 x Gigabit network ports
Free PCIe slot (half-height)
Small footprint
Fairly silent
good build quality

The Microserver Gen8 can be a better solution than the offerings of - for example - Synology or QNAP because you can create a more reliable system based on ECC-memory and ZFS.

Please note that the G1610T version of the Microserver Gen8 does not ship with a DVD/CD drive as depicted in the image above.

The Gen8 can be found fairly cheap on the European market at around 240 Euro including taxes and if you put in an extra 8 GB of memory on top of the 2 GB installed you have a total of 10 GB, which is more than enough to support ZFS.

The Gen8 has room for 4 x 3.5" hard drives so with todays large disk sizes you can pack quite a bit of storage inside this compact machine.

Netto storage capacity:

This table gives you a quick overview of the netto storage capacity you would get depending on the chosen drive size and redundancy.

Drive size	RAIDZ	RAIDZ2 or Mirror
3 TB	9 TB	6 TB
4 TB	12 TB	8 TB
6 TB	18 TB	12 TB
8 TB	24 TB	16 TB

Boot device

If you want to use all four drive slots for storage, you need to boot this machine from either the fifth internal SATA port, the internal USB 2.0 port or the microSD card slot.

The fifth SATA port is not bootable if you disable the on-board RAID controller and run in pure AHCI mode. This mode is probably the best mode for ZFS as there seems to be no RAID controller firmware active between the disks and ZFS. However, only the four 3.5" drive bays are bootable.

The fifth SATA port is bootable if you configure SATA to operate in Legacy mode. This is not recommended as you lose the benefits of AHCI such as hot-swap of disks and there are probably also performance penalties.

The fifth SATA port is also bootable if you enable the on-board RAID controller, but do not configure any RAID arrays with the drives you plan to use with ZFS (Thanks Mikko Rytilahti). You do need to put the boot drive in a RAID volume in order to be able to boot from the fifth SATA port.

The unconfigured drives will just be passed as AHCI devices to the OS and thus can be used in your ZFS array. The big question here is what happens if you encounter read errors or other drive problems that ZFS could handle, but would be a reason for the RAID controller to kick a drive off the SATA bus. I have no information on that.

I myself used an old 2.5" hard drive with a SATA-to-USB converter which I stuck in the case (use double-sided tape or velcro to mount it to the PSU). Booting from USB stick is also an option, although a regular 2.5" hard drive or SSD is probably more reliable (flash wear) and faster.

Boot performance

The Microserver Gen8 takes about 1 minute and 50 seconds just to pass the BIOS boot process and start booting the operating system (you will hear a beep).

Test method and equipment

I'm running Debian Jessie with the latest stable ZFS-on-Linux 0.6.4. Please note that reportedly FreeNAS also runs perfectly fine on this box.

I had to run my tests with the disk I had available:

root@debian:~# show disk -sm
-----------------------------------
| Dev | Model              | GB   |   
-----------------------------------
| sda | SAMSUNG HD103UJ    | 1000 |   
| sdb | ST2000DM001-1CH164 | 2000 |   
| sdc | ST2000DM001-1ER164 | 2000 |   
| sdd | SAMSUNG HM250HI    | 250  |   
| sde | ST2000DM001-1ER164 | 2000 |   
-----------------------------------

The 250 GB is a portable disk connected to the internal USB port. It is used as the OS boot device. The other disks, 1 x 1 TB and 3 x 2 TB are put together in a single RAIDZ pool, which results in 3 TB of storage.

Tests with 4-disk RAIDZ VDEV

root@debian:~# zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
testpool  48.8G  2.54T  48.8G  /testpool
root@debian:~# zpool status
  pool: testpool
 state: ONLINE
  scan: none requested
config:

    NAME                        STATE     READ WRITE CKSUM
    testpool                    ONLINE       0     0     0
      raidz1-0                  ONLINE       0     0     0
        wwn-0x50000f0008064806  ONLINE       0     0     0
        wwn-0x5000c5006518af8f  ONLINE       0     0     0
        wwn-0x5000c5007cebaf42  ONLINE       0     0     0
        wwn-0x5000c5007ceba5a5  ONLINE       0     0     0

errors: No known data errors

Because a NAS will face data transfers that are sequential in nature, I've done some tests with 'dd' to measure this performance.

Read performance:

root@debian:~# dd if=/testpool/test.bin of=/dev/null bs=1M
50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 162.429 s, 323 MB/s

Write performance:

root@debian:~# dd if=/dev/zero of=/testpool/test.bin bs=1M count=50000 conv=sync 50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 169.572 s, 309 MB/s

Test with 3-disk RAIDZ VDEV

After the previous test I wondered what would happen if I would exclude the older 1 TB disk and create a pool with just the 3 x 2 TB drives. This is the result:

Read performance:

root@debian:~# dd if=/testpool/test.bin of=/dev/null bs=1M conv=sync 50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 149.509 s, 351 MB/s

Write performance:

root@debian:~# dd if=/dev/zero of=/testpool/test.bin bs=1M count=50000 conv=sync 50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 144.832 s, 362 MB/s

The performance is clearly better even there's one disk less in the VDEV. I would have liked to test with an additional 2 TB drive what kind of performance would be achieved with four drives but I only have three.

The result does show that the pool is more than capable of sustaining gigabit network transfer speeds.

This is confirmed when performing the actual network file transfers. In the example below, I simulate a copy of a 50 GB test file from the Gen8 towards a test system using NFS. Tests are performed using the 3-disk pool.

NFS read performance:

root@nano:~# dd if=/mnt/server/test2.bin of=/dev/null bs=1M
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 443.085 s, 118 MB/s

NFS write performance:

root@nano:~# dd if=/dev/zero of=/mnt/server/test2.bin bs=1M count=50000 conv=sync 
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 453.233 s, 116 MB/s

I think these results are excellent. Tests with the 'cp' command give the same results.

I've also done some test with the SMB/CIFS protocol. I've used a second Linux box as a CIFS client to connect to the Gen8.

CIFS read performance:

root@nano:~# dd if=/mnt/test/test.bin of=/dev/null bs=1M
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 527.778 s, 99.3 MB/s

CIFS write performance:

root@nano:~# dd if=/dev/zero of=/mnt/test/test3.bin bs=1M count=50000 conv=sync
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 448.677 s, 117 MB/s

Hot-swap support

Although it's even printed on the hard drive caddies that hot-swap is not supported, it does seem to work perfectly fine if you run the SATA controller in AHCI mode.

Fifth SATA port for SSD SLOG/L2ARC?

If you buy a converter cable that converts a floppy power connector to a SATA power connector, you could install an SSD. This SSD can then be used as a dedicated SLOG device and/or L2ARC cache if you have a need for this.

RAIDZ, is that OK?

If you want maximum storage capacity with redundancy RAIDZ is the only option. RAID6 or two mirrored VDEVs is more reliable, but will reduce available storage space by a third.

The main risk of RAIDZ is a double-drive failure. As with larger drive sizes, a resilver of a VDEV will take quite some time. It could take more than a day before the pool is resilvered, during which you run without redundancy.

With the low number of drives in the VDEV the risk of a second drive failure may be low enough to be acceptable. That's up to you.

Noise levels

In the past, there have been reports about the Gen8 making tons of noise because the rear chasis fan spins at a high RPM if the RAID card is set to AHCI mode.

I myself have not encountered this problem. The machine is almost silent.

Power consumption

With drives spinning: 50-55 Watt. With drives standby: 30-35 Watt.

Conclusion

I think my benchmarks show that the Microserver Gen8 could be an interesting platform if you want to create your own ZFS-based NAS.

Please note that it is likely that since the Gen9 server platform is already out for some time, HP may release a Gen9 version of the microserver in the near future. However as of August 2015, there is no information on this yet and it is not clear if a successor is going to be released.

The sorry state of CoW file systems

2015-03-01T12:00:00+01:00

I'd like to argue that both ZFS and BTRFS both are incomplete file systems with their own drawbacks and that it may still be a long way off before we have something truly great.

Both ZFS and BTRFS are two heroic feats of engineering, created by people who are probably ten times more capable and smarter than me. There is no question about my appreciation for these file systems and what they accomplish.

Still, as an end-user, I would like to see some features that are often either missing or not complete. Make no mistake, I believe that both ZFS and BTRFS are probably the best file systems we have today. But they can be much better.

I want to start with a terse and quick overview on why both ZFS and BTRFS are such great file systems and why you should take some interest in them.

Then I'd like to discuss their individual drawbacks and explain my argument.

Why ZFS and BTRFS are so great

Both ZFS and BTRFS are great for two reasons:

They focus on preserving data integrity
They simplify storage management

Data integrity

ZFS and BTRFS implement two important techniques that help preserve data.

Data is checksummed and its checksum is verified to guard against bit rot due to broken hard drives or flaky storage controllers. If redundancy is available (RAID), errors can even be corrected.
Copy-on-Write (CoW), existing data is never overwritten, so any calamity like sudden power loss cannot cause existing data to be in an inconsistent state.

Simplified storage management

In the old days, we had MDADM or hardware RAID for redundancy. LVM for logical volume management and then on top of that, we have the file system of choice (EXT3/4, XFS, REISERFS, etc).

The main problem with this approach is that the layers are not aware of each other and this makes things very inefficient and more difficult to administer. Each layer needs it's own attention.

For example, if you simply want to expand storage capacity, you need to add drives to your RAID array and expand it. Then, you have to alert the LVM layer of the extra storage and as a last step, grow the file system.

Both ZFS and BTRFS make capacity expansion a simple one line command that addresses all three steps above.

Why are ZFS and BTRFS capable of doing this? Because they incorporate RAID, LVM and the file system in one single integrated solution. Each 'layer' is aware of the other, they are tightly integrated. Because of this integration, rebuilds after a drive faillure are often faster than with 'legacy RAID' solutions, because they only need to rebuild the actual data, not the entire drive.

And I'm not even talking about the joy of snapshots here.

The inflexibility of ZFS

The storage building block of ZFS is a VDEV. A VDEV is either a single disk (not so interesting) or some RAID scheme, such as mirroring, single-parity (RAIDZ), dual-parity (RAIDZ2) and even tripple-parity (RAIDZ3).

To me, a big downside to ZFS is the fact that you cannot expand a VDEV. Ok, the only way you can expand the VDEV is quite convoluted. You have to replace all of the existing drives, one by one, with bigger ones and rebuild the VDEV each time you replace one of the drives. Then, when all drives are of the higher capacity, you can expand your VDEV. This is quite impractical and time-consuming, if you ask me.

ZFS expects you just to add extra VDEVS. So if you start with a single 6-drive RAIDZ2 (RAID6), you are expected to add another 6-drive RAIDZ2 if you want to expand capacity.

What I would want to do is just to ad one or two more drives and grow the VDEV, as is possible with many hardware RAID solutions and with "MDADM --grow" for ages.

Why do I prefer this over adding VDEVS? Because it's quite evident that this is way more economical. If I can just expand my RAIDZ2 from 6 drives to 12 drives, I would only sacrifice two drives for parity. If I add two VDEVS each of them RAIDZ2, I sacrifice four drives (16% vs 33% capacity loss).

I can imagine that in the enterprise world, this is just not that big of a deal, a bunch of drives are a rounding error on the total budget and availability and performance are more important. Still, I'd like to have this option.

Either you are forced to buy and implement the storage you may expect to need in the future, or you must add it later on, wasting drives on parity you would otherwise not have done.

Maybe my wish for a zpool grow option is more geared to hobbyist or home usage of ZFS and ZFS was always focussed on enterprise needs, not the needs of hobbyists. So I'm aware of the context here.

I'm not done with ZFS however, because the way ZFS works, there is another great inflexibility. If you don't put the 'right' number of drives in a VDEV, you may lose significant portions of storage, which is a side-effect of how ZFS works.

The following ZFS pool configurations are optimal for modern 4K sector harddrives:
RAID-Z: 3, 5, 9, 17, 33 drives
RAID-Z2: 4, 6, 10, 18, 34 drives
RAID-Z3: 5, 7, 11, 19, 35 drives

I've seen first-hand with my 71 TiB NAS that if you don't use the optimal number of drives in a VDEV, you may lose whole drives worth of netto storage capacity. In that regard, my 24-drive chassis is very suboptimal.

The sad state of RAID on BTRFS

BTRFS has none of the downsides of ZFS as described in the previous section as far as I'm aware of. It has plenty of its own, though. First of all: BTRFS is still not stable, especially the RAID 5/6 part is unstable.

The RAID 5 and RAID 6 implementation are so new, the ink they were written with is still wet (February 8th 2015). Not something you want to trust your important data to I suppose.

I did setup a test environment to play a bit with this new Linux kernel (3.19.0) and BTRFS to see how it works and although it is not production-ready yet, I really like what I see.

With BTRFS you can just add or remove drives to a RAID6 array as you see fit. Add two? Subtract 3? Whatever, the only thing you have to wait for is BTRFS rebalancing the data over either the new or remaining drives.

This is friggin' awesome.

If you want to remove a drive, just wait for BTRFS to copy the data from that drive to the other remaining drives and you can remove it. You want to expand storage? Just add the drives to your storage pool and have BTRFS rebalance the data (which may take a while, but it works).

But I'm still a bit sad. Because BTRFS does not support anything beyond RAID6. No multiple RAID6 (RAID60) arrays or tripple-parity, as ZFS supports for ages. As with my 24-drive file server, putting 24 drives in a single RAID6, starts to feel like I'm asking for trouble. Tripple-parity or RAID 60 would probably be more reasonable. But no luck with BTRFS.

However, what really frustrates me is this article by Ronny Egner. The author of snapraid, Andrea Mazzoleni, has written a functional patch for BTRFS that implements not only tripple-parity RAID, but even up to six parity disks for a volume.

The maddening thing is that the BTRFS maintainers are not planning to include this patch into the BTRFS code base. Please read Ronny's blog. The people working on BTRFS are working for enterprises who want enterprise features. They don't care about tripple-parity or features like that because they have access to something presumably better: distributed file systems, which may do away with the need for larger disk arrays and thus tripple-parity.

BTRFS is in development for a very long time and only recently has RAID 5/6 support been introduced. The risk of the write-hole, something addressed by ZFS ages ago, is still an open issue. Considering all of this, BTRFS is still a very long way off, of being the file system of choice for larger storage arrays.

BTRFS seems to be way more flexible in terms of storage expansion or shrinking, but it slow pace of development makes it still unusable for anything serious for at least the next year I guess.

Conclusion

BTRFS addresses all the inflexibilities of ZFS but it's immaturity and lack of more advanced RAID schemes makes it unusable for larger storage solutions. This is so sad because by design it seems to be the better, way more flexible option as compared to ZFS.

I do understand the view of the BTRFS developers. With the enterprise data sets, at scale, it's better to use distributed file systems to handle storage and redundancy, than on the smaller system scale. But this kind of environment is not reachable for many.

So at the moment, compared to BTRFS, ZFS is still the better option for people who want to setup large, reliable storage arrays.

Configuring SCST iSCSI target on Debian Linux (Wheezy)

2015-02-01T12:00:00+01:00

My goal is to export ZFS zvol volumes through iSCSI to other machines. The platform I'm using is Debian Wheezy.

There are three iSCSI target solutions available for Linux:

I've briefly played with LIO but the targetcli tool is interactive only. If you want to automate and use scripts, you need to learn the Python API. I wonder what's wrong with a plain old text-based configuration file.

iscsitarget or IET is broken on Debian Wheezy. If you just 'apt-get install iscsitarget', the iSCSI service will just crash as soon as you connect to it. This has been the case for years. I wonder why they don't just drop this package. It is true that you can manually download the "latest" version of IET, but don't bother, it seems abandoned. The latest release stems from 2010.

It seems that SCST is at least maintained and uses plain old text-based configuration files. So it has that going for it, which is nice. SCST does not require kernel patches to run. But particularly a patch regarding "CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION" is said to improve performance.

To use full power of TCP zero-copy transmit functions, especially
dealing with user space supplied via scst_user module memory, iSCSI-SCST
needs to be notified when Linux networking finished data transmission.
For that you should enable CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION
kernel config option. This is highly recommended, but not required.
Basically, iSCSI-SCST works fine with an unpatched Linux kernel with the
same or better speed as other open source iSCSI targets, including IET,
but if you want even better performance you have to patch and rebuild
the kernel.

So in general, patching your kernel is not always required, but an example will be given anyway.

Getting the source

cd /usr/src

We need the following files:

wget http://heanet.dl.sourceforge.net/project/scst/scst/scst-3.0.0.tar.bz2
wget http://heanet.dl.sourceforge.net/project/scst/iscsi-scst/iscsi-scst-3.0.0.tar.bz2
wget http://heanet.dl.sourceforge.net/project/scst/scstadmin/scstadmin-3.0.0.tar.bz2

We extract them with:

tar xjf scst-3.0.0.tar.bz2
tar xjf iscsi-scst-3.0.0.tar.bz2
tar xjf scstadmin-3.0.0.tar.bz2

Patching the kernel

You can skip this part if you don't feel like you need or want to patch your kernel.

apt-get install linux-source kernel-package

We need to extract the kernel source:

cd /usr/src
tar xjf linux-source-3.2.tar.bz2
cd linux-source-3.2

Now we first copy the kernel configuration from the current system:

cp /boot/config-3.2.0-4-amd64 .config

We patch the kernel with two patches:

patch -p1 < /usr/src/scst-3.0.0/kernel/scst_exec_req_fifo-3.2.patch
patch -p1 < /usr/src/iscsi-scst-3.0.0/kernel/patches/put_page_callback-3.2.57.patch

It seems that for many different kernel versions, separate patches can be found in the above paths. If you follow these steps at a later date, please check the version numbers.

The patches are based on stock kernels from kernel.org. I've applied the patches against the Debian-patched kernel and faced no problems, but your milage may vary.

Let's build the kernel (will take a while):

yes | make-kpkg -j $(nproc) --initrd --revision=1.0.custom.scst kernel_image

The 'yes' is piped into the make-kpkg command to answer some questions with 'yes' during compilation. You could also add the appropriate value in the .config file.

The end-result of this command is a kernel package in .deb format in /usr/src. Install it like this:

dpkg -i /usr/src/<custom kernel image>.deb

Now reboot into the new kernel:

reboot

Compiling SCST, ISCS-SCST and SCSTADMIN

cd /usr/src/scst-3.0.0
make install

cd /usr/src/iscsi-scst-3.0.0
make install

cd /usr/src/scstadmin-3.0.0
make install

Make SCST start at boot

On Debian Jessie:

systemctl enable scst.service

Configure SCST

Copy the example configuration file to /etc:

cp /usr/src/iscsi-scst-3.0.0/etc/scst.conf /etc

Edit /etc/scst.conf to your liking. This is an example:

HANDLER vdisk_fileio {
        DEVICE disk01 {
                filename /dev/sdb
                nv_cache 1
        }
}

TARGET_DRIVER iscsi {
        enabled 1

        TARGET iqn.2015-10.net.vlnb:tgt {
                IncomingUser "someuser somepasswordof12+chars"
                HeaderDigest   "CRC32C,None"
                DataDigest   "CRC32C,None"
                LUN 0 disk01

                enabled 1
        }
}

Please note that the password must be at least 12 characters.

After this, you can start the SCST module and connect your initiator to the appropriate LUN.

/etc/init.d/scst start

Closing words

It turned out that setting up SCST and compiling a kernel wasn't that much of a hassle. The main issue with patching kernels is that you have to repeat the procedure every time a new kernel version is released. And there is always a risk that a new kernel version breaks the SCST patches.

However, the whole process can be easily automated and thus run as a test in a virtual environment.

71 TiB DIY NAS based on ZFS on Linux

2014-08-02T21:08:00+02:00

This is my new 71 TiB DIY NAS. This server is the successor to my six year old, twenty drive 18 TB NAS (17 TiB). With a storage capacity four times higher than the original and an incredible read (2.5 GB/s)/write (1.9 GB/s) performance, it's a worthy successor.

Purpose

The purpose of this machine is to store backups and media, primarily video.

The specs

Part	Description
Case	Ri-vier RV-4324-01A
Processor	Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
RAM	16 GB ECC
Motherboard	Supermicro X9SCM-F
LAN	Intel Gigabit
Storage Connectivity	~~InfiniBand MHGA28-XTC~~ 2023: Mellanox ConnectX-3 Pro 10Gbit Ethernet (X312B-XCCT)
PSU	Seasonic Platinum 860
Controller	3 x IBM M1015
Disk	24 x HGST HDS724040ALE640 4 TB (7200RPM)
SSD	2 x Crucial M500 120GB in RAID 1 for boot drives
Arrays	Boot: 2 x 120 GB RAID 1 and storage: 18 disk RAIDZ2+ 6 disk RAIDZ2
Brutto storage	86 TiB (96 TB)
Netto storage	71 TiB (78 TB)
OS	~~Linux Debian Wheezy~~2023: Ubuntu 22.04
Filesystem	ZFS
Rebuild time	Depends on amount of data (rate is 4 TB/Hour)
UPS	~~Back-UPS RS 1200 LCD using Apcupsd~~ None
Power usage	about 200 Watt idle

CPU

The Intel Xeon E3-1230 V2 is not the latest generation but one of the cheapest Xeons you can buy and it supports ECC memory. It's a quad-core processor with hyper-threading.

Here you can see how it performs compared to other processors.

Memory

The system has 16 GB ECC RAM. Memory is relatively cheap these days but I don't have any reason to upgrade to 32 GB. I think that 8 GB would have been fine with this system.

Motherboard

The server is build around the SuperMicro X95SCM-F motherboard.

This is a server-grade motherboard and comes with typical features you might expect, like ECC memory support and out-of-band management (IPMI).

This motherboard has four PCIe slots (2 x 8x and 2 x 4x) in an 8x physical slot. My build requires four PCIe 4x+ slots and there aren't (m)any other server boards at this price point that support four PCIe slots in a 8x sized slot.

The chassis

The chassis has six rows of four drive bays that are kept cool by three 120mm fans in a fan wall behind the drive bays. At the rear of the case, there are two 'powerful' 80mm fans that remove the heat from the case, together with the PSU.

The chassis has six SAS backplanes that connect four drives each. The backplanes have dual molex power connectors, so you can put redundant power supplies into the chassis. Redundant power supplies are more expensive and due to their size, often have smaller, thus noisier fans. As this is a home build, I opted for just a single regular PSU.

When facing the front, there is a place at the left side of the chassis to mount a single 3.5 inch or two 2.5 inch drives next to each other as boot drives. I've mounted two SSDs (RAID1).

This particular chassis version has support for SPGIO, which should help identifying which drive has failed. The IBM 1015 cards I use do support SGPIO. Through the LSI megaraid CLI I have verified that SGPIO works, as you can use this tool as a drive locator. I'm not entirely sure how well SGPIO works with ZFS.

Power supply

I was using a Corsair 860i before, but it was unstable and died on me.

The Seasonic Platinum 860 may seem like overkill for this system. However, I'm not using staggered spinup for the 24 drives. So the drives all spinup at once and this results in a peak power usage of 600+ watts.

The PSU has a silent mode that causes the fan only to spin if the load reaches a certain threshold. Since the PSU fan also helps removing warm air from the chassis, I've disabled this feature, so the fan is spinning at all times.

Drive management

I've written a tool called lsidrivemap that displays each drive in an ASCII table that reflects the physical layout of the chassis.

The data is based on the output of the LSI 'megacli' tool for my IBM 1015 controllers.

root@nano:~# lsidrivemap disk

| sdr | sds | sdt | sdq |
| sdu | sdv | sdx | sdw |
| sdi | sdl | sdp | sdm |
| sdj | sdk | sdn | sdo |
| sdb | sdc | sde | sdf |
| sda | sdd | sdh | sdg |

This layout is 'hardcoded' for my chassis but the Python script can be easily tailored for your own server, if you're interested.

It can also show the temperature of the disk drives in the same table:

root@nano:~# lsidrivemap temp

| 36 | 39 | 40 | 38 |
| 36 | 36 | 37 | 36 |
| 35 | 38 | 36 | 36 |
| 35 | 37 | 36 | 35 |
| 35 | 36 | 36 | 35 |
| 34 | 35 | 36 | 35 |

These temperatures show that the top drives run a bit hotter than the other drives. An unverified explanation could be that the three 120mm fans are not in the center of the fan wall. They are skewed to the bottom of the wall, so they may favor the lower drive bays.

Filesystem (ZFS)

I'm using ZFS as the file system for the storage array. At this moment, there is no other file system that has the same features and stability as ZFS. BTRFS is not even finished.

The number one design goal of ZFS was assuring data integrity. ZFS checksums all data and if you use RAIDZ or a mirror, it can even repair data. Even if it can't repair a file, it can at least tell you which files are corrupt.

ZFS is not primarily focussed on performance, but to get the best performance possible, it makes heavy usage of RAM to cache both reads and writes. This is why ECC memory is so important.

ZFS also implements RAID. So there is no need to use MDADM. My previous file server was running a single RAID 6 of 20 x 1TB drives. With this new system I've created a single pool with two RAIDZ2 VDEVs.

Capacity

Vendors still advertise the capacity of their hard drives in TB whereas the operating system works with TiB. So the 4 TB drives I use are in fact 3.64 TiB.

The total raw storage capacity of the system is about 86 TiB.

My zpool is the 'appropriate' number of disks (2^n + parity^) in the VDEVs. So I have one 18 disk RAIDZ2 VDEV (2^4+2) and one 6 disk RAIDZ2 VDEV (2^2+2^) for a total of 24 drives.

Different VDEV sizes in a single pool are often not recommended, but ZFS is very smart and cool: it load-balances the data across the VDEVs based on the size of the VDEV. I could verify this with zpool iostat -v 5 and witness this in real-time. The small VDEV got just a fraction of the data compared to the large VDEV.

This choice leaves me with less capacity (71 TiB vs. 74 TiB for RAIDZ3) and also has a bit more risk to it, with the eighteen-disk RAIDZ2 VDEV. Regarding this latter risk, I've been running a twenty-disk MDADM RAID6 for the last 6 years and haven't seen any issues. That does not tell everything, but I'm comfortable with this risk.

Originalyl I was planning on using RAIDZ3 and by using ashift=9 (512 byte sectors) I would recuperate most of the space lost to the non-optimal number of drives in the VDEV. So why did I change my mind? Because the performance of my ashift=9 pool on my 4K drives deteriorated so much that a resilver of a failed drive would take ages.

Storage controllers

The IBM 1015 HBA's are reasonably priced and buying three of them, is often cheaper than buying just one HBA with a SAS expander. However, it may be cheaper to search for an HP SAS expander and use it with just one M1015 and save a PCIe slot.

I have not flashed the controllers to 'IT mode', as most people do. They worked out-of-the-box as HBAs and although it may take a little bit longer to boot the system, I decided not to go through the hassle.

The main risk here is how the controller handles a drive if a sector is not properly read. It may disable the drive entirely, which is not necessary for ZFS and often not preferred.

Storage performance

With twenty-four drives in a chassis, it's interesting to see what kind of performance you can get from the system.

Let's start with a twenty-four drive RAID 0. The drives I use have a sustained read/write speed of 160 MB/s so it should be possible to reach 3840 MB/s or 3.8 GB/s. That would be amazing.

This is the performance of a RAID 0 (MDADM) of all twenty-four drives.

root@nano:/storage# dd if=/dev/zero of=test.bin bs=1M count=1000000
1048576000000 bytes (1.0 TB) copied, 397.325 s, 2.6 GB/s

root@nano:/storage# dd if=test.bin of=/dev/null bs=1M
1048576000000 bytes (1.0 TB) copied, 276.869 s, 3.8 GB/s

Dead on, you would say, but if you divide 1 TB with 276 seconds, it's more like 3.6 GB/s. I would say that's still quite close.

This machine will be used as a file server and a bit of redundancy would be nice. So what happens if we run the same benchmark on a RAID6 of all drives?

root@nano:/storage# dd if=/dev/zero of=test.bin bs=1M count=100000
104857600000 bytes (105 GB) copied, 66.3935 s, 1.6 GB/s

root@nano:/storage# dd if=test.bin of=/dev/null bs=1M
104857600000 bytes (105 GB) copied, 38.256 s, 2.7 GB/s

I'm quite pleased with these results, especially for a RAID6. However, RAID6 with twenty-four drives feels a bit risky. So since there is no support for a three-parity disk RAID in MDADM/Linux, I use ZFS.

Sacrificing performance, I decided - as I mentioned earlier - to use ashift=9 on those 4K sector drives, because I gained about 5 TiB of storage in exchange.

This is the performance of twenty-four drives in a RAIDZ3 VDEV with ashift=9.

root@nano:/storage# dd if=/dev/zero of=ashift9.bin bs=1M count=100000 
104857600000 bytes (105 GB) copied, 97.4231 s, 1.1 GB/s

root@nano:/storage# dd if=ashift9.bin of=/dev/null bs=1M
104857600000 bytes (105 GB) copied, 42.3805 s, 2.5 GB/s

Compared to the other results, write performance is way down, although not too bad.

This is the write performance of the 18 disk RAIDZ2 + 6 disk RAIDZ2 zpool (ashift=12):

root@nano:/storage# dd if=/dev/zero of=test.bin bs=1M count=1000000 
1048576000000 bytes (1.0 TB) copied, 543.072 s, 1.9 GB/s

root@nano:/storage# dd if=test.bin of=/dev/null bs=1M 
1048576000000 bytes (1.0 TB) copied, 400.539 s, 2.6 GB/s

As you may notice, the write performance is better than the ashift=9 or ashift=12 RAIDZ3 VDEV.

In the end I chose to use the 18 disk RAIDZ2 + 6 disk RAIDZ2 setup because of the better performance and to adhere to the standards of ZFS.

I have not benchmarked random I/O performance as it is not relevant for this system. And with ZFS, the random I/O performance of a VDEV is that of a single drive.

Boot drives

I'm using two Crucial M500 120GB SSD drives. They are configured in a RAID1 (MDADM) and I've installed Debian Wheezy on top of them.

At first, I was planning on using a part of the capacity for caching purposes in combination with ZFS. However, there's no real need to do so. In hindsight I could also have used to very cheap 2.5" hard drives (simmilar to my older NAS), which would have cost less than a single M500.

Update 2014-09-01: I actually reinstalled Debian and kept about 50% free space on both M500s and put this space in a partition. These partitions have been provided to the ZFS pool as L2ARC cache. I did this because I could, but on the other hand, I wonder if I'm only really just wearing out my SSDs faster.

Update 2015-10-04: I saw no reason why I would wear out my SSDs as a L2ARC so I removed them from my pool. There is absolutely no benefit in my case.

Networking (updated 2017-03-25)

Current: I have installed a Mellanox MHGA28-XTC InfiniBand card. I'm using InfiniBand over IP so the InfiniBand card is effectively a faster network card. I have a point-to-point connection with another server, I do not have an InfiniBand switch.

I get about 6.5 Gbit from this card, which is not even near the theoretical performance limit. However, this translate into a constant 750 MB/s file transfer speed over NFS, which is amazing.

Using Linux bonding and the quad-port Ethernet adapter, I only got 400 MB/s and transfer speeds were fluctuating a lot.

Original: Maybe I will invest in 10Gbit ethernet or InfiniBand hardware in the future, but for now I settled on a quad-port gigabit adapter. With Linux bonding, I can still get 450+ MB/s data transfers, which is sufficient for my needs.

The quad-port card is in addition to the two on-board gigabit network cards. I use one of the on-board ports for client access. The four ports on the quad-port card are all in different VLANs and not accessible for client devices.

The storage will be accessible over NFS and SMB. Clients will access storage over one of the on-board Gigabit LAN interfaces.

Keeping things cool and quiet

It's important to keep the drive temperature at acceptable levels and with 24 drives packet together, there is an increased risk of overheating.

The chassis is well-equipped to keep the drives cool with three 120mm fans and two strong 80mm fans, all supporting PWM (pulse-width modulation).

The problem is that by default, the BIOS runs the fans at a too low speed to keep the drives at a reasonable temperature. I'd like to keep the hottest drive at about forty degrees Celsius. But I also want to keep the noise at reasonable levels.

I wrote a python script called storagefancontrol that automatically adjusts the fan speed based on the temperature of the hottest drive.

UPS

I'm running a HP N40L micro server as my firewall/router. My APC Back-UPS RS 1200 LCD (720 Watt) is connected with USB to this machine. I'm using apcupsd to monitor the UPS and shutdown servers if the battery runs low.

All servers, including my new build, run apcupsd in network mode and talk to the N40L to learn if power is still OK.

Keeping power consumption reasonable

So these are the power usage numbers.

 96 Watt with disks in spin down.
176 Watt with disks spinning but idle.
253 Watt with disks writing.

Edit 2015-10-04: I do have an unresolved issue where the drives keep spinning up even with all services on the box killed, including Cron. So it's configured so that the drives are always spinning. /end edit

But the most important stat is that it's using 0 Watt if powered off. The system will be turned on only when necessary through wake-on-lan. It will be powered off most of the time, like when I'm at work or sleeping.

Cost

The system has cost me about €6000. All costs below are in Euro and include taxes (21%).

Description	Product	Price	Amount	Total
Chassis	Ri-vier 4U 24bay storage chassis RV-4324-01A	554	1	554
CPU	Intel Xeon E3-1230V2	197	1	197
Mobo	SuperMicro X9SCM-F	157	1	157
RAM	Kingston DDR3 ECC KVR1333D3E9SK2/16G	152	1	152
PSU	AX860i 80Plus Platinum	175	1	175
Network Card	NC364T PCI Express Quad Port Gigabit	145	1	145
HBA Controller	IBM SERVERAID M1015	118	3	354
SSDs	Crucial M500 120GB	62	2	124
Fan	Zalman FB123 Casefan Bracket + 92mm Fan	7	1	7
Hard Drive	Hitachi 3.5 4TB 7200RPM (0S03356)	166	24	3984
SAS Cables		25	6	150
Fan cables		6	1	6
Sata-to-Molex		3,5	1	3,5
Molex splitter		3	1	3
				6012

Closing words

If you have any questions or remarks about what could have been done differently feel free to leave a comment, I appreciate it.

ZFS: Performance and capacity impact of ashift=9 on 4K sector drives

2014-07-31T00:00:00+02:00

Update 2014-8-23: I was testing with ashift for my new NAS. The ashift=9 write performance deteriorated from 1.1 GB/s to 830 MB/s with just 16 TB of data on the pool. Also I noticed that resilvering was very slow. This is why I decided to abandon my 24 drive RAIDZ3 configuration.

I'm aware that drives are faster at the outside of the platter and slower on the inside, but the performance deteriorated so dramatically that I did not wanted to continue further.

My final setup will be a RAIDZ2 18 drive VDEV + RAIDZ2 6 drive VDEV which will give me 'only' 71 TiB of storage, but read performance is 2.6 GB/s and write performance is excellent at 1.9 GB/s. I've written about 40+ TiB to the array and after those 40 TiB, write performance was about 1.7 GB/s, so still very good and what I would expect as drives fill up.

So actually, based on these results, I have learned not to deviate from the ZFS best practices too much. Use ashift=12 and put drives in VDEVS that adhere to the 2^n+parity rule.

The uneven VDEVs (18 disk vs. 6 disks) are not according to best practice but ZFS is smart: it distributes data across the VDEVs based on their size. So they fill up equally.

Choosing between ashift=9 and ashift=12 for 4K sector drives is not always a clear cut case. You have to choose between raw performance or storage capacity.

My testplatform is Debian Wheezy with ZFS on Linux. I'm using a system with 24 x 4 TB drives in a RAIDZ3. The drives have a native sector size of 4K, and the array is formatted with ashift=12.

First we create the array like this:

zpool create storage -o ashift=12 raidz3 /dev/sd[abcdefghijklmnopqrstuvwx]

Note: NEVER use /dev/sd? drive names for an array, this is just for testing, always use /dev/disk/by-id/ names.

Then we run a simple sequential transfer benchmark with dd:

root@nano:/storage# dd if=/dev/zero of=ashift12.bin bs=1M count=100000 
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 66.4922 s, 1.6 GB/s
root@nano:/storage# dd if=ashift12.bin of=/dev/null bs=1M
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 42.0371 s, 2.5 GB/s

This is quite impressive. With these speeds, you can saturate 10Gbe ethernet. But how much storage space do we get?

df -h:

Filesystem                            Size  Used Avail Use% Mounted on
storage                                69T  512K   69T   1% /storage

zfs list:

NAME      USED  AVAIL  REFER  MOUNTPOINT
storage  1.66M  68.4T   435K  /storage

Only 68.4 TiB of storage? That's not good. There should be 24 drives minus 3 for parity is 21 x 3.6 TiB = 75 TiB of storage.

So the performance is great, but somehow, we lost about 6 TiB of storage, more than a whole drive.

So what happens if you create the same array with ashift=9?

zpool create storage -o ashift=9 raidz3 /dev/sd[abcdefghijklmnopqrstuvwx]

These are the benchmarks:

root@nano:/storage# dd if=/dev/zero of=ashift9.bin bs=1M count=100000 
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 97.4231 s, 1.1 GB/s
root@nano:/storage# dd if=ashift9.bin of=/dev/null bs=1M
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 42.3805 s, 2.5 GB/s

So we lose about a third of our write performance, but the read performance is not affected, probably by read-ahead caching but I'm not sure.

With ashift=9, we do lose some write performance, but we can still saturate 10Gbe.

Now look what happens to the available storage capacity:

df -h:

Filesystem                         Size  Used Avail Use% Mounted on
storage                             74T   98G   74T   1% /storage

zfs list:

NAME      USED  AVAIL  REFER  MOUNTPOINT
storage   271K  73.9T  89.8K  /storage

Now we have a capacity of 74 TiB, so we just gained 5 TiB with ashift=9 over ashift=12, at the cost of some write performance.

So if you really care about sequential write performance, ashift=12 is the better option. If storage capacity is more important, ashift=9 seems to be the best solution for 4K drives.

The performance of ashift=9 on 4K drives is always described as 'horrible' but I think it's best to run your own benchmarks and decide for yourself.

Caveat: I'm quite sure about the benchmark performance. I'm not 100% sure how reliable the reported free space is according to df -h or zfs list.

Edit: I have added a bit of my own opinion on the results.

Achieving 2.3 GB/s with 16 x 4 TB drives

2014-07-12T12:00:00+02:00

I'm in the process of building a new storage server to replace my 18 TB NAS.

The server is almost finished, it's now down to adding disk drives. I'm using the HGST 4 TB 7200 RPM drive for this build (SKU 0S03356) (review).

I have not bought all drives at once, but slowly adding them in smaller quantities. I just don't want to feel too much pain in my wallet at once I guess.

According to my own tests, this drive has a read/write throughput of 160 MB/s, which is in tune with it's specification.

So the theoretical performance of a RAID 0 with 16 drives x 160 MB/s = 2560 MB/s. That's over 2.5 gigabytes per second.

This is the actual real-life performance I was able to achieve.

root@nano:/storage# dd if=pureawesomeness.dd of=/dev/null bs=1M
1000000+0 records in
1000000+0 records out
1048576000000 bytes (1.0 TB) copied, 453.155 s, 2.3 GB/s

Personally, 2.3 GB/s is not too shabby in my opinion. Please note that I used a test file of one terabyte, so the 16 GB of RAM my server has, doesn't skew the result.

This result is very nice, but in practice almost useless. I can saturate dual 10 Gbit NICs with this system, but I don't have that kind of equipment or any other device that could handle such performance.

But I think it's amazing anyway.

I'm quite curious how the final 24 drive array will perform in a RAID 0.

How to resolve extreme memory usage on Windows 2008 R2-based file servers

2014-06-15T12:00:00+02:00

I'm responsible for a file server with about 5 terrabytes of data. The file server is based on Windows 2008 R2. I've noticed extreme memory usage on the server. After a reboot, it slowly builds up until almost all RAM memory is consumed.

So I googled around and found this post and it turned out I had the same exact issue.

I've confirmed with the tool 'RAMmap' that NTFS metadata is the issue. Microsoft also created a blog post about this.

The author of the first article resolved the issue by adding more RAM memory. But with 16 GB already assigned, I was not to happy to add more memory to the virtual file server, eating away on the RAM resources of our virtualisation platform.

I could never find a root cause of the issue. In that case, you need to obtain the 'Microsoft Windows Dynamic Cache Service'. This application allows you to configure how large the medata caching may grow.

Please note that this services is not a next-next-finish installation. Follow the included Word document with instructions carefully and configure a sane memory setting for your server. I limited the cache to half the RAM available to the server and this works out well.

My experiences with DFS replication on Windows 2008 R2

2014-06-15T12:00:00+02:00

If you are considering implementing DFS replication, consider using Windows 2012 R2 because DFS replication has been massively improved. It supports larger data sets and performance has dramatically been improved over Windows 2008 R2.

I've implemented DFS replication to keep two file servers synchronised. Click here if or there you want to learn more about DFS itself.

With DFS, I wanted to create a high-available file server service, based on two file servers, each with their own physical storage. DFS replication makes sure that both file servers are kept in sync.

If you setup DFS, you need to copy all the data from the original server to the secondary server. This is called seeding and I've used robocopy as recommended by Microsoft in the linked article.

Seeding is not mandatory. You can just start with an empty folder on the secondary server and just have DFS replicate all files. I've experienced myself that DFS replication can be extremely slow on Windows 2008 R2.

Once all files are seeded and DFS is configured, the initial replication can still takes days. Replication times are based on:

the number of files
the size of the data
the performance of the disk subsystems of both source and destination

Note: windows 2012 R2 improves DFS replication dramatically, only more reason to upgrade your file servers to 2012 R2 or higher.

If you seed the files, DFS will not transfer files if they are identical, thus this saves bandwidth and time. DFS checks if files differ based on their hash. So even if you seed all data, the initial replication can take a while.

On our virtualised platform, the initial replication of 2.5 GB of data consisting of about five million files took about a full week. To me, that is not a very desirable outcome, but once the initial replication is done, there is no performance issue and all changes are nearly instantly replicated to the secondary server.

For the particular configuration I've setup, the performance storage subsystem could contribute to the slow initial replication.

To speed up the replication process, it's important that you install the latest version of robocopy for Windows 2008 R2 on both systems. There is a bug in older versions of robocopy that do not properly set permissions on files. This results in file hash mismatches, causing DFS to replicate all files, nullifying the benefit of seeding.

Hotfixes for Windows 2008 R2: Hotfixes for Windows 2012 R2:

To verify if a file on both servers has identical hashes, follow these instructions

If you've checked a few files and assured that the hashes are identical, it's ok to configure DFS replication. If you see a lot of Event ID 4412 messages in the DFS Replication event log, there probably is an issue with the file hashes.

Monitoring HP MSA P2000 G3 I/O latency with Nagios

2014-02-04T12:00:00+01:00

At work we have a couple of HP MSA P2000 G3 SANs. These are entry-level SANs that still seem to have almost all features you might want from a SAN, except for official SSD-support.

It seems that the new MSA 2040 adds support for SSDs and also provides 4 GB cache per controller instead of the somewhat meager 2GB of the P2000.

Anyway, a very nice feature of the MSA P2000 G3 is the fact that the management interface also provides a well-documented API that allows you to collect detailed stats on subjects like:

Overall enclosure status
Reports on failed drives
Controller CPU usage
IOPs per controller
IOPs per vdisk
IOPs per disk

Thomas Weaver has written a Nagios plugin that does that: it collects this information and in turn you can graph it with pnp4nagios.

In more recent firmware updates HP has added support to monitor read and write I/O latency per vdisk. Latency is an important indicator for application-level performance so I was quite happy with that.

As the plugin by Thomas did not support reading these parameters yet, I spend some time implementing this check and submitted this new version of check_p2000_api.php back to Thomas.

Read Latency of a RAID 6 array

Write Latency of a RAID 6 array

You will notice that the write latency of this disk array is very high at times, which seem to indicate that this vdisk is taxed too much with too many I/O-requests.

I'd like to thank Thomas Weaver for writing this plugin, I think it's very useful.

An affordable server platform based on server-grade hardware

2013-12-13T12:00:00+01:00

Updated post (June 2014) found HERE

There are some reasons why you should consider buying true server-grade harware when building a server, wether it's for home or business use.

This is why I want to introduce you to the Supermicro X9SCM-F motherboard.

This motherboard costs about $160 or €160, which is more than a desktop grade motherboard, but you get quite a lot in return.

Remote KVM over IP

First of all, this motherboard has a separate network interface dedicated to an on-board Keyboard-Video-Mouse interface (IPMI). This interface allows you to power the server on or off, enter the BIOS all through a web interface, through the network. You never need to be in the vicinity of your server unless you need to perform some hardware maintenance.

Support for ECC (Error-correcting Code) RAM

All server grade hardware from manufactures like Supermicro, Dell or HP all ship their servers with ECC RAM. This type of memory is more expensive than regular RAM, but based on the name, you may guess the benefit: it detects and corrects memory errors. If you truly care about data integrity and availability, this is a recommended feature.

Support for both cheap or faster and more expensive processors

Cheap processor with ECC RAM support

The cheapest solution seems to be the Intel Pentium G2030 @ 2.00 GHz. For only 72 dollar or 52 Euro it can be yours and it supports ECC memory. Performance is not stellar but sufficient for most NAS builds. More proof that a build based on ECC does not have to be expensive.

One step up the ladder is the Intel Core i3-3220. It retails for about $124 or €125 and it supports ECC memory, something I didn't suspect of the i3 series. At 3.3Ghz it's quite fast and it is a dual-core processor with hyper-threading support. The processor also supports virtualisation (VT-x), but it does not support I/O virtualisation (VT-d).

Faster processor with full virtualisation support

At $229 or €200 the Intel Xeon E3-1230 V2 cost you twice the price of the Core i3, but you get twice the processor cores (quad-core) with hyper-threading, turbo-boost up to 3.7 ghz and full virtualisation support (both VT-x and VT-d). For the money, you get a beast of a processor with ECC RAM support.

Overview of CPU's

CPU	Passmark score	Price in Euro	Price in Dollars
Intel Pentium G2030 @ 3.00GHz	3008	52 Euro	72 Dollar*
Intel Core i3-3220 @ 3.30GHz	4226	97 Euro	125 Dollar
Intel Xeon E3-1230 V2 @ 3.30GHz	8890	196 Euro	230 Dollar

Dollars are from Newegg (* is estimate).
Euros are including taxes.

Support for 32 GB RAM

32 GB is quite some RAM and the maximum amount supported by the free version of VMware ESXi (ESXi 5.5 does not have this limit I've been told). Basically, it supports entry-level virtualisation needs or web-application needs. The 32 GB RAM limit is one of the biggest constraints of this platform, but may be sufficient for most applications except for large virtualisation loads or big databases.

32 GB is more than enough for everyone who likes to build a beefy storage server.

Expandability: 4 x PCI-e 8x (physical size) slots

This is probably why the Supermicro X9SCM-F motherboard is so interesting for NAS or storage builders. It has four PCI-e slots in a 8x physical form factor. Two of them are PCI-e 2.0 4x in a physical 8x slot and the other two are true PCI-e 3.0 8x slots. But even a PCI-e 2.0 4x slot provides you with 4 x 500 MB/s = 2000 MB/s per slot. The 8x slots have 8 lanes at 985 MB/s each thus totalling almost 8 GB/s! Let's be honest: the PCI-e 2.0 4x provide more than enough bandwidth to power any disk controller or network interface.

Because there are four slots, there are several possibilities. For example, you could populate them with four IBM M1015 HBA's providing you with 32 SAS/SATA ports for a total of 32 disks. With the six on-board SATA ports, you could theoretically connect 38 disks.

If you put only 3 x IBM M1015 HBA's in the motherboard you will leave room for fast networking, like a quad-port gigabit network card. You can then put the four gigabit ports into Linux interface bonding to achieve 400 MB/s network transfer speeds.

You could also consider to implement 10GB network cards (more expensive) or take a look into fibre channel or infiniband network connectivity.

With those four PCI-e slots, you can come a very long way.

Two on-board Gigabit network interfaces

The motherboard supports two on-board network interfaces and you can use them to bond them together for extra performance and/or fail-over.

You could also use one interface for regular LAN traffic and the other one for iSCSI or other protocols.

Conclusion

I believe the Supermicro X9SCM-F motherboard is a very interesting all-round platform for any type of server and particularly interesting as a platform for storage servers.

If you take the motherboard ($160), 8 GB RAM ($75) and entry level processor ($125) you get a server-grade platform for $360. The same platform with 16 GB RAM will cost you about $435. The prices in Euros will be about the same.

Script that shows smart values of all disks'

2013-10-05T01:00:00+02:00

Please use this tool on github instead of this ancient script.

So you have a Linux system with a lot of hard drives. If you want to quickly check on some key SMART values to determine the health of individual disks, you might be interested in this script.

I wrote a small Python script called 'showsmart' that displays key SMART values for all disks that support SMART. This has been tested on Ubuntu and Debian.

REALLOC. = Reallocated Sector Count
PENDING = Current Pending Sector
CRC ERR. = UDMA CRC ERROR

This script only requires python and smartctl (smartmontools).

I hope someone will find it useful.

Script that shows relevant disk information

2013-10-02T01:00:00+02:00

I wrote a small Python script called 'showdisks' that displays relevant information about any physical storage devices supported by hdparm.

Information such as model and capacity are shown, but also controller and device path.

This script only requires python and hdparm.

I hope someone will find it useful.

Using iSCSI with time machine and Super Duper

2013-07-21T16:00:00+02:00

In the past, as a Mac user, I've used separate external drives for Time Machine backups and Super Duper clones but I'm not happy with that. External hard drives make noise and create clutter.

I'd like to move away all my storage from my living room (or home office) and put it in another room or even closet.

A NAS may help with that but a NAS does not solve all problems. The main problem being the reliability of network-based Time Machine backups. Those NAS devices pretend to be Time Capsules, but there's always the risk that Apple breaks compatibility with a future update.

From my experience, Time Machine backups are only 100% reliable with local attached storage - like external hard drives.

Now there is a cool technology called iSCSI. It's basically a storage protocol tunneled through your home LAN network instead of a USB / Firewire or Thunderbolt cable. Most NAS devices support iSCSI and allow you to carve out some local NAS storage and present it to your computer through the network as if it was just local storage. Since iSCSI uses your Gigabit network as a transport, you can achieve transfer speeds of around ~110 MB/s easily, which should suit most needs*.

This is very cool, because you can export entire hard drives through the network to your computer. Your computer does not see the difference between an external USB hard drive and a hard drive exported through your NAS to your computer. iSCSI is totally transparent from the perspective of the operating system.

This trick allows you to create bootable Super Duper clones of your boot drive through the network. I would just hook up an external USB drive to my NAS and export it through iSCSI.

In case of an emergency - when your boot drive dies - you can boot from this external hard drive. Just disconnect it from your NAS and hook it up to your Mac.

Because hard drives attached through iSCSI are seen as normal storage, you can also encrypt them with the Apple build-int whole-drive (or whole-partition) encryption.

Now there is one caveat. Mac OS X does not natively support iSCSI, it has no native iSCSI initiator (client). In contrast, Windows 7 does have a very good iSCSI initiator. I think it's a shame, but Mac users must buy an iSCSI initiator from either:

I've only used the GlobalSAN iSCSI initiator and it seems to work fine. I believe that $89 is well worth the money: all your storage tucked away from your home office or living room.

Another caveat is that iSCSI requires reliable networking or otherwise there is a possible risk of data corruption, so I would not advice using iSCSI over a wireless network connection, although it is possible.

For the most popular NAS vendors, I've added some tutorials on how to setup iSCSI.

P.S. The GlobalSAN iSCSI initiator does support sleep and hibernate, in contrast to what some tutorials may tell you.

Improving iSCSI Native Multi Pathing Round Robin performance

2013-05-27T00:00:00+02:00

10 Gb ethernet is still quite expensive. You not only need to buy appropriate NICS, but you must also upgrade your network hardware as well. You may even need to replace existing fiber optic cabling if it's not rated for 10 Gbit.

So I decided to still just go for plain old 1 Gbit iSCSI based on copper for our backup SAN. After some research I went for the HP MSA P2000 G3 with dual 1 Gbit iSCSI controllers.

Each controller has 4 x 1 Gbit ports, so the box has a total of 8 Gigabit ports. This is ideal for redundancy, performance and cost. This relatively cheap SAN does support active/active mode, so both controllers can share the I/O load.

The problem with storage is that a single 1 Gbit channel is just not going to cut it when you need to perform bandwidth intensive tasks, such as moving VMs between datastores (within VMware).

Fortunately, iSCSI Multi Pathing allows you to do basically a RAID 0 over multiple network cards, combining their performance. So four 1 Gbit NICS can provide you with 4 Gbit of actual storage throughput.

The trick is not only to configure iSCSI Multi Pathing using regular tutorials, but also to enable the Round Robin setting on each data store or each RAW device mapping.

So I dit all this and still I got less than 1 Gb/s performance, but fortunately, there is only one little trick to get to the actual performance you might expect.

I found this at multiple locations but the explanation on Justin's IT Blog is best.

By default, VMware issues 1000 IOPS to a NIC before switching (Round Robin) to the next one. This really hampers performance. You need to set this value to 1.

esxcli storage nmp psp roundrobin deviceconfig set -d $DEV --iops 1 --type iops

This configuration tweak is recommended by HP, see page 28 of the linked PDF.

Once I configured all iSCSI paths to this setting, I got 350 MB/s of sequential write performance from a single VM to the datastore. That's decent enough for me.

How do you do this? It's a simple one liner that sets the iops value to 1, but I'm so lazy, I don't want to copy/past devices and run the command by hand each time.

I used a simple CLI script (VMware 5) to configure this setting for all devices. SSH to the host and then run this script:

for x in `esxcli storage nmp device list | grep ^naa`
do
    echo "Configuring Round Robin iops value for device $x"
    esxcli storage nmp psp roundrobin deviceconfig set -d $x --iops 1 --type iops
done

This is not the exact script I used, I have to verify this code, but basically it just configures this value for all storage devices. Devices that don't support this setting will raise an error message that can be ignored (if the VMware host also has some local SAS or SATA storage, this is expected).

The next step is to check if this setting is permanent and survives a host reboot.

Anyway, I verified the performance using a Linux VM and just writing a simple test file:

dd if=/dev/zero of=/storage/test.bin bs=1M count=30000

To see the Multi Pathing + Round Robin in action, run esxtop at the cli and then press N. You will notice that with four network cards, VMware will use all four channels available.

This all is to say that plain old 1 Gbit iSCSI can still be fast. But I believe that 10 Gbit ethernet does probably provide better latency. If that's really an issue for your environment, is something I can't tell.

Changing the IOPS parameter to 1 IOPS also seems to improve random I/O performance, according to the table in Justin's post.

Still, although 1 Gbit iSCSI is cheap, it may be more difficult to get the appropriate performance levels you need. If you have time, but little money, it may be the way to go. However, if time is not on your side and money isn't the biggest problem, I would definitely investigate the price difference with going for fibre channel or with 10Gbit iSCSI.

Creating storage benchmark charts with FIO and GNUplot

2013-05-22T20:00:00+02:00

Edit 2019: I've made a new tool called 'fio-plot'to create various graphs.

I use FIO to perform storage IO performance benchmarks. FIO does provide a script called "fio_generate_plots" which generates PNG or JPG based charts based on the data generated by FIO. The charts are created with GNUplot.

The "fio_generate_plots" didn't make me very happy as it didn't generate the kind of graphs I wanted. Furthermore, the script just contains some copy/pastes of the same blocks of code, slightly altered for the different benchmark types. I understand that the focus lies on FIO itself not some script to generate some fancy graphs, so don't get me wrong, but the script could be improved.

I used this script as the basis for a significantly reworked version, putting the code in a function that can be called with different parameters for the different benchmark types.

The result of this new script is something like this:

You can download this new script here. This script requires GNUplot 4.4 or higher.

Update 2013/05/26

I've submitted the script as a patch to the maintainers of FIO and it has been committed to the source tree. I'm not sure how this will work out but I assume that this script will be part of newer FIO releases.

Storage and I/O: reads vs. writes

2013-04-02T00:00:00+02:00

There is a fundamental difference between a read operation and a write operation. Storage can lie about completing a write operation, but it can never lie about completing a read operation. Therefore read and writes have different characteristics. This is what I've learned.

About writes

So what does this mean? Well, if you write data to disk, the I/O subsystem only has to acknowledge that it has written the data to the actual medium. Basically, the application says "please write this data to disk" and the I/O subsystem answers "done, feel free to give me another block of data!".

But the application cannot be sure that the I/O subsystem actually wrote that data to disk. More likely, the application can be sure the I/O subsystem lied.

Compared to RAM, non-volatile storage like hard-drives are slow. Orders of magnitudes slower. And the worst-case scenario, which is often also the real-life scenario, is that both read and write patterns are random as perceived from the storage I/O subsystem.

So you have this mechanical device with rotating platters and a moving arm, governed by Newtons rules of physics, trying to compete with CPUs and memory that are so small that they are affected by quantum mechanical effects. No way that device is going to be able to keep up with that.

So the I/O subsystem cheats. Hard drives are relatively great at reading and writing blocks of data sequentially, it's the random access patterns that wreaks havoc on performance. So the trick is to lie to the application and collect a bunch of writes in a cache, in memory.

So, meanwhile, the I/O subsystem looks at the data to be written to disk, and reorders the write operations, so that it becomes as 'serialised' as possible. It tries to take into account all the latencies involved in moving the arm, timing that with the rotation of the platter and that kind of stuff.

A 7200 RPM hard drive can do only 75 IOPS with random access patterns, but that is a worst-case of worst-case scenario's. Real-life usage scenario's often allow for some optimalisation.

I used FIO to perform some random-IO performance benchmarks on different hard drive types and RAID configurations. It turns out that read performance was conform the 75 IOPS, but writes where in the thousands of IOPS, not a realistic figure. The operating system (Linux) employed heavy caching of the writes, lying to FIO about the actual IOPS being written to disk.

Thousands of IOPS sounds great, but you can only lie until your write cache is full. There comes a time when you have to actually deliver and write this data to disk. This is where you see large drops in performance, to almost zero IOPS.

Most of the time, this behaviour is overall beneficial to application performance, as long as the application usage patterns are often short bursts of data, that need to be written to disk. With more steady streams of data being written to disk in a random order, this might influence application responsiveness. The application might become periodically unresponsive as data is flushed from the cache to disk.

This write-caching behaviour is often desired, because by reordering and optimising the order of the write requests, the actual overall obtained random I/O write performance is often significantly higher than could be achieved by the disk subsystem itself.

If the disk subsystem is not just a single disk, but a RAID array, comprised of multiple drives, write-caching is often even more important to keep performance acceptable, especially for RAID arrays with parity, such as RAID 5 and RAID 6.

Write-back caching may help increase performance significantly, but it may come at a cost. As the I/O subsystem lies about data being written to disk, that data may get lost if the system crashes or loses power. There is a risk of data loss or data corruption. Only use write-back caching on equipment that is supported by battery backup units and a UPS. Due to the risks associated with write-back caching, there might be use cases where it might be advised not to enable it to retain data consistency.

About reads

The I/O subsystem can't lie to the application about reads. If the application asks the I/O subsystem "can I have the contents of file X", the I/O subsystem can't just say "well, yes, sure". It actually has to deliver that data. So any arbitrary write can be easily cached and written to disk in a more optimised way, reads may be harder. There is no easy way out, the I/O subsystem must deliver.

Where any arbitrary write can be cached, only a limited number of reads can be cached. Cache memory is relatively small compared to the storage of the disk subsystem. The I/O subsystem must be smart about which data needs to be cached.

More complex storage solutions keep track of 'hot spots' and keep that data cached. As a side note, such caching constructions can now also be found in consumer grade equipment: Apple's fusion drive uses the SSD as a cache and stores the data that is less frequently accessed on the HDD.

But in the end, regarding reads, chances are higher that data must be retrieved that is not stored in cache (cache miss) and thus the drives must do actual work. Fortunately, that work is not as 'expensive' as writes for RAID 5 or RAID 6 arrays.

Furthermore, reads can also be 'grouped' and serialised (increased queue depth) at the cost of latency to optimise them (setup a more sequential read access pattern for the disk subsystem) and achieve better performance. But again, at the cost of latency, thus responsiveness. That may or may not be a problem depending of the type of application.

Some remarks

If possible, it's better to try and avoid having to access the storage subsystem in the first place, if possible. Try and trow RAM memory at the problem. Buy systems with sufficient RAM memory, so that the entire database fits in RAM memory. A few years ago this was unthinkable, but 128 GB of RAM memory can be had for less than two thousand dollars.

If RAM isn't an option (dataset is too large) still try and put in as much RAM as possible. Also, try and see if server grade Solid State Drives (SSDs) are an option (always RAID 1 at least for redundancy!), although their cost may be an obstacle.

The gateway of last resort is the old trusted hard drive. If random I/O is really an issue, take a look at 15000 RPM or at least 10000 RPM SAS drives and a good RAID controller with loads of cache memory. In general, more drives or more 'spindles' equals more I/O performance.

You might encounter a situation where you want to add drives to increase I/O performance, not for the storage. More important: you may choose not to use that extra storage as it may decrease performance. Because if you put more data on a disk, the head must cover larger areas of the disk platter, increasing latency.

There are usecases where drives are intentionally under-partitioned to (artificially) increase the performance of the drives.

Benchmark results of Random I/O performance of different RAID levels

2013-01-01T00:00:00+01:00

Introduction

I have performed some benchmarks to determine how different RAID levels perform when handling a 100% random workload of 4K requests. This is a worst-case scenario for almost every storage subsystem. Normal day-to-day workloads may not be that harsh in a real-life environment, but worst-case, these tests show what kind of performance you might expect when you face such a workload.

To create a worst-case worst-case solution, I even disabled write-caching for all write-related tests.

At the moment, I only have access to some consumer-level test hardware. In the future, I'd like to rerun these tests on some 10K RPM SAN storage drives to see how this turns out.

RAID levels tested

I have tested the following RAID levels:

RAID 0
RAID 10
RAID 5
RAID 6

Test setup

CPU: Intel Core i5 2400s @ 2.5 GHz
RAM: 4 GB
Drives: 6 x 500 GB, 7200 RPM drives (SATA).
Operating system: Ubuntu Linux
RAID: Build-in Linux software RAID (MDADM)
File system: XFS
Test file size: 10 GB
Test software: FIO read-config & write-config
Queue depth: 4
Test script that generates RAID arrays, file systems and runs the tests.
Cache: all write caching was disabled during testing (see script)

Test results

I also tested various chunk sizes for each RAID level. These are the results for RAID 10.

If you don't see any images, you are not using Internet Explorer 9, or a recent version of Google Chrome, Mozilla Firefox or Apple Safari.

Analysis

With this kind of testing, there are so many variables that it will be difficult to make any solid observations. But these results are interesting.

Results are in line with reality

First of all, the results do not seem unexpected. Six drives at 7200 RPM should each provide about 75 IOPS. This should result in a total of 450 IOPS for the entire array. The read performance does show exactly this kind of performance.

With all caching disabled, write performance is worse. And especially the RAID levels with parity (RAID 5 and RAID 6) show a significant drop in performance when it comes to random writes. RAID 6 write performance got so low and erratic that I wonder if there is something wrong with the driver or the setup. Especially the I/O latency is off-the-charts with RAID 6, so there must be something wrong.

Read performance is equal for all RAID levels

However, the most interesting graphs are about IOPS and latency. Read performance of all different RAID arrays is almost equal. RAID 10 seems to have the upper hand in all read benchmarks. I'm not sure why this is. Both bandwidth and latency are better than the other RAID levels. I'm really curious about a good technical explanation about why this should be expected. Edit: RAID10 is basically multiple RAID 1 sets stuck together. Data is striped across RAID 1 sets. When reading, a single stripe can be deliverd by both disks in the particular RAID mirror it resides on, thus there is a higher risk that one of the heads is in the vicinity of the requested sector.

RAID 0 is not something that should be used in a production environment, but it is included to provide a comparison for the other RAID levels. The IOPS graph regarding write performance is most telling. With RAID 10 using 6 drives, you only get the effective IOPS of 3 drives, thus about 225 IOPS. This is exactly what the graph is showing us.

Raid with parity suffers regarding write performance

RAID 5 needs four write I/Os for every application-level write request. So with 6 x 75 = 450 IOPS divided by 4, we get 112,5 IOPS. This is also on par with the graph. This is still ok, but notice the latency: it is clearly around 40 milliseconds, whereas 20 milliseconds is the rule of thumb where performance will start to significantly degrade.

RAID 6 needs six write I/Os for every application-level write request. So with 450 IOPS total, divided by 6, we only have single-disk performance of 75 IOPS. If we average the line, we do approximately get this performance, but the latency is so erratic that it would not be usable.

RAID chunk size and performance

So I was wondering if the RAID array chunk size does impact random I/O performance. It seems not.

Conclusion

Overall, the results seem to indicate that the actual testing itself is realistic. We do get figures that are in tune with theoretical results.

The erratic RAID 6 write performance would need a thorougher explanation, one that I can't give.

Based on the test results, it seems that random I/O performance for a single test file is not affected by the chunk size or stripe size of an RAID array.

The results show to me that my benchmarking method provides a nice basis for further testing.

Understanding IOPS, latency and storage performance

2012-11-25T21:00:00+01:00

Update 2020: I've written another blogpost about this topic, including some benchmark examples.

When most people think about storage performance, they think about throughput. But throughput is similar to the top speed of a car. In reality, you will almost never reach the top speed of your car (unless you are living in Germany). And that's fine, because in most situations that's not so relevant.

For instance, properties like how fast your car accelerates and how well the car handles bends and corners are often more important than its top speed. And this example also holds for storage performance.

Most people know that SSDs are often way faster than regular mechanical hard drives. But it's not about the throughput of these devices. Its all about Input/Output operations per second (IOPS). If you can handle a high number of IOPS, that is great for real life application performance. But IOPS does not tell you the whole story. To be more precise: IOPS is a meaningless figure unless tied to an average latency and a certain request size (how much data is processed with the I/O). Let's first focus on IOPS and Latency and talk about the request size later.

Latency is how fast a single I/O-request is handled. This is very important, because a storage subsystem that can handle 1000 IOPS with an average latency of 10ms may get better application performance than a subsystem that can handle 5000 IOPS with an average latency of 50ms. Especially if the application is sensitive to latency, such as a database service.

This is a very important thing to understand: how IOPS and latency relate to each other. Here, the car analogy probably breaks down. We need a different one to better understand what is going on. So picture you are in a super market. This is a special supermarket, where customers (I/Os) are served by cashiers (disk) at an average speed of 10ms. If you divide one second with 10ms, we understand that this cashier can handle 100 customers per second. But only one at a time, in succession.

It is clear that although the cashier can handle 100 customers per second, he cannot handle them at the same time! So when a customer arrives at the register, and within those 10ms handling time, a second customer arrives, that customer must wait. Once the waiting customer is handled by the cashier, handling of that customer still takes just 10ms, but the overal processing time was maybe 15ms or worst case (two customers arriving at the same time) even 20ms.

So it is very important to understand that although a disk may handle individual I/Os with an average latency of 10ms, the actual latency as perceived by the application may be higher as some I/Os must wait in line.

This example also illustrates that waiting in line increases the latency for a particular I/O to be handled. So if you increase the Read I/O queue, you will notice that the average latency will increase. Longer queues will mean higher latency, but also more IOPS!!!

How is that possible? How can a disk drive suddenly do more random IOPs at the cost of latency? The trick lies in that the storage subsystem can be smart and look at the queue and then order the I/Os in such a way that the actual access pattern to disk will be more serialised. So a disk can serve more IOPS/s at the cost of an increase in average latency. Depending on the achieved latency and the performance requirements of the application layer, this can be acceptable or not.

In future blog posts I will show some performance benchmarks of a single drive to illustrate these examples.

Speeding up Linux MDADM RAID array rebuild time using bitmaps

2011-12-22T21:00:00+01:00

Update 2020: Please beware of the impact of random write I/O performance.

Please note that with a modern Linux distribution, bitmaps are enabled by default. They will not help speed up a rebuild after a failed drive. But it will help resync an array that got out-of-sync due to power failure or another intermittent cause.

When a disk fails or gets kicked out of your RAID array, it often takes a lot of time to recover the array. It takes 5 hours for my own array of 20 disks to recover a single drive.

Wouldn't it be nice if that time can be reduced? Even to 5 seconds?

Although not enabled by default, you can enable so called 'bitmaps'. As I understand it, a bitmap is basically a map of your RAID array and it charts which areas need to be resynced if a drive fails.

This is great, because I have the issues that of every 30 reboots, sometimes a disk won't get recognized and the array is degraded. Adding the disk back into the array will mean that the system will be recovering for 5+ hours.

I enabled Bitmaps and after adding a missing disk back into the array, the array was recovered instantly.

Isn't that cool?

So there are two types of bitmapsL

internal: part of the array itself
external: a file residing on an external drive outside the array

The internal bitmap is integrated in the array itself. Keeping the bitmap up to date will probably affect performance of the array. However I didn't notice any performance degradation.

The external bitmap is a file that must reside on a EXT2 or EXT3 based file system that is not on top of the RAID array. So this means that you need an extra drive for this or need to use your boot drive for this. I can imagine that this solution will have less impact on the performance of the array but it is a bit more hassle to maintain.

I enabled an internal bitmap on my RAID arrays like this:

mdadm --grow /dev/md5 --bitmap=internal

This is all there is to it. You can configure an external bitmap like this:

mdadm --grow /dev/md5 --bitmap=/some/directory/somefilename

There probably will be some performance penalty involved, but it does not seem to affect sequential throughput, which is the only thing that is important for my particular case.

For most people, I would recommend configuring an internal bitmap, unless you really know why you would have to use an external bitmap.

Cheap solution for putting an SSD in an iMac

2011-07-11T20:00:00+02:00

If you want to order a new iMac with an SSD there are two problems:

An SSD of 256 gigabytes will cost you a lot: 600 euros.
The SSD is of medium quality, and does not justify the cost of 600 euros.

Here you can find information about the Toshiba SSD that Apple provides.

You can consider buying an SSD for half the price and put it inside your new iMac yourself afterwards. However, this is a hassle and you will probably void your waranty.

The solution is to buy an SSD yourself and put it in an external Firewire 800 casing. Firewire 800 provides you with about 70 megabytes per second of troughput, which is enough for most applications. It is about three times faster than USB and the latency of Firewire is lower than that of USB, thus improving responsiveness.

Thunderbolt would provide the best solution, but as of July 2011, there are no Thunderbolt products on the market.

The product I use is an external 2,5 inch Firewire 800 casing of OWC: the Mercury Elite-AL Pro mini. This casing cost me about 60 euros or 80 dollars including shipping costs. I admit that this solution is not cheap, but it does work. And you won't void your warranty and it is way cheaper than the Apple SSD.

I've put my Intel SSD inside this casing and my mac boots in about 15 seconds. After entering username and password, login is almost instantaneously.

Caveat

I encountered one problem with the OWC casing. I encountered random system freezes, related to the external OWC housing. Using the internal 1 TB hard drive, I had no problems. The cause of this issue is probably related to power not being provided to the OCW casing after the systems returns from sleep.

I resolved this issue by connecting a special USB cable to the iMac and connecting the other end to the 5V DC Power Input on the OWC casing. After that, the daily random system freezes vanished. These cables can be obtained almost everywhere.

Thunderbolt, a cheap high-speed storage interconnect?

2011-02-25T21:30:00+01:00

Intel and Apple released Thunderbolt a high-speed (10 Gigabit/s) interface, that seems to replace both USB and Firewire. It is mainly targeted at end-user systems allowing to connect peripherals with just a single cable to a computer. Thunderbolt devices, like external hard drives or displays can be daisy chained, like Firewire. In short, Thunderbolt removes the cable clutter and ads a significant speed bonus.

For NAS owners and storage enthusiasts, this is also a very interesting technology. Just like Firewire, it seems to support computer-to-computer communication. So Thunderbolt could be used as a high-speed link between your homegrown NAS device and your PC workstation. Or between two storage / server system.

The only downside to Thunderbolt is the maximum cable length of 3 meters between devices. Thunderbolt doesn't seem to be the ideal replacement for your Gigabit network, but if most of your computer systems are close to each other, it might be very interesting.

Do not buy a hardware RAID controller for home use

2010-11-17T21:00:00+01:00

Hardware RAID controllers are considered 'the best' solution for high performance and high availability. However, this is not entirely true. Using a hardware RAID controller might even endanger your precious data.

For enterprise environments, where performance is critical, it is more important that the arrays keeps on delivering data at a high speed. Professional RAID controllers use TLER with TLER-enabled disks to limit the time spend on recovering bad sectors. If a disk encounters a bad sector, there is no time to pause and try to fix it. The disk is just dropped out of the RAID array after just a couple of seconds. At that moment, the array still performes relatively well, but there is no redundancy. If another disk fails (another bad sector?) the array is lost, with all its data.

More people are building NAS boxes for centralized storage of data, for private home use. Since disks are cheap, it is possible to create lots of storage capacity for little money. Creating backups of terabytes of data is however not cheap. Or you have to create two NAS boxes. But that is very expensive and not worth the effort.

People seem to spend lots of money on expensive enterprise level hardware RAID cards, not understanding that the whole TLER-mechanism causes an increased risk for their data. In enterprise environments, budgets are relatively big, and data is always backed up. They can afford to take the risk of losing a RAID array due to these backups. But consumers often don't have the money to spend on creating backups of terabytes of data. They just go for RAID 5 or RAID 6 and hope for the best.

For consumers, if the RAID array goes, all data is lost.

So consumers should choose a RAID solution that will do its best to recover from hardware failure. Performance is not so much an issue. Reliability is. So consumers do want disks to spend 'ages' on recovering bad sectors if this means that the RAID array will survive.

Linux software RAID and ZFS do not use TLER and therefore are a safer choice for your data then regular hardware RAID controllers. You may still use such controllers (but please test them properly) but only to provide SATA ports with individual disks, the RAID part should be handled by Linux.

So in my opinion, hardware RAID controllers are more expensive, require more expensive (enterprise) disks and are less safe for your data.

Linux Software RAID benchmarking script

2010-09-29T20:58:00+02:00

Just a small post.

To benchmark your Linux software RAID array as setup with MDADM, please use my new benchmark script. I used this script to create these results.

You may need to configure some values within the header of this file to make it fit your enviroment.

 DEVICES="/dev/sd[a-f]"
 NO_OF_DEVICES=6
 ARRAY=/dev/md5
 CHUNKS="4 8 16 32 64 128 256 512 1024"
 MOUNT=/storage
 LOG=/var/log/raid-test.log
 LOGDEBUG=/var/log/raid-test-debug.log
 LEVEL="0 5 6 10"
 TESTFILE=$MOUNT/test.bin
 TESTFILESIZE=10000 (IN MB, thus this is 10 GB)
 TRIES=5 (how many times to run a benchmark.)

By default, the script wil format the array using XFS, feel free to format it with another filesystem such as EXT4 or EXT3 or whatever you want to test.

RAID 5 vs. RAID 6 or do you care about your data?

2010-08-13T15:06:00+02:00

Storage is cheap. Lots of storage with 10+ hard drives is still cheap. Running 10 drives increases the risk of a drive failure tenfold. So often RAID 5 is used to keep your data up and running if one single disks fails.

But disks are so cheap and storage arrays are getting so vast that RAID 5 does not cut it anymore. With larger arrays, the risk of a second drive failure while your failed array is in a degraded state (a drive already failed and the array is rebuilding or waiting for a replacement), is serious.

RAID 6 uses two parity disks, so you loose two disks of capacity, but the rewards in terms of availability are very large. Especially regarding larger arrays.

I found a blog posting that showed the results on a big simulation run on the reliability of various RAID setups. One picture of this post is important and it is shown below. This picture shows the risk of the entire RAID array failing before 3 years.

From this picture, the difference between RAID 5 and RAID 6 regarding reliability (availability) is astounding. There is a strong relation with the size of the array (number of drives) and the increased risk that more than one drive fails, thus destroying the array. Notice the strong contrast with RAID 6.

Even with a small RAID 5 array of 6 disks, there is already a 1 : 10 chance that the array will fail within 3 years. Even with 60+ drives, a RAID 6 array never comes close to a risk like that.

Creating larger RAID 5 arrays beyond 8 to 10 disks means there is a 1 : 8 to 1: 5 chance that you will have to recreate the array and restore the contents from backup (which you have of course).

I have a 20 disk RAID 6 running at home. Even with 20 disks, the risk that the entire array fails due to failure of more than 2 disks is very small. It is more likely that I lose my data due to failure of a RAID controller, motherboard or PSU than dying drives.

There are more graphs that are worth viewing, so take a look at this excelent blog post.

Lustre and the risk of Serious Data Loss

2010-07-03T22:53:00+02:00

Personally I have a weakness for big-ass storage. Say 'petabyte' and I'm interested. So I was thinking about how you would setup a large, scalable storage infrastructure. How should such a thing work?

Very simple: you should be able just to add hosts with some bad-ass huge RAID arrays attached to them. Maybe even not that huge, say 8 TB RAID 6 arrays or maybe bigger. You use these systems as building blocks to create a single and very large storage space. And then there is one additional requirement: as the number of these building blocks increase, you must be able to loose some and not loose data or availability. You should be able to continue operations without one or two of those storage building blocks before you would loose data and/or availability. Like RAID 5 or 6 but then over server systems instead of hard drives.

The hard part is in connecting all this separate storage to one virtual environment. A solution to this problem is Lustre.

Lustre is a network clustering filesystem. What does that mean? You can use Lustre to create a scalable storage platform. A single filesystem that can grow to multiple Petabytes. Lustre is deployed within production environments at large scale sites involving some of the fastest and largest computer clusters. Luster is thus something to take seriously.

Lustre stores all metadata about files on a separate MetaDataServer (MDS). Al actual file data is stored on Object Storage Targets (OSTs). These are just machines with one or more big RAID arrays (or simple disks) attached to them. The OSTs are not directly accessible by clients, but through an Object Storage Server (OSS). The data stored within a file can be striped over multiple OSTs for performance reasons. A sort of network RAID 0.

Lustre does not only allow scaling up to Petabytes of storage, but allows also a parallel file transfer performance in excess of 100 GB/s. How you like them apples? That is just wicked sick.

Just take a look at this diagram about how Lustre operates:

I'm not going into the details about Lustre. I want to discuss a shortcoming that may pose a serious risk of data loss: if you loose a single OST with any attached storage, you will lose all data stored on that OST.

Lustre cannot cope with the loss of a single OST! Even if you buy fully redundant hardware, with double RAID controllers, ECC memory, double PSU, etc, even then, if the motherboard gets fried, you will loose data. Surely not everything, but let's say 'just' 8 TB maybe?

I guess the risk is assumed to be low, because of the wide scale deployment of Lustre. Deployed by people who actually use it and have way more experience and knowledge than me about this whole stuff. So maybe I'm pointing out risks that are just very small. But I have seen server systems fail this bad as described. I don't think the risk, especially at this scale, is not that small.

I am certainly not the first to point out this risk.

The solution for Lustre to became truly awesome is to implement some kind of network based RAID 6 striping so you could loose one or even two OSTs and not have any impact on availability except maybe for performance. But it doesn't (yet).

This implies that you have to create your OSTs super-reliable, which would be very expensive (does not scale). Or have some very high-capacity backup solution, which would be able to restore some data. But you would have downtime.

So my question to you is: is there an actual scalable filesystem as Lustre that actually is capable of withstanding the failure of a single storage building block? If you have something to point out, please do.

BTW: please note that the loss of an OSS can be overcome because another OSS can take over the OSTs of a failed OSS.

'Linux RAID level and chunk size: the benchmarks'

2010-05-23T19:11:00+02:00

Introduction

When configuring a Linux RAID array, the chunk size needs to get chosen. But what is the chunk size?

When you write data to a RAID array that implements striping (level 0, 5, 6, 10 and so on), the chunk of data sent to the array is broken down in to pieces, each part written to a single drive in the array. This is how striping improves performance. The data is written in parallel to the drive.

The chunk size determines how large such a piece will be for a single drive. For example: if you choose a chunk size of 64 KB, a 256 KB file will use four chunks. Assuming that you have setup a 4 drive RAID 0 array, the four chunks are each written to a separate drive, exactly what we want.

This also makes clear that when choosing the wrong chunk size, performance may suffer. If the chunk size would be 256 KB, the file would be written to a single drive, thus the RAID striping wouldn't provide any benefit, unless manny of such files would be written to the array, in which case the different drives would handle different files.

In this article, I will provide some benchmarks that focus on sequential read and write performance. Thus, these benchmarks won't be of much importance if the array must sustain a random IO workload and needs high random iops.

Test setup

All benchmarks are performed with a consumer grade system consisting of these parts:

Processor: AMD Athlon X2 BE-2300, running at 1.9 GHz.

RAM: 2 GB

Disks: SAMSUNG HD501LJ (500GB, 7200 RPM)

SATA controller: Highpoint RocketRaid 2320 (non-raid mode)

Tests are performed with an array of 4 and an array of 6 drives.

All drives are attached to the Highpoint controller. The controller is not used for RAID, only to supply sufficient SATA ports. Linux software RAID with mdadm is used.
A single drive provides a read speed of 85 MB/s and a write speed of 88 MB/s
The RAID levels 0, 5, 6 and 10 are tested.
Chunk sizes starting from 4K to 1024K are tested.
XFS is used as the test file system.
Data is read from/written to a 10 GB file.
The theoretical max through put of a 4 drive array is 340 MB/s. A 6 drive array should be able to sustain 510 MB/s.

About the data:

All tests have been performed by a Bash shell script that accumulated all data, there was no human intervention when acquiring data.
All values are based on the average of five runs. After each run, the RAID array is destroyed, re-created and formatted.
For every RAID level + chunk size, five tests are performed and averaged.
Data transfer speed is measured using the 'dd' utility with the option bs=1M.

Test results

Results of the tests performed with four drives:

Test results with six drives:

Analysis and conclusion

Based on the test results, several observations can be made. The first one is that RAID levels with parity, such as RAID 5 and 6, seem to favor a smaller chunk size of 64 KB.

The RAID levels that only perform striping, such as RAID 0 and 10, prefer a larger chunk size, with an optimum of 256 KB or even 512 KB.

It is also noteworthy that RAID 5 and RAID 6 performance don't differ that much.

Furthermore, the theoretical transfer rates that should be achieved based on the performance of a single drive, are not met. The cause is unknown to me, but overhead and the relatively weak CPU may have a part in this. Also, the XFS file system may play a role in this. Overall, it seems that on this system, software RAID does not seem to scale well. Since my big storage monster (as seen on the left) is able to perform way better, I suspect that it is a hardware issue.

because the M2A-VM consumer-grade motherboard can't go any faster.

20 disk 18 TB RAID 6 storage based on Debian Linux

2009-07-21T21:08:00+02:00

This system is no longer operational and has been decomissioned (2017)

This is my NAS storage server based on Debian Linux. It uses software RAID and 20 one terrabyte hard drives. It provides a total usable storage capacity of 18 terrabytes in a single RAID 6 array.

One of the remarkable side effects of using 20 drives within a single array is the read performance of over one gigabyte per second.

Case:	Norco RPC-4020
Processor:	Core 2 duo E7400 @ 2.8GHz
RAM:	4 GB
Motherboard:	Asus P5Q-EM DO
LAN:	Intel Gigabit
PSU:	~~Coolermaster 600 Watt~~ Corsair CMPSU-750HX 750 Watt (Coolermaster died)
Controller:	HighPoint RocketRAID 2340 (16) and on-board controller (6).
Disks:	20 x Samsung Spinpoint F1 (1 TB) and 2 x FUJITSU MHY2060BH (60 GB)
Arrays:	Boot: 2x 60 GB RAID 1 and storage: 20 x 1 TB RAID 6
RAID setup:	Linux software RAID using MDADM.
RAM:	4 GB
Read performance:	1.1 GB/s (yes this is correct, not a typo)
Write performance:	~~350~~ 450 MB/s. (suddenly faster after Debian update)
OS:	Linux Debian Squeeze 64-bit
Filesystem:	XFS (can handle > 16 TB partitions.)
Rebuild time:	about 5 hours.
UPS:	Back-UPS RS 1200 LCD using Apcupsd
Idle power usage:	about 140 Watt

Script that shows ETA of RAID rebuild / reshape

2009-06-28T22:43:00+02:00

I made a small script that converts the output of cat /proc/mdstat to an actual date and time telling you when the RAID rebuild / reshape is finished.

This is the link to the correct version of the script.

Example:

debian:~# ./raid-rebuild-eta.sh

Estimated time of finishing rebuild / reshape:
Mon Jun 29 00:45:07 CEST 2009

Calculating EXT2 EXT3 EXT4 stride size when using RAID

2008-12-20T22:15:00+01:00

When formatting a RAID device with an EXT filesystem, it is always advised to specify a stride size. The format utility will take this stride size into account when formatting a device. The stride size is the number you get when you divide the 'chunck' size, as specified with MDADM by the filesystem block size (almost always 4K).

So a 128 KB chunck size gives you a stride of 32. A nice and simple utility to calculate your stride can be found here:

http://busybox.net/~aldot/mkfs_stride.html

Please note that the stripe-with option does not seem to work on Debian Etch. Maybe because that option is too old or too new.

I also think that most recent tools automatically detect and calculate the correct stride size for you.

Is storage really that cheap?

2008-10-05T21:47:00+02:00

Nowadays you can buy a 1 TB harddrive for less than 100 euro's . So I build myself a 4 TB NAS box, which is already 50% full. However, although it is to some degree fault-tollerant by using RAID 6, one mistake or catastrophic hardware faillure and all data is lost.

And that's where the 'problem' start. For every € spend on storage, you may need another € to secure that storage.

You can choose to take and accept the risk outlined earlier and not to make backups. However, if you do want to make backups of terrabytes of storage, how are you going to pull that off without too much cost?

In my opinion, the only reliable and usable solution is to build a second NAS box and sync the two. Ideally, both machines reside at different locations, but hey, I'm talking about a home solution, not a professional environment. Although you might ask why on earth you need 4 TB of space at home in the first place.

Anyway, the point I'm trying to make is that although storage in itself is cheap, if you want to keep all that data safe, storage is far more expensive than you might think.

Building a RAID 6 array of mixed drives

2008-08-10T11:36:00+02:00

To be honest, 4 TB of storage isn't really necessary for home usage. However, I like to collect movies in full DVD or HD quality and so I need some storage.

I decided to build myself a NAS box based on Debian Etch. Samba is used to allow clients to access the data. The machine itself was initially based on 4 x 0.5 TB disks using the four SATA ports on the mainboard. With Linux build-in support for software RAID, I created a RAID 5 array, giving me 1.5 TB of storage space. Since a single movie is around 4 GB, the 1.5 TB turned out to become rather tight.

So I bought 4 x 1 TB disks and a Highpoint RocketRaid 2320 controller (SATA 4x). I put all 8 disks on this controller.

I wanted to create a single RAID 6 array using both the 1 TB disks and the 0.5 TB disks. I didn't want to create two separate array's because although it would have provided additional space, it wouldn't have given me the same safety level as RAID 6 does.

I mainly chose for RAID 6 since I cannot afford a backup solution for this amount of data. I'm aware that RAID is no substitute for a proper backup, but it's an accepted risk for me.

Using both 1 TB disks and 0.5 TB disks, how to create a RAID 6 array using different drive sizes? The solution is fairly simple. Just put two 0.5 TB disks together in one RAID 0 volume and you'll have a 'virtual' 1 TB disk. Since I had four 0.5 TB disks, I could create 2 'virtual' 1 TB disks.

The only downside is that I had to skim a little bit of storage capacity of the native 1 TB drives, because 2 x 0.5 TB provides slightly less storage space than a single 1 TB disk. We're talking about something like 50 MB here, so It's not a big deal in my opinion.

The funny thing is that this array actually performs rather well. The disks are connected using a HighPoint RocketRaid 2320 controller. This controller is used just for it's SATA-ports, the on-board RAID functionality is not used. For RAID, I use Linux software RAID, using mdadm. This is how the RAID 6 array looks like:

        server:~# mdadm --detail /dev/md5
        /dev/md5:
        Version : 00.90.03
  Creation Time : Thu Jul 24 22:40:26 2008
     Raid Level : raid6
     Array Size : 3906359808 (3725.40 GiB 4000.11 GB)
    Device Size : 976589952 (931.35 GiB 1000.03 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 5
    Persistence : Superblock is persistent
    Update Time : Sun Aug 10 15:36:18 2008
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0
     Chunk Size : 128K
           UUID : 0442e8fa:acd9278e:01f9e43d:ac30fbff (local to host server)
         Events : 0.14170

Number   Major   Minor   RaidDevice State
   0       8        1        0      active sync   /dev/sda1
   1       8       17        1      active sync   /dev/sdb1
   2       8       33        2      active sync   /dev/sdc1
   3       8       49        3      active sync   /dev/sdd1
   4       9        0        4      active sync   /dev/md0
   5       9        1        5      active sync   /dev/md1

And this is how this array performs:

server:~# dd if=/storage/test.bin of=/dev/null bs=1M

10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 45.7107 seconds, 229 MB/s

server:~# dd if=/dev/zero of=/storage/test.bin bs=1M count=10000

10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 81.0798 seconds, 129 MB/s

With 229 MB/s read performance and 129 MB/s write performance using RAID 6, I think I should be content.